Discrimination slope and integrated discrimination improvement - properties, relationships and impact of calibration

Development and validation of survival nomograms in elder triple-negative invasive ductal breast carcinoma patients

BACKGROUND

We aimed to develop a nomogram to predict the overall survival of elderly patients with Triple-negative invasive ductal breast carcinoma (TNIDC).

RESEARCH DESIGN AND METHODS

12165 elderly patients with nonmetastatic TNIDC were retrieved from the SEER database from 2010 to 2019 and were randomly assigned to training and validation cohorts. Stepwise Cox regression analysis was used to select variables for the nomogram based on the training cohort. Univariate and multivariate Cox analyses were used to calculate the correlation between variables and prognosis of the patients. Survival analysis was performed for high- and low-risk subgroups based on risk score.

RESULTS

Eleven predictive factors were identified to construct our nomograms. Compared with the TNM stage, the discrimination of the nomogram revealed good prognostic accuracy and clinical applicability as indicated by C-index values of 0.741 (95% CI 0.728-0.754) against 0.708 (95% CI 0.694-0.721) and 0.765 (95% CI 0.747-0.783) against 0.725 (95% CI 0.705-0.744) for the training and validation cohorts, respectively. Differences in OS were also observed between the high- and low-risk groups (p < 0.001).

CONCLUSION

The proposed nomogram provides a convenient and reliable tool for individual evaluations for elderly patients with M0_stage TNIDC. However, the model may only for Americans.

Machine-learning predictions for acute kidney injuries after coronary artery bypass grafting: a real-life muticenter retrospective cohort study

BACKGROUND

Acute kidney injury (AKI) after coronary artery bypass grafting (CABG) surgery is associated with poor outcomes. The objective of this study was to apply a new machine learning (ML) method to establish prediction models of AKI after CABG.

METHODS

A total of 2,780 patients from two medical centers in East China who underwent primary isolated CABG were enrolled. The dataset was randomly divided for model training (80%) and model testing (20%). Four ML models based on LightGBM, Support vector machine (SVM), Softmax and random forest (RF) algorithms respectively were established in Python. A total of 2,051 patients from two other medical centers were assigned to an external validation group to verify the performances of the ML prediction models. The models were evaluated using the area under the receiver operating characteristics curve (AUC), Hosmer-Lemeshow goodness-of-fit statistic, Bland-Altman plots, and decision curve analysis. The outcome of the LightGBM model was interpreted using SHapley Additive exPlanations (SHAP).

RESULTS

The incidence of postoperative AKI in the modeling group was 13.4%. Similarly, the incidence of postoperative AKI of the two medical centers in the external validation group was 8.2% and 13.6% respectively. LightGBM performed the best in predicting, with an AUC of 0.8027 in internal validation group and 0.8798 and 0.7801 in the external validation group. The SHAP revealed the top 20 predictors of postoperative AKI ranked according to the importance, and the top three features on prediction were the serum creatinine in the first 24 h after operation, the last preoperative Scr level, and body surface area.

CONCLUSION

This study provides a LightGBM predictive model that can make accurate predictions for AKI after CABG surgery. The LightGBM model shows good predictive ability in both internal and external validation. It can help cardiac surgeons identify high-risk patients who may experience AKI after CABG surgery.

Net Reclassification Index Statistics Do Not Help Assess New Risk Models

When evaluating a new risk factor for disease (eg, a measurement from imaging studies), many investigators examine its value above and beyond existing biomarkers and risk factors. They compare the performance of an "old" risk model using established predictors and a "new" risk model that adds the new factor. Net reclassification index (NRI) statistics are a family of metrics for comparing two risk models. NRI statistics became popular in some medical fields and have appeared in high-impact journals. This article reviews NRI statistics and describes several issues with them. Problems include unacceptable statistical behavior, incorrect statistical inferences, and lack of interpretability. NRI statistics are unhelpful (at best) and misleading (at worst).

Number of positive lymph nodes combined with the logarithmic ratio of positive lymph nodes predicts long-term survival for patients with node-positive parotid gland carcinoma after surgery: a SEER population-based study

PURPOSE

To evaluate the prognostic value of the number of positive lymph nodes (NPLN), the ratio of positive lymph nodes (pLNR), and the logarithmic ratio of positive lymph nodes (LODDS) in patients with parotid gland carcinoma. On this basis, establishing and validating an optimal nomogram.

METHODS

A total of 895 patients with T1-4N1-3M0 parotid gland carcinoma were included in our study from the Surveillance, Epidemiology, and End Results (SEER) database. Patients' data were randomly assigned to the training cohort and the validation cohort by a ratio of 7:3. Univariate and multivariate COX regression analysis were used to explore the relationship between the study factors and the prognosis of parotid gland carcinoma, including overall survival (OS) and cause-specific survival (CSS). The Akaike Information Criterion (AIC) was used to evaluate model fit. Harrell's concordance index (C-index), integrated discrimination improvement (IDI), and net reclassification index (NRI) were used to evaluate the predictive ability of these models. The decision curve analysis was used to evaluate the clinical benefit of the nomograms compared with the TNM stage.

RESULTS

NPLN, pLNR, and LODDS are independent risk factors for the prognostic of PGC. According to the AIC, C index, IDI, and NRI, the models combined with NPLN and LODDS were the best. The decision curves suggested that our nomograms had good predictive abilities for the prognosis of parotid gland carcinoma.

CONCLUSION

The two nomograms which contained NPLN and LODDS had the potential to predict OS and CSS in patients with parotid gland carcinoma.

Predicting risk of peripartum blood transfusion during vaginal and cesarean delivery: A risk prediction model

OBJECTIVE

The objective of this study is to develop a model that will help predict the risk of blood transfusion using information available prior to delivery.

STUDY DESIGN

The study is a secondary analysis of the Consortium on Safe Labor registry. Women who had a delivery from 2002 to 2008 were included. Pre-delivery variables that had significant associations with transfusion were included in a multivariable logistic regression model predicting transfusion. The prediction model was internally validated using randomly selected samples from the same population of women.

RESULTS

Of 156,572 deliveries, 5,463 deliveries (3.5%) required transfusion. Women who had deliveries requiring transfusion were more likely to have a number of comorbidities such as preeclampsia (6.3% versus 4.1%, OR 1.21, 95% CI 1.08-1.36), placenta previa (1.8% versus 0.4%, OR 4.11, 95% CI 3.25-5.21) and anemia (10.6% versus 5.4%, OR 1.30, 95% CI 1.21-1.41). Transfusion was least likely to occur in university teaching hospitals compared to community hospitals. The c statistic was 0.71 (95% CI 0.70-0.72) in the derivation sample. The most salient predictors of transfusion included type of hospital, placenta previa, multiple gestations, diabetes mellitus, anemia, asthma, previous births, preeclampsia, type of insurance, age, gestational age, and vertex presentation. The model was well-calibrated and showed strong internal validation.

CONCLUSION

The model identified independent risk factors that can help predict the risk of transfusion prior to delivery. If externally validated in another dataset, this model can assist health care professionals counsel patients and prepare facilities/resources to reduce maternal morbidity.

A machine learning-Based model to predict early death among bone metastatic breast cancer patients: A large cohort of 16,189 patients

Purpose: This study aims to develop a prediction model to categorize the risk of early death among breast cancer patients with bone metastases using machine learning models. Methods: This study examined 16,189 bone metastatic breast cancer patients between 2010 and 2019 from a large oncological database in the United States. The patients were divided into two groups at random in a 90:10 ratio. The majority of patients (n = 14,582, 90%) were served as the training group to train and optimize prediction models, whereas patients in the validation group (n = 1,607, 10%) were utilized to validate the prediction models. Four models were introduced in the study: the logistic regression model, gradient boosting tree model, decision tree model, and random forest model. Results: Early death accounted for 17.4% of all included patients. Multivariate analysis demonstrated that older age; a separated, divorced, or widowed marital status; nonmetropolitan counties; brain metastasis; liver metastasis; lung metastasis; and histologic type of unspecified neoplasms were significantly associated with more early death, whereas a lower grade, a positive estrogen receptor (ER) status, cancer-directed surgery, radiation, and chemotherapy were significantly the protective factors. For the purpose of developing prediction models, the 12 variables were used. Among all the four models, the gradient boosting tree had the greatest AUC [0.829, 95% confident interval (CI): 0.802-0.856], and the random forest (0.828, 95% CI: 0.801-0.855) and logistic regression (0.819, 95% CI: 0.791-0.847) models came in second and third, respectively. The discrimination slopes for the three models were 0.258, 0.223, and 0.240, respectively, and the corresponding accuracy rates were 0.801, 0.770, and 0.762, respectively. The Brier score of gradient boosting tree was the lowest (0.109), followed by the random forest (0.111) and logistic regression (0.112) models. Risk stratification showed that patients in the high-risk group (46.31%) had a greater six-fold chance of early death than those in the low-risk group (7.50%). Conclusion: The gradient boosting tree model demonstrates promising performance with favorable discrimination and calibration in the study, and this model can stratify the risk probability of early death among bone metastatic breast cancer patients.

A Comparison of Clinical Prognostic Indices in Elderly Patients with Diffuse Large B-Cell Lymphoma Treated with a Pegylated Liposomal Doxorubicin Combination Regimen in China

Background

There is no consensus regarding the risk stratification scores for elderly patients with diffuse large B-cell lymphoma (DLBCL). We aimed to compare the prognostic predictive ability of the current clinical scoring indices in DLBCL elderly patients treated with the R-CODP regimen (rituximab, cyclophosphamide, pegylated liposomal doxorubicin, vincristine, and prednisone).

Methods

We retrospectively collected the data of elderly DLBCL patients who received the R-CODP regimen as the first-line treatment. The efficacy of the regimen was evaluated. The Akaike information criteria (AIC), concordance index (C-index), and integrated discrimination improvement (IDI) were used to assess the fitness and prognostic performance of the current clinical prognostic indices.

Results

In the total of 158 patients enrolled, the median follow-up time was 6.7 years (95% CI: 6.3–7.9), and the 5-year OS was 52.8% (95% CI: 45.5%–61.2%). The International Prognostic Index (IPI), National Comprehensive Cancer Network-IPI (NCCN-IPI), and Elderly International Prognostic Index (E-IPI) were all significantly associated with OS (P < 0.001 for all). However, no significance was observed in 5-year OS in the low- vs low-intermediate-risk groups for IPI (P = 0.377), NCCN-IPI (P = 0.238), and E-IPI (P = 0.080). Compared with the IPI and NCCN-IPI, the E-IPI had the lowest AIC value of 747.5 and the highest C-index of 0.692. For predicting 5-year mortality, the E-IPI showed better performance (AUC: 0.715 for E-IPI vs 0.676 for IPI, P = 0.036), with the IDI of 6.29% (95% CI: 3.71%-8.88%, P < 0.001) and 4.80% (95% CI: 1.32%-8.28%, P = 0.007) compared to the IPI and NCCN-IPI, respectively.

Conclusion

The E-IPI might be a better prognostic prediction model in Chinese DLBCL generics treated with R-CODP for predicting 5-year mortality. However, the IPI, NCCN-IPI, and E-IPI did not seem to be able to distinguish patients with a favorable prognosis well.

Predictive Ability of European Heart Surgery Risk Assessment System II (EuroSCORE II) and the Society of Thoracic Surgeons (STS) Score for in-Hospital and Medium-Term Mortality of Patients Undergoing Coronary Artery Bypass Grafting

Objective

To evaluate the powers of European Heart Surgery Risk Assessment System II (EuroSCORE II) and the Society of Thoracic Surgeons (STS) score in predicting in-hospital and medium-term mortality of patients undergoing coronary artery bypass grafting (CABG).

Methods

Totally 1628 Chinese patients were included between January 2000 and January 2018. Their perioperative clinical data were collected and the patients were closely followed up. According to the length of follow-up time, the total cohort was divided into 1-year, 2-year, 3-year, 4-year and 5-year groups. The in-hospital and medium-term risk prediction of EuroSCORE II and STS score were comparatively assessed by calibration, discrimination, decision curve analysis (DCA), net reclassification index (NRI), integrated discrimination improvement (IDI) and Bland-Altman analysis.

Results

About 36 (2.21%) patients died during hospitalization. Both EuroSCORE II and STS score performed extremely well in predicting in-hospital mortality (area under curve = 0.900 and 0.879, respectively). However, calibration and discrimination analyses showed gradual decrease when these two risk evaluation systems were used to predict mortality during the follow-up period. At the same time, the predictive ability of EuroSCORE II was better than STS score. DCA curves showed that the performances of the two evaluation systems were roughly equal between the threshold probability of 0% to 20%. The percentage of correct reclassification of EuroSCORE II was 21.64% higher than that of STS score in predicting 2-year postoperative mortality. The IDI index showed that the predictive capabilities of these two systems were roughly equivalent. Bland-Altman analysis showed no significant difference between the values of the two systems.

Conclusion

EuroSCORE II and STS score have excellent predictive powers in predicting in-hospital mortality of patients undergoing CABG. In particular, EuroSCORE II is superior in calibration and discrimination. The prediction efficiency of the two risk evaluation systems is still acceptable for two-year postoperative mortality, but decreases year by year.

Radiomics model based on shear-wave elastography in the assessment of axillary lymph node status in early-stage breast cancer

OBJECTIVES

To develop and validate an ultrasound elastography radiomics nomogram for preoperative evaluation of the axillary lymph node (ALN) burden in early-stage breast cancer.

METHODS

Data of 303 patients from hospital #1 (training cohort) and 130 cases from hospital #2 (external validation cohort) between Jun 2016 and May 2019 were enrolled. Radiomics features were extracted from shear-wave elastography (SWE) and corresponding B-mode ultrasound (BMUS) images. The minimum redundancy maximum relevance and least absolute shrinkage and selection operator algorithms were used to select ALN status-related features. Proportional odds ordinal logistic regression was performed using the radiomics signature together with clinical data, and an ordinal nomogram was subsequently developed. We evaluated its performance using C-index and calibration.

RESULTS

SWE signature, US-reported LN status, and molecular subtype were independent risk factors associated with ALN status. The nomogram based on these variables showed good discrimination in the training (overall C-index: 0.842; 95%CI, 0.773-0.879) and the validation set (overall C-index: 0.822; 95%CI, 0.765-0.838). For discriminating between disease-free axilla (N0) and any axillary metastasis (N + (≥ 1)), it achieved a C-index of 0.845 (95%CI, 0.777-0.914) for the training cohort and 0.817 (95%CI, 0.769-0.865) for the validation cohort. The tool could also discriminate between low (N + (1-2)) and heavy metastatic ALN burden (N + (≥ 3)), with a C-index of 0.827 (95%CI, 0.742-0.913) in the training cohort and 0.810 (95%CI, 0.755-0.864) in the validation cohort.

CONCLUSION

The radiomics model shows favourable predictive ability for ALN staging in patients with early-stage breast cancer, which could provide incremental information for decision-making.

KEY POINTS

• Radiomics analysis helps radiologists to evaluate the axillary lymph node status of breast cancer with accuracy. • This multicentre retrospective study showed that radiomics nomogram based on shear-wave elastography provides incremental information for risk stratification. • Treatment can be given with more precision based on the model.

Intratumoral and peritumoral radiomics analysis for preoperative Lauren classification in gastric cancer

Background

Preoperative prediction of the Lauren classification in gastric cancer (GC) is very important to the choice of therapy, the evaluation of prognosis, and the improvement of quality of life. However, there is not yet radiomics analysis concerning the prediction of Lauren classification straightly. In this study, a radiomic nomogram was developed to preoperatively differentiate Lauren diffuse type from intestinal type in GC.

Methods

A total of 539 GC patients were enrolled in this study and later randomly allocated to two cohorts at a 7:3 ratio for training and validation. Two sets of radiomic features were derived from tumor regions and peritumor regions on venous phase computed tomography (CT) images, respectively. With the least absolute shrinkage and selection operator logistic regression, a combined radiomic signature was constructed. Also, a tumor-based model and a peripheral ring-based model were built for comparison. Afterwards, a radiomic nomogram integrating the combined radiomic signature and clinical characteristics was developed. All the models were evaluated regarding classification ability and clinical usefulness.

Results

The combined radiomic signature achieved an area under receiver operating characteristic curve (AUC) of 0.715 (95% confidence interval [CI], 0.663–0.767) in the training cohort and 0.714 (95% CI, 0.636–0.792) in the validation cohort. The radiomic nomogram incorporating the combined radiomic signature, age, CT T stage, and CT N stage outperformed the other models with a training AUC of 0.745 (95% CI, 0.696–0.795) and a validation AUC of 0.758 (95% CI, 0.685–0.831). The significantly improved sensitivity of radiomic nomogram (0.765 and 0.793) indicated better identification of diffuse type GC patients. Further, calibration curves and decision curves demonstrated its great model fitness and clinical usefulness.

Conclusions

The radiomic nomogram involving the combined radiomic signature and clinical characteristics holds potential in differentiating Lauren diffuse type from intestinal type for reasonable clinical treatment strategy.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40644-020-00358-3.

TRIPOD statement: a preliminary pre-post analysis of reporting and methods of prediction models

OBJECTIVES

To assess the difference in completeness of reporting and methodological conduct of published prediction models before and after publication of the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.

METHODS

In the seven general medicine journals with the highest impact factor, we compared the completeness of the reporting and the quality of the methodology of prediction model studies published between 2012 and 2014 (pre-TRIPOD) with studies published between 2016 and 2017 (post-TRIPOD). For articles published in the post-TRIPOD period, we examined whether there was improved reporting for articles (1) citing the TRIPOD statement, and (2) published in journals that published the TRIPOD statement.

RESULTS

A total of 70 articles was included (pre-TRIPOD: 32, post-TRIPOD: 38). No improvement was seen for the overall percentage of reported items after the publication of the TRIPOD statement (pre-TRIPOD 74%, post-TRIPOD 76%, 95% CI of absolute difference: -4% to 7%). For the individual TRIPOD items, an improvement was seen for 16 (44%) items, while 3 (8%) items showed no improvement and 17 (47%) items showed a deterioration. Post-TRIPOD, there was no improved reporting for articles citing the TRIPOD statement, nor for articles published in journals that published the TRIPOD statement. The methodological quality improved in the post-TRIPOD period. More models were externally validated in the same article (absolute difference 8%, post-TRIPOD: 39%), used measures of calibration (21%, post-TRIPOD: 87%) and discrimination (9%, post-TRIPOD: 100%), and used multiple imputation for handling missing data (12%, post-TRIPOD: 50%).

CONCLUSIONS

Since the publication of the TRIPOD statement, some reporting and methodological aspects have improved. Prediction models are still often poorly developed and validated and many aspects remain poorly reported, hindering optimal clinical application of these models. Long-term effects of the TRIPOD statement publication should be evaluated in future studies.

Body mass index versus bioelectrical impedance analysis for classifying physical function impairment in a racially diverse cohort of midlife women: the Study of Women's Health Across the Nation (SWAN)

BACKGROUND

Body composition strongly influences physical function in older adults. Bioelectrical impedance analysis (BIA) differentiates fat mass from skeletal muscle mass, and may be more useful than body mass index (BMI) for classifying women on their likelihood of physical function impairment.

AIMS

This study tested whether BIA-derived estimates of percentage body fat (%BF) and height-normalized skeletal muscle mass (skeletal muscle mass index; SMI) enhance classification of physical function impairment relative to BMI.

METHOD

Black, White, Chinese, and Japanese midlife women (N = 1482) in the Study of Women's Health Across the Nation (SWAN) completed performance-based measures of physical function. BMI (kg/m²) was calculated. %BF and SMI were derived through BIA. Receiver-operating characteristic (ROC) curve analysis, conducted in the overall sample and stratified by racial group, evaluated optimal cutpoints of BMI, %BF, and SMI for classifying women on moderate-severe physical function impairment.

RESULTS

In the overall sample, a BMI cutpoint of ≥ 30.1 kg/m² correctly classified 71.1% of women on physical function impairment, and optimal cutpoints for %BF (≥ 43.4%) and SMI (≥ 8.1 kg/m²) correctly classified 69% and 62% of women, respectively. SMI did not meaningfully enhanced classification relative to BMI (change in area under the ROC curve = 0.002; net reclassification improvement = 0.021; integrated discrimination improvement = - 0.003). Optimal cutpoints for BMI, %BF, and SMI varied substantially across race. Among Black women, a %BF cutpoint of 43.9% performed somewhat better than BMI (change in area under the ROC curve = 0.017; sensitivity = 0.69, specificity = 0.64).

CONCLUSION

Some race-specific BMI and %BF cutpoints have moderate utility for identifying impaired physical function among midlife women.

Estimation of Atherosclerotic Cardiovascular Disease Risk Among Patients in the Veterans Affairs Health Care System

IMPORTANCE

Current guidelines recommend statin therapy for millions of US residents for the primary prevention of atherosclerotic cardiovascular disease (ASCVD). It is unclear whether traditional prediction models that do not account for current widespread statin use are sufficient for risk assessment.

OBJECTIVES

To examine the performance of the Pooled Cohort Equations (PCE) for 5-year ASCVD risk estimation in a contemporary cohort and to test the hypothesis that inclusion of statin therapy improves model performance.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study included adult patients in the Veterans Affairs health care system without baseline ASCVD. Using national electronic health record data, 3 Cox proportional hazards models were developed to estimate 5-year ASCVD risk, as follows: the variables and published β coefficients from the PCE (model 1), the PCE variables with cohort-derived β coefficients (model 2), and model 2 plus baseline statin use (model 3). Data were collected from January 2002 to December 2012 and analyzed from June 2016 to March 2020.

EXPOSURES

Traditional ASCVD risk factors from the PCE plus baseline statin use.

MAIN OUTCOMES AND MEASURES

Incident ASCVD and ASCVD mortality.

RESULTS

Of 1 672 336 patients in the cohort (mean [SD] baseline age 58.0 [13.8] years, 1 575 163 [94.2%] men, 1 383 993 [82.8%] white), 312 155 (18.7%) were receiving statin therapy at baseline. During 5 years of follow-up, 66 605 (4.0%) experienced an ASCVD event, and 31 878 (1.9%) experienced ASCVD death. Compared with the original PCE, the cohort-derived model did not improve model discrimination in any of the 4 age-sex strata but did improve model calibration. The PCE overestimated ASCVD risk compared with the cohort-derived model; 211 237 of 1 136 161 white men (18.6%), 29 634 of 218 463 black men (13.6%), 1741 of 44 399 white women (3.9%), and 836 of 16 034 black women (5.2%) would be potentially eligible for statin therapy under the PCE but not the cohort-derived model. When added to the cohort-derived model, baseline statin therapy was associated with a 7% (95% CI, 5%-9%) lower relative risk of ASCVD and a 25% (95% CI, 23%-28%) lower relative risk for ASCVD death.

CONCLUSIONS AND RELEVANCE

In this study, lower than expected rates of incident ASCVD events in a contemporary national cohort were observed. The PCE overestimated ASCVD risk, and more than 15% of patients would be potentially eligible for statin therapy based on the PCE but not on a cohort-derived model. In the statin era, health care professionals and systems should base ASCVD risk assessment on models calibrated to their patient populations.

Nomogram For The Prediction Of Malignancy In Small (8-20 mm) Indeterminate Solid Solitary Pulmonary Nodules In Chinese Populations

Purpose

This study aimed to develop and validate a nomogram for predicting the malignancy of small (8–20 mm) solid indeterminate solitary pulmonary nodules (SPNs) in a Chinese population by using routine clinical and computed tomography data.

Methods

The prediction model was developed using a retrospective cohort that comprised 493 consecutive patients with small indeterminate SPNs who were treated between December 2012 and December 2016. The model was independently validated using a second retrospective cohort comprising 216 consecutive patients treated between January 2017 and May 2018. The investigated variables included patient characteristics (e.g., age and smoking history), nodule parameters (e.g., marginal spiculation and significant enhancement), and tumor biomarker levels (e.g., carcinoembryonic antigen). A prediction model was developed by using multivariable logistic regression analysis, and the model’s performance was presented as a nomogram. The model was evaluated based on its discriminative ability, calibration, and clinical usefulness.

Results

The developed nomogram was ultimately based on age, marginal spiculation, significant enhancement, and pleural indentation. The Harrell concordance index values were 0.869 in the training cohort (95% confidence interval: 0.837–0.901) and 0.847 in the validation cohort (95% confidence interval: 0.792–0.902). The Hosmer-Lemeshow test revealed good calibration in each of the training and validation cohorts. Decision curve analysis confirmed that the nomogram was clinically useful (risk threshold from 0.10 to 0.85).

Conclusion

Patient age, marginal spiculation, significant enhancement, and pleural indentation are independent predictors of malignancy in small indeterminate solid SPNs. The developed nomogram is easy-to-use and may allow the accurate prediction of malignancy in small indeterminate solid SPNs among Chinese patients.

The power-integrated discriminant improvement: An accurate measure of the incremental predictive value of additional biomarkers

The predictive performance of biomarkers is a central concern in biomedical research. This is often evaluated by comparing two statistical models: a "new" model incorporating additional biomarkers and an "old" model without them. In 2008, the integrated discrimination improvement (IDI) was proposed for cases when the response variable is binary, and it is now widely applied as a promising alternative to conventional measures, such as the difference of the area under the receiver operating characteristic curve. However, the IDI can erroneously identify a significant improvement in the new model even if no additional information has been provided by new biomarkers. In order to overcome problems with existing measures, in this study, we propose the power-IDI as a measure of incremental predictive value. Our study explains why the IDI cannot avoid false detection of apparent improvements in a new model and we show that our proposed measure is better able to capture improvements in prediction. Numerical simulations and examples using real empirical data reveal that the power-IDI is not only more powerful but also incurs fewer false detections of improvement.

Measures for evaluation of prognostic improvement under multivariate normality for nested and nonnested models

When comparing performances of two risk prediction models, several metrics exist to quantify prognostic improvement, including the change in the area under the Receiver Operating Characteristic curve, the Integrated Discrimination Improvement, the Net Reclassification Index at event rate, the change in Standardized Net Benefit, the change in Brier score, and the change in scaled Brier score. We explore the behavior and interrelationships between these metrics under multivariate normality in nested and nonnested model comparisons. We demonstrate that, within the framework of linear discriminant analysis, all six statistics are functions of squared Mahalanobis distance, a robust metric that properly measures discrimination by quantifying the separation between the risk scores of events and nonevents. These relationships are important for overall interpretability and clinical usefulness. Through simulation, we demonstrate that the performance of the theoretical estimators under normality is comparable or superior to empirical estimation methods typically used by investigators. In particular, the theoretical estimators for the Net Reclassification Index and the change in Standardized Net Benefit exhibit less variability in their estimates as compared to their empirically estimated counterparts. Finally, we explore how these metrics behave with potentially nonnormal data by applying these methods in a practical example based on the sex-specific cardiovascular disease risk models from the Framingham Heart Study. Our findings aim to give greater insight into the behavior of these measures and the connections existing among them and to provide additional estimation methods with less variability for the Net Reclassification Index and the change in Standardized Net Benefit.

A comparison of statistical learning methods for deriving determining factors of accident occurrence from an imbalanced high resolution dataset

One of the main aims of accident data analysis is to derive the determining factors associated with road traffic accident occurrence. While current studies mainly use variants of count data regression to achieve this aim, the problem can also be considered as a binary classification task, with the dichotomous target variable indicating events (accidents) and non-events (no accidents). The effects of 45 variables - describing road condition and geometry, traffic volume and regulations, weather, and accident time - are analyzed using a dataset in high temporal (1 h) and spatial (250 m) resolution, covering the whole highway network of Austria over the period of four consecutive years. A combination of synthetic minority oversampling and maximum dissimilarity undersampling is used to balance the training dataset. We employ and compare a series of statistical learning techniques with respect to their predictive performance and discuss the importance of determining factors of accident occurrence from the ensemble of models. Findings substantiate that a trade-off between accuracy and sensitivity is inherent to imbalanced classification problems. Results show satisfying performance of tree-based methods which exhibit accuracies between 75% and 90% while exhibiting sensitivities between 30% and 50%. Overall, this analysis emphasizes the merits of using high-resolution data in the context of accident analysis.

Early Identification of Preterm Neonates at Birth With a Tablet App for the Simplified Gestational Age Score (T-SGAS) When Ultrasound Gestational Age Dating Is Unavailable: Protocol for a Validation Study

Background

Although rates of preterm birth continue to increase globally, identification of preterm from low birth weight infants remains a challenge. The burden of low birth weight vs preterm is greatest in resource-limited settings, where gestational age (GA) prior to delivery is frequently not known because ultrasound in early pregnancy is not available and estimates of the date of the mother’s last menstrual period (LMP) may not be reliable. An alternative option is to assess GA at birth to optimize referral and care of preterm newborns. We previously developed and pilot-tested a system to measure the simplified gestational age score (SGAS) based on 4 easily observable neonatal characteristics.

Objective

The objective of this study is to adapt the scoring system as a tablet app (potentially scalable approach) to assess feasibility of use and to validate whether the scoring system accurately predicts prematurity by itself, over and above birth weight in a large sample of newborns.

Methods

The study is based in Nagpur, India, at the Research Unit of the National Institute of Child Health and Human Development’s Global Network for Women’s and Children’s Health Research. The Android tablet app for the SGAS (T-SGAS) displays de-identified photographs of skin, breasts, and genitalia across a range of GAs and line drawings of infant posture. Each item is associated with a score. The user is trained to choose the photograph or line drawing that most closely matches the newborn being evaluated, and the app determines the neonate’s GA category (preterm or term) from the cumulative score. The validation study will be conducted in 3 second level care facilities (most deliveries in India occur in hospitals, and women known to be at risk of preterm birth are referred to second level care facilities). Within 24 hours of delivery, women and their babies who are stable will be enrolled in the study. Two auxiliary nurse midwives (ANMs) blinded to prior GA assessments will use the T-SGAS to estimate the GA status of the newborn. An independent data collector will abstract the GA from the ultrasound recorded in the hospital chart and record the date of the mother’s LMP. Eligibility for analysis is determined by the ultrasound and LMP data being collected within 1 week of each other to have a rigorous assessment of true GA.

Results

Publication of the results of the study is anticipated in 2019.

Conclusions

Until GA dating by ultrasound is universally available and easy to use in resource-limited settings, and where there are restrictions on ultrasound use due to their use for sex determination and abortion of female fetuses, this study will determine whether the T-SGAS app can accurately assess GA in risk categories at birth.

Trial Registration

ClinicalTrials.gov NCT02408783; https://clinicaltrials.gov/ct2/show/NCT02408783 (Archived by Webcite at http://www.webcitation.org/75S2kmr3T)

International Registered Report Identifier (IRRID)

DERR1-10.2196/11913

Relation of Serum and Urine Renal Biomarkers to Cardiovascular Risk in Patients with Type 2 Diabetes Mellitus and Recent Acute Coronary Syndromes (From the EXAMINE Trial)

A deeper understanding of the interplay between the renal axis and cardiovascular (CV) disease is needed in type 2 diabetes mellitus (T2DM). We aimed to explore the prognostic value of a comprehensive panel of renal biomarkers in patients with T2DM at high CV risk. We evaluated the prognostic performance of both serum (Cystatin C) and urine renal biomarkers (neutrophil gelatinase-associated lipocalin, kidney injury molecule-1 protein, and indices of urinary protein excretion) in 5,380 patients with T2DM and recent acute coronary syndromes in the EXAMINE trial. Patients requiring dialysis within 14 days were excluded. Single- and multimarker covariate-adjusted Cox proportional hazards models were developed to predict times to events. Primary endpoint was composite nonfatal myocardial infarction, nonfatal stroke, or CV death. Median age was 61 years, 68% were men, and mean baseline estimated glomerular filtration rate (eGFR) was 74 mL/min/1.73 m². During median follow-up of 18 months, 621 (11.5%) experienced the primary endpoint and 326 (6.1%) patients had died. All renal biomarkers were robustly associated with adverse CV events in step-wise fashion, independent of baseline eGFR. However, in the multimarker prediction model, only Cystatin C (per 1 SD) was associated with the primary endpoint (hazard ratio [HR] 1.28 [1.14 to 1.45]; p ≤ 0.001), death (HR 1.51 [1.30 to 1.74]; p ≤ 0.001), and heart failure hospitalization (HR 1.20 [0.96 to 1.49]; p = 0.11). Association between Cystatin C and the primary endpoint was similar in baseline eGFR above and below 60 mL/min/1.73 m² (P_interaction > 0.05). In conclusion, serum and urine renal biomarkers, when tested alone, independently predict long-term adverse CV events in high-risk patients with T2DM. In an integrative panel of renal biomarkers, only serum Cystatin C remained independently associated with subsequent CV risk. Renal biomarkers informing various aspects of kidney function may further our understanding of the complex interplay between diabetic kidney disease and CV disease.

First things first: risk model performance metrics should reflect the clinical application

Developing new measures of risk model performance is an active line of research, often motivated by the conventional wisdom that area under the ROC curve is an 'insensitive' measure of the additional predictive capacity offered by new biomarkers. Without endorsing area under the ROC curve, we argue that this charge is not substantiated. Three articles in this issue discuss alternative metrics of risk model performance: NRI(p) (two-category net reclassification index at the event rate), integrated discrimination index, and R-squared statistics. Guided by the principle that performance metrics should match the intended use of a risk prediction model, we argue that routine use of these indices is not justified. Instead, we recommend decision-theoretic measures to evaluate risk prediction models for applications in which clinically relevant risk thresholds have been established for classifying individuals. In the absence of established risk thresholds, additional research is needed to develop suitable metrics. Copyright © 2017 John Wiley & Sons, Ltd.

Clinical risk reclassification at 10 years

Three papers in this issue focus on the role of calibration in model fit statistics, including the net reclassification improvement (NRI) and integrated discrimination improvement (IDI). This commentary reviews the development of such reclassification statistics along with more recent advances in our understanding of these measures. We show how the two-category NRI and the IDI are affected by changes in the event rate in theory and in an applied example. We also describe the role of calibration and how it may be assessed. Finally, we discuss the relevance of the event rate NRI for clinical use. Copyright © 2017 John Wiley & Sons, Ltd.

Kidney Biomarkers and Decline in eGFR in Patients with Type 2 Diabetes

BACKGROUND AND OBJECTIVES

Biomarkers may improve identification of individuals at risk of eGFR decline who may benefit from intervention or dialysis planning. However, available biomarkers remain incompletely validated for risk stratification and prediction modeling.

DESIGN, SETTING, PARTICIPANTS, & MEASUREMENTS

We examined serum cystatin C, urinary kidney injury molecule-1 (uKIM-1), and urinary neutrophil gelatinase-associated lipocalin (UNGAL) in 5367 individuals with type 2 diabetes mellitus and recent acute coronary syndromes enrolled in the Examination of Cardiovascular Outcomes with Alogliptin versus Standard of Care (EXAMINE) trial. Baseline concentrations and 6-month changes in biomarkers were also evaluated. Cox proportional regression was used to assess associations with a 50% decrease in eGFR, stage 5 CKD (eGFR<15 ml/min per 1.73 m2), or dialysis.

RESULTS

eGFR decline occurred in 98 patients (1.8%) over a median of 1.5 years. All biomarkers individually were associated with higher risk of eGFR decline (P<0.001). However, when adjusting for baseline eGFR, proteinuria, and clinical factors, only baseline cystatin C (adjusted hazard ratio per 1 SD change, 1.66; 95% confidence interval, 1.41 to 1.96; P<0.001) and 6-month change in urinary neutrophil gelatinase-associated lipocalin (adjusted hazard ratio per 1 SD change, 1.07; 95% confidence interval, 1.02 to 1.12; P=0.004) independently associated with CKD progression. A base model for predicting kidney function decline with nine standard risk factors had strong discriminative ability (C-statistic 0.93). The addition of baseline cystatin C improved discrimination (C-statistic 0.94), but it failed to reclassify risk categories of individuals with and without eGFR decline.

CONCLUSIONS

The addition of cystatin C or biomarkers of tubular injury did not meaningfully improve the prediction of eGFR decline beyond common clinical factors and routine laboratory data in a large cohort of patients with type 2 diabetes and recent acute coronary syndrome.

PODCAST

This article contains a podcast at https://www.asn-online.org/media/podcast/CJASN/2018_01_16_CJASNPodcast_18_3_G.mp3.

Quantifying the added value of new biomarkers: how and how not

Over the past few decades, interest in biomarkers to enhance predictive modeling has soared. Methodology for evaluating these has also been an active area of research. There are now several performance measures available for quantifying the added value of biomarkers. This commentary provides an overview of methods currently used to evaluate new biomarkers, describes their strengths and limitations, and offers some suggestions on their use.

Asymptotic distribution of ∆AUC, NRIs, and IDI based on theory of U-statistics

The change in area under the curve (∆AUC), the integrated discrimination improvement (IDI), and net reclassification index (NRI) are commonly used measures of risk prediction model performance. Some authors have reported good validity of associated methods of estimating their standard errors (SE) and construction of confidence intervals, whereas others have questioned their performance. To address these issues, we unite the ∆AUC, IDI, and three versions of the NRI under the umbrella of the U-statistics family. We rigorously show that the asymptotic behavior of ∆AUC, NRIs, and IDI fits the asymptotic distribution theory developed for U-statistics. We prove that the ∆AUC, NRIs, and IDI are asymptotically normal, unless they compare nested models under the null hypothesis. In the latter case, asymptotic normality and existing SE estimates cannot be applied to ∆AUC, NRIs, or IDI. In the former case, SE formulas proposed in the literature are equivalent to SE formulas obtained from U-statistics theory if we ignore adjustment for estimated parameters. We use Sukhatme-Randles-deWet condition to determine when adjustment for estimated parameters is necessary. We show that adjustment is not necessary for SEs of the ∆AUC and two versions of the NRI when added predictor variables are significant and normally distributed. The SEs of the IDI and three-category NRI should always be adjusted for estimated parameters. These results allow us to define when existing formulas for SE estimates can be used and when resampling methods such as the bootstrap should be used instead when comparing nested models. We also use the U-statistic theory to develop a new SE estimate of ∆AUC. Copyright © 2017 John Wiley & Sons, Ltd.

Genetic Drivers of von Willebrand Factor Levels in an Ischemic Stroke Population and Association With Risk for Recurrent Stroke

BACKGROUND AND PURPOSE

von Willebrand factor (vWF) plays an important role in thrombus formation during cerebrovascular damage. We sought to investigate the potential role of circulating vWF in recurrent cerebrovascular events and identify genetic contributors to variation in vWF level in an ischemic stroke population.

METHODS

We analyzed the effect of circulating vWF on risk of recurrent stroke using survival models in the VISP trial (Vitamin Intervention for Stroke Prevention) and the use of vWF in reclassification over traditional factors. We conducted a genome-wide association study) with imputation, based on 1000 Genomes Project data, for circulating vWF levels and then interrogated loci previously associated with vWF levels. We performed expression quantitative trait locus analysis for vWF across different tissues.

RESULTS

Elevated vWF levels were associated with increased risk for recurrent stroke in VISP. Adding vWF to traditional clinical parameters also improved recurrent stroke risk prediction. We identified single-nucleotide polymorphisms significantly associated with circulating vWF at the ABO locus (P<5×10^-8) and replicated findings from previous genetic associations of vWF levels in humans. Expression quantitative trait locus analyses demonstrate that most associated ABO single-nucleotide polymorphisms were also associated with vWF gene expression.

CONCLUSIONS

Elevated vWF levels are associated with recurrent stroke in VISP. In the VISP population, genetic determinants of vWF levels that impact vWF gene expression were identified. These data add to our knowledge of the pathophysiologic and genetic basis for recurrent stroke risk and may have implications for clinical care decision making.