Review and evaluation of performance measures for survival prediction models in external validation settings

The natural course of binge-eating disorder: findings from a prospective, community-based study of adults

BACKGROUND

Epidemiological data offer conflicting views of the natural course of binge-eating disorder (BED), with large retrospective studies suggesting a protracted course and small prospective studies suggesting a briefer duration. We thus examined changes in BED diagnostic status in a prospective, community-based study that was larger and more representative with respect to sex, age of onset, and body mass index (BMI) than prior multi-year prospective studies.

METHODS

Probands and relatives with current DSM-IV BED (n = 156) from a family study of BED ('baseline') were selected for follow-up at 2.5 and 5 years. Probands were required to have BMI > 25 (women) or >27 (men). Diagnostic interviews and questionnaires were administered at all timepoints.

RESULTS

Of participants with follow-up data (n = 137), 78.1% were female, and 11.7% and 88.3% reported identifying as Black and White, respectively. At baseline, their mean age was 47.2 years, and mean BMI was 36.1. At 2.5 (and 5) years, 61.3% (45.7%), 23.4% (32.6%), and 15.3% (21.7%) of assessed participants exhibited full, sub-threshold, and no BED, respectively. No participants displayed anorexia or bulimia nervosa at follow-up timepoints. Median time to remission (i.e. no BED) exceeded 60 months, and median time to relapse (i.e. sub-threshold or full BED) after remission was 30 months. Two classes of machine learning methods did not consistently outperform random guessing at predicting time to remission from baseline demographic and clinical variables.

CONCLUSIONS

Among community-based adults with higher BMI, BED improves with time, but full remission often takes many years, and relapse is common.

External Validation of a Risk Score Model for Predicting Major Clinical Events in Adults After Atrial Switch

BACKGROUND

A risk model has been proposed to provide a patient individualized estimation of risk for major clinical events (heart failure events, ventricular arrhythmia, all-cause mortality) in patients with transposition of the great arteries and atrial switch surgery. We aimed to externally validate the model.

METHODS AND RESULTS

A retrospective, multicentric, longitudinal cohort of 417 patients with transposition of the great arteries (median age, 24 years at baseline [interquartile range, 18-30]; 63% men) independent of the model development and internal validation cohort was studied. The performance of the prediction model in predicting risk at 5 years was assessed, and additional predictors of major clinical events were evaluated separately in our cohort. Twenty-five patients (5.9%) met the major clinical events end point within 5 years. Model validation showed good discrimination between high and low 5-year risk patients (Harrell C index of 0.73 [95% CI, 0.65-0.81]) but tended to overestimate this risk (calibration slope of 0.20 [95% CI, 0.03-0.36]). In our population, the strongest independent predictors of major clinical events were a history of heart failure and at least mild impairment of the subpulmonary left ventricle function.

CONCLUSIONS

We reported the first external validation of a major clinical events risk model in a large cohort of adults with transposition of the great arteries. The model allows for distinguishing patients at low risk from those at intermediate to high risk. Previous episode of heart failure and subpulmonary left ventricle dysfunction appear to be key markers in the prognosis of patients. Further optimizing risk models are needed to individualize risk predictions in patients with transposition of the great arteries.

A dedicated risk prediction model of 1-year mortality following endovascular aortic aneurysm repair involving the renal-mesenteric arteries

OBJECTIVE

Treatment goals of prophylactic endovascular aortic repair of complex aneurysms involving the renal-mesenteric arteries (complex endovascular aortic repair [cEVAR]) include achieving both technical success and long-term survival benefit. Mortality within the first year after cEVAR likely indicates treatment failure owing to associated costs and procedural complexity. Notably, no validated clinical decision aid tools exist that reliably predict mortality after cEVAR. The purpose of this study was to derive and validate a preoperative prediction model of 1-year mortality after elective cEVAR.

METHODS

All elective cEVARs including fenestrated, branched, and/or chimney procedures for aortic disease extent confined proximally to Ishimaru landing zones 6 to 9 in the Society for Vascular Surgery Vascular Quality Initiative were identified (January 2012 to August 2023). Patients (n = 4053) were randomly divided into training (n = 3039) and validation (n = 1014) datasets. A logistic regression model for 1-year mortality was created and internally validated by bootstrapping the AUC and calibration intercept and slope, and by using the model to predict 1-year mortality in the validation dataset. Independent predictors were assigned an integer score, based on model beta-coefficients, to generate a simplified scoring system to categorize patient risk.

RESULTS

The overall crude 1-year mortality rate after elective cEVAR was 11.3% (n = 456/4053). Independent preoperative predictors of 1-year mortality included chronic obstructive pulmonary disease, chronic renal insufficiency (creatinine >1.8 mg/dL or dialysis dependence), hemoglobin <12 g/dL, decreasing body mass index, congestive heart failure, increasing age, American Society of Anesthesiologists class ≥IV, current tobacco use, history of peripheral vascular intervention, and increasing extent of aortic disease. The 1-year mortality rate varied from 4% among the 23% of patients classified as low risk to 23% for the 24% classified as high risk. Performance of the model in validation was comparable with performance in the training data. The internally validated scoring system classified patients roughly into quartiles of risk (low, low/medium, medium/high and high), with 52% of patients categorized as medium/high to high risk, which had corresponding 1-year mortality rates of 11% and 23%, respectively. Aneurysm diameter was below Society for Vascular Surgery recommended treatment thresholds (<5.0 cm in females, <5.5 cm in males) in 17% of patients (n = 679/3961), 41% of whom were categorized as medium/high or high risk. This subgroup had significantly increased in-hospital complication rates (18% vs 12%; P = .02) and 1-year mortality (13% vs 5%; P < .0001) compared with patients in the low- or low/medium-risk groups with guideline-compliant aneurysm diameters (≥5.0 cm in females, ≥5.5 cm in males).

CONCLUSIONS

This validated preoperative prediction model for 1-year mortality after cEVAR incorporates physiological, functional, and anatomical variables. This novel and simplified scoring system can effectively discriminate mortality risk and, when applied prospectively, may facilitate improved preoperative decision-making, complex aneurysm care delivery, and resource allocation.

Personalised decision making to predict absolute metastatic risk in cutaneous squamous cell carcinoma: development and validation of a clinico-pathological model

Background

Cutaneous squamous cell carcinoma (cSCC) is a common skin cancer, affecting more than 2 million people worldwide yearly and metastasising in 2-5% of patients. However, current clinical staging systems do not provide estimates of absolute metastatic risk, hence missing the opportunity for more personalised treatment advice. We aimed to develop a clinico-pathological model that predicts the probability of metastasis in patients with cSCC.

Methods

Nationwide cohorts from (1) all patients with a first primary cSCC in The Netherlands in 2007-2008 and (2) all patients with a cSCC in 2013-2015 in England were used to derive nested case-control cohorts. Pathology records of primary cSCCs that originated a loco-regional or distant metastasis were identified, and these cSCCs were matched to primary cSCCs of controls without metastasis (1:1 ratio). The model was developed on the Dutch cohort (n = 390) using a weighted Cox regression model with backward selection and validated on the English cohort (n = 696). Model performance was assessed using weighted versions of the C-index, calibration metrics, and decision curve analysis; and compared to the Brigham and Women's Hospital (BWH) and the American Joint Committee on Cancer (AJCC) staging systems. Members of the multidisciplinary Skin Cancer Outcomes (SCOUT) consortium were surveyed to interpret metastatic risk cutoffs in a clinical context.

Findings

Eight out of eleven clinico-pathological variables were selected. The model showed good discriminative ability, with an optimism-corrected C-index of 0.80 (95% Confidence interval (CI) 0.75-0.85) in the development cohort and a C-index of 0.84 (95% CI 0.81-0.87) in the validation cohort. Model predictions were well-calibrated: the calibration slope was 0.96 (95% CI 0.76-1.16) in the validation cohort. Decision curve analysis showed improved net benefit compared to current staging systems, particularly for thresholds relevant for decisions on follow-up and adjuvant treatment. The model is available as an online web-based calculator (https://emc-dermatology.shinyapps.io/cscc-abs-met-risk/).

Interpretation

This validated model assigns personalised metastatic risk predictions to patients with cSCC, using routinely reported histological and patient-specific risk factors. The model can empower clinicians and healthcare systems in identifying patients with high-risk cSCC and offering personalised care/treatment and follow-up. Use of the model for clinical decision-making in different patient populations must be further investigated.

Funding

PPP Allowance made available by Health-Holland, Top Sector Life Sciences & Health, to stimulate public-private partnerships.

Enhanced liver Fibrosis® test predicts liver-related outcomes in the general population

Background & Aims

The Enhanced Liver Fibrosis® (ELF) test exhibits good discriminative performance in detecting advanced liver fibrosis and in predicting liver-related outcomes in patients with specific liver diseases, but large population-based studies are missing. We analysed the predictive performance of the ELF test in a general population cohort.

Methods

Data were sourced from the Health 2000 study, a Finnish population-based health examination survey conducted in 2000-2001. Subjects with baseline liver disease were excluded. The ELF test was performed on blood samples collected at baseline. Data were linked with national healthcare registers for liver-related outcomes (hospitalisation, cancer, and death).

Results

The cohort comprised 6,040 individuals (mean age 52.7. 45.6% men) with 67 liver-related outcomes during a median 13.1-year follow-up. ELF predicted liver outcomes (unadjusted hazards ratio 2.70, 95% CI 2.16-3.38). with 5- and 10-year AUCs of 0.81 (95% CI 0.71-0.91) and 0.71 (95% CI 0.63-0.79) by competing-risk methodology. The 10-year risks for liver outcomes increased from 0.5% at ELF <9.8 to 7.1% at ELF ≥11.3, being higher among men than women at any given ELF level. Among individuals with body mass index ≥30 kg/m2, diabetes, or alanine aminotransferase >40 U/L. Five-year AUCs for ELF were 0.85, 0.87, and 0.88, respectively. The predictive ability of the ELF test decreased with time: the 10-year AUCs were 0.78, 0.69, and 0.82, respectively.

Conclusions

The ELF test shows good discriminative performance in predicting liver-related outcomes in a large general population cohort and appears particularly useful for predicting 5-year outcomes in persons with risk factors.

Impact and implications

The Enhanced Liver Fibrosis test exhibits good performance for predicting liver-related outcomes (hospitalisation, liver cancer, or liver-related death) in the general population, especially in those with risk factors.

FDG metabolic parameter-based models for predicting recurrence after upfront surgery in synchronous colorectal cancer liver metastasis

OBJECTIVE

This study aimed to develop and validate post- and preoperative models for predicting recurrence after curative-intent surgery using an FDG PET-CT metabolic parameter to improve the prognosis of patients with synchronous colorectal cancer liver metastasis (SCLM).

METHODS

In this retrospective multicenter study, consecutive patients with resectable SCLM underwent upfront surgery between 2006 and 2015 (development cohort) and between 2006 and 2017 (validation cohort). In the development cohort, we developed and internally validated the post- and preoperative models using multivariable Cox regression with an FDG metabolic parameter (metastasis-to-primary-tumor uptake ratio [M/P ratio]) and clinicopathological variables as predictors. In the validation cohort, the models were externally validated for discrimination, calibration, and clinical usefulness. Model performance was compared with that of Fong's clinical risk score (FCRS).

RESULTS

A total of 374 patients (59.1 ± 10.5 years, 254 men) belonged in the development cohort and 151 (60.3 ± 12.0 years, 94 men) in the validation cohort. The M/P ratio and nine clinicopathological predictors were included in the models. Both postoperative and preoperative models showed significantly higher discrimination than FCRS (p < .05) in the external validation (time-dependent AUC = 0.76 [95% CI 0.68-0.84] and 0.76 [0.68-0.84] vs. 0.65 [0.57-0.74], respectively). Calibration plots and decision curve analysis demonstrated that both models were well calibrated and clinically useful. The developed models are presented as a web-based calculator ( https://cpmodel.shinyapps.io/SCLM/ ) and nomograms.

CONCLUSIONS

FDG metabolic parameter-based prognostic models are well-calibrated recurrence prediction models with good discriminative power. They can be used for accurate risk stratification in patients with SCLM.

KEY POINTS

• In this multicenter study, we developed and validated prediction models for recurrence in patients with resectable synchronous colorectal cancer liver metastasis using a metabolic parameter from FDG PET-CT. • The developed models showed good predictive performance on external validation, significantly exceeding that of a pre-existing model. • The models may be utilized for accurate patient risk stratification, thereby aiding in therapeutic decision-making.

Development and validation of questionnaire-based machine learning models for predicting all-cause mortality in a representative population of China

Background

Considering that the previously developed mortality prediction models have limited applications to the Chinese population, a questionnaire-based prediction model is of great importance for its accuracy and convenience in clinical practice.

Methods

Two national cohort, namely, the China Health and Nutrition Survey (8,355 individual older than 18) and the China Health and Retirement Longitudinal Study (12,711 individuals older than 45) were used for model development and validation. One hundred and fifty-nine variables were compiled to generate predictions. The Cox regression model and six machine learning (ML) models were used to predict all-cause mortality. Finally, a simple questionnaire-based ML prediction model was developed using the best algorithm and validated.

Results

In the internal validation set, all the ML models performed better than the traditional Cox model in predicting 6-year mortality and the random survival forest (RSF) model performed best. The questionnaire-based ML model, which only included 20 variables, achieved a C-index of 0.86 (95%CI: 0.80-0.92). On external validation, the simple questionnaire-based model achieved a C-index of 0.82 (95%CI: 0.77-0.87), 0.77 (95%CI: 0.75-0.79), and 0.79 (95%CI: 0.77-0.81), respectively, in predicting 2-, 9-, and 11-year mortality.

Conclusions

In this prospective population-based study, a model based on the RSF analysis performed best among all models. Furthermore, there was no significant difference between the prediction performance of the questionnaire-based ML model, which only included 20 variables, and that of the model with all variables (including laboratory variables). The simple questionnaire-based ML prediction model, which needs to be further explored, is of great importance for its accuracy and suitability to the Chinese general population.

Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models

Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression.

As a motivating case study, the authors consider the prediction of the composite outcome of recurrence or death (the "event") in patients with breast cancer after surgery. They developed a simple Cox regression model with 3 predictors, as in the Nottingham Prognostic Index, in 2982 women (1275 events over 5 years of follow-up) and externally validated this model in 686 women (285 events over 5 years). Improvement in performance was assessed after the addition of progesterone receptor as a prognostic biomarker.

The model predictions can be evaluated across the full range of observed follow-up times or for the event occurring by the end of a fixed time horizon of interest. The authors first discuss recommended statistical measures that evaluate model performance in terms of discrimination, calibration, or overall performance. Further, they evaluate the potential clinical utility of the model to support clinical decision making according to a net benefit measure. They provide SAS and R code to illustrate internal and external validation.

The authors recommend the proposed set of performance measures for transparent reporting of the validity of predictions from survival models.

Revised MALT-IPI: A new predictive model that identifies high-risk patients with extranodal marginal zone lymphoma

Extranodal marginal zone lymphoma (EMZL) is a heterogeneous disease with a subset of patients exhibiting a more aggressive course. We previously reported that EMZL with multiple mucosal sites (MMS) at diagnosis is characterized by shorter survival. To better recognize patients with different patterns of progression-free survival (PFS) we developed and validated a new prognostic index primarily based on patient's disease characteristics. We derived the "Revised mucosa-associated lymphoid tissue International Prognostic Index" (Revised MALT-IPI) in a large data set (n = 397) by identifying candidate variables that showed highest prognostic association with PFS. The revised MALT-IPI was validated in two independent cohorts, from the University of Iowa/Mayo Clinic (n = 297) and from IELSG-19 study (n = 400). A stepwise Cox regression analysis yielded a model including four independent predictors of shorter PFS. Revised MALT-IPI has scores ranging from 0 to 5, calculated as a sum of one point for each of the following- age >60 years, elevated LDH, and stage III-IV; and two points for MMS. In the training cohort, the Revised MALT-IPI defined four risk groups: low risk (score 0, reference group), low-medium risk (score 1, HR = 1.85, p = .008), medium-high risk (score 2, HR = 3.84, p < .0001), and high risk (score 3+, HR = 8.48, p < .0001). Performance of the Revised MALT-IPI was similar in external validation cohorts. Revised MALT-IPI is a new index centered on disease characteristics that provides robust risk-stratification identifying a group of patients characterized by earlier progression of disease. Revised MALT-IPI can allow a more disease-adjusted management of patients with EMZL in clinical trials and practice.

Development and external validation of a risk prediction model for falls in patients with an indication for antihypertensive treatment: retrospective cohort study

OBJECTIVE

To develop and externally validate the STRAtifying Treatments In the multi-morbid Frail elderlY (STRATIFY)-Falls clinical prediction model to identify the risk of hospital admission or death from a fall in patients with an indication for antihypertensive treatment.

DESIGN

Retrospective cohort study.

SETTING

Primary care data from electronic health records contained within the UK Clinical Practice Research Datalink (CPRD).

PARTICIPANTS

Patients aged 40 years or older with at least one blood pressure measurement between 130 mm Hg and 179 mm Hg.

MAIN OUTCOME MEASURE

First serious fall, defined as hospital admission or death with a primary diagnosis of a fall within 10 years of the index date (12 months after cohort entry). Model development was conducted using a Fine-Gray approach in data from CPRD GOLD, accounting for the competing risk of death from other causes, with subsequent recalibration at one, five, and 10 years using pseudo values. External validation was conducted using data from CPRD Aurum, with performance assessed through calibration curves and the observed to expected ratio, C statistic, and D statistic, pooled across general practices, and clinical utility using decision curve analysis at thresholds around 10%.

RESULTS

Analysis included 1 772 600 patients (experiencing 62 691 serious falls) from CPRD GOLD used in model development, and 3 805 366 (experiencing 206 956 serious falls) from CPRD Aurum in the external validation. The final model consisted of 24 predictors, including age, sex, ethnicity, alcohol consumption, living in an area of high social deprivation, a history of falls, multiple sclerosis, and prescriptions of antihypertensives, antidepressants, hypnotics, and anxiolytics. Upon external validation, the recalibrated model showed good discrimination, with pooled C statistics of 0.833 (95% confidence interval 0.831 to 0.835) and 0.843 (0.841 to 0.844) at five and 10 years, respectively. Original model calibration was poor on visual inspection and although this was improved with recalibration, under-prediction of risk remained (observed to expected ratio at 10 years 1.839, 95% confidence interval 1.811 to 1.865). Nevertheless, decision curve analysis suggests potential clinical utility, with net benefit larger than other strategies.

CONCLUSIONS

This prediction model uses commonly recorded clinical characteristics and distinguishes well between patients at high and low risk of falls in the next 1-10 years. Although miscalibration was evident on external validation, the model still had potential clinical utility around risk thresholds of 10% and so could be useful in routine clinical practice to help identify those at high risk of falls who might benefit from closer monitoring or early intervention to prevent future falls. Further studies are needed to explore the appropriate thresholds that maximise the model's clinical utility and cost effectiveness.

Quantifying the impact of environment factors on the risk of medical responders' stress-related absenteeism

Medical emergency response staff are exposed to incidents which may involve high-acuity patients or some intractable or traumatic situations. Previous studies on emergency response staff stress-related absence have focused on perceived factors and their impacts on absence leave. To date, analytical models on absenteeism risk prediction use past absenteeism to predict risk of future absenteeism. We show that these approaches ignore environment data, such as stress factors. The increased use of digital systems in emergency services allows us to gather data that were not available in the past and to apply a data-driven approach to quantify the effect of environment variables on the risk of stress-related absenteeism. We propose a two-stage data-driven framework to identify the variables of importance and to quantify their impact on medical staff stress-related risk of absenteeism. First, machine learning techniques are applied to identify the importance of different stressors on staff stress-related risk of absenteeism. Second, the Cox proportional-hazards model is applied to estimate the relative risk of each stressor. Four significant stressors are identified, these are the average night shift, past stress leave, the squared term of death confirmed by the Emergency Services and completion of the safeguarding form. We discuss counterintuitive results and implications to policy.

Avoiding C-hacking when evaluating survival distribution predictions with discrimination measures

MOTIVATION

In this article, we consider how to evaluate survival distribution predictions with measures of discrimination. This is non-trivial as discrimination measures are the most commonly used in survival analysis and yet there is no clear method to derive a risk prediction from a distribution prediction. We survey methods proposed in literature and software and consider their respective advantages and disadvantages.

RESULTS

Whilst distributions are frequently evaluated by discrimination measures, we find that the method for doing so is rarely described in the literature and often leads to unfair comparisons or 'C-hacking'. We demonstrate by example how simple it can be to manipulate results and use this to argue for better reporting guidelines and transparency in the literature. We recommend that machine learning survival analysis software implements clear transformations between distribution and risk predictions in order to allow more transparent and accessible model evaluation.

AVAILABILITY AND IMPLEMENTATION

The code used in the final experiment is available at https://github.com/RaphaelS1/distribution_discrimination.

External validation of the HCM Risk-Kids model for predicting sudden cardiac death in childhood hypertrophic cardiomyopathy

AIMS

Sudden cardiac death (SCD) is the most common mode of death in childhood hypertrophic cardiomyopathy (HCM). The newly developed HCM Risk-Kids model provides clinicians with individualized estimates of risk. The aim of this study was to externally validate the model in a large independent, multi-centre patient cohort.

METHODS AND RESULTS

A retrospective, longitudinal cohort of 421 patients diagnosed with HCM aged 1-16 years independent of the HCM Risk-Kids development and internal validation cohort was studied. Data on HCM Risk-Kids predictor variables (unexplained syncope, non-sustained ventricular tachycardia, maximal left ventricular wall thickness, left atrial diameter, and left ventricular outflow tract gradient) were collected from the time of baseline clinical evaluation. The performance of the HCM Risk-Kids model in predicting risk at 5 years was assessed. Twenty-three patients (5.4%) met the SCD end-point within 5 years, with an overall incidence rate of 2.03 per 100 patient-years [95% confidence interval (CI) 1.48-2.78]. Model validation showed a Harrell's C-index of 0.745 (95% CI 0.52-0.97) and Uno's C-index 0.714 (95% 0.58-0.85) with a calibration slope of 1.15 (95% 0.51-1.80). A 5-year predicted risk threshold of ≥6% identified 17 (73.9%) SCD events with a corresponding C-statistic of 0.702 (95% CI 0.60-0.81).

CONCLUSIONS

This study reports the first external validation of the HCM Risk-Kids model in a large and geographically diverse patient population. A 5-year predicted risk of ≥6% identified over 70% of events, confirming that HCM Risk-Kids provides a method for individualized risk predictions and shared decision-making in children with HCM.

Comparison of methods for predicting COVID-19-related death in the general population using the OpenSAFELY platform

BACKGROUND

Obtaining accurate estimates of the risk of COVID-19-related death in the general population is challenging in the context of changing levels of circulating infection.

METHODS

We propose a modelling approach to predict 28-day COVID-19-related death which explicitly accounts for COVID-19 infection prevalence using a series of sub-studies from new landmark times incorporating time-updating proxy measures of COVID-19 infection prevalence. This was compared with an approach ignoring infection prevalence. The target population was adults registered at a general practice in England in March 2020. The outcome was 28-day COVID-19-related death. Predictors included demographic characteristics and comorbidities. Three proxies of local infection prevalence were used: model-based estimates, rate of COVID-19-related attendances in emergency care, and rate of suspected COVID-19 cases in primary care. We used data within the TPP SystmOne electronic health record system linked to Office for National Statistics mortality data, using the OpenSAFELY platform, working on behalf of NHS England. Prediction models were developed in case-cohort samples with a 100-day follow-up. Validation was undertaken in 28-day cohorts from the target population. We considered predictive performance (discrimination and calibration) in geographical and temporal subsets of data not used in developing the risk prediction models. Simple models were contrasted to models including a full range of predictors.

RESULTS

Prediction models were developed on 11,972,947 individuals, of whom 7999 experienced COVID-19-related death. All models discriminated well between individuals who did and did not experience the outcome, including simple models adjusting only for basic demographics and number of comorbidities: C-statistics 0.92-0.94. However, absolute risk estimates were substantially miscalibrated when infection prevalence was not explicitly modelled.

CONCLUSIONS

Our proposed models allow absolute risk estimation in the context of changing infection prevalence but predictive performance is sensitive to the proxy for infection prevalence. Simple models can provide excellent discrimination and may simplify implementation of risk prediction tools.

Comparing Modeling Approaches for Discrete Event Simulations With Competing Risks Based on Censored Individual Patient Data: A Simulation Study and Illustration in Colorectal Cancer

OBJECTIVES

This study aimed to provide detailed guidance on modeling approaches for implementing competing events in discrete event simulations based on censored individual patient data (IPD).

METHODS

The event-specific distributions (ESDs) approach sampled times from event-specific time-to-event distributions and simulated the first event to occur. The unimodal distribution and regression approach sampled a time from a combined unimodal time-to-event distribution, representing all events, and used a (multinomial) logistic regression model to select the event to be simulated. A simulation study assessed performance in terms of relative absolute event incidence difference and relative entropy of time-to-event distributions for different types and levels of right censoring, numbers of events, distribution overlap, and sample sizes. Differences in cost-effectiveness estimates were illustrated in a colorectal cancer case study.

RESULTS

Increased levels of censoring negatively affected the modeling approaches' performance. A lower number of competing events and higher overlap of distributions improved performance. When IPD were censored at random times, ESD performed best. When censoring occurred owing to a maximum follow-up time for 2 events, ESD performed better for a low level of censoring (ie, 10%). For 3 or 4 competing events, ESD better represented the probabilities of events, whereas unimodal distribution and regression better represented the time to events. Differences in cost-effectiveness estimates, both compared with no censoring and between approaches, increased with increasing censoring levels.

CONCLUSIONS

Modelers should be aware of the different modeling approaches available and that selection between approaches may be informed by data characteristics. Performing and reporting extensive validation efforts remains essential to ensure IPD are appropriately represented.

Radiomics in Oncology: A Practical Guide

Radiomics refers to the extraction of mineable data from medical imaging and has been applied within oncology to improve diagnosis, prognostication, and clinical decision support, with the goal of delivering precision medicine. The authors provide a practical approach for successfully implementing a radiomic workflow from planning and conceptualization through manuscript writing. Applications in oncology typically are either classification tasks that involve computing the probability of a sample belonging to a category, such as benign versus malignant, or prediction of clinical events with a time-to-event analysis, such as overall survival. The radiomic workflow is multidisciplinary, involving radiologists and data and imaging scientists, and follows a stepwise process involving tumor segmentation, image preprocessing, feature extraction, model development, and validation. Images are curated and processed before segmentation, which can be performed on tumors, tumor subregions, or peritumoral zones. Extracted features typically describe the distribution of signal intensities and spatial relationship of pixels within a region of interest. To improve model performance and reduce overfitting, redundant and nonreproducible features are removed. Validation is essential to estimate model performance in new data and can be performed iteratively on samples of the dataset (cross-validation) or on a separate hold-out dataset by using internal or external data. A variety of noncommercial and commercial radiomic software applications can be used. Guidelines and artificial intelligence checklists are useful when planning and writing up radiomic studies. Although interest in the field continues to grow, radiologists should be familiar with potential pitfalls to ensure that meaningful conclusions can be drawn.

Online supplemental material is available for this article.

Published under a CC BY 4.0 license.

External validation and comparison of the predictive performance of 10 different tooth-level prognostic systems

AIM

Tooth-level prognostic systems can be used for treatment planning and risk assessment. This retrospective longitudinal study aimed to evaluate the prognostic performance of 10 different tooth-level risk assessment systems in terms of their ability to predict periodontal-related tooth loss (TLP).

MATERIALS AND METHODS

Data were retrieved retrospectively from patients who received surgical and non-surgical periodontal treatment. Data on medical history and smoking status at baseline and the last maintenance visit were collected. Ten tooth-level prognostic systems were compared using both univariate and multivariate Cox proportional hazard regression models to analyse the prognostic capability of each system for predicting TLP risk.

RESULTS

One-hundred and forty-eight patients with 3787 teeth, followed-up for a mean period of 26.5 ± 7.4 years, were evaluated according to 10 different tooth-level prognostic systems, making up a total of 37,870 individual measurements. All compared prognostic systems were able to stratify the risk of TLP at baseline when different classes of association were compared. After controlling for maintenance, age, and gender, all systems exhibited excellent predictive capacity for TLP with no system scoring a Harrell's C-index less than 0.925.

CONCLUSIONS

All tooth-level prognostic systems displayed excellent predictive capability for TLP. Overall, the Miller and McEntire system may have shown the best discrimination and model fit, followed by the Nunn et al. system.

Cox-sMBPLS: An Algorithm for Disease Survival Prediction and Multi-Omics Module Discovery Incorporating Cis-Regulatory Quantitative Effects

Background

The development of high-throughput techniques has enabled profiling a large number of biomolecules across a number of molecular compartments. The challenge then becomes to integrate such multimodal Omics data to gain insights into biological processes and disease onset and progression mechanisms. Further, given the high dimensionality of such data, incorporating prior biological information on interactions between molecular compartments when developing statistical models for data integration is beneficial, especially in settings involving a small number of samples.

Results

We develop a supervised model for time to event data (e.g., death, biochemical recurrence) that simultaneously accounts for redundant information within Omics profiles and leverages prior biological associations between them through a multi-block PLS framework. The interactions between data from different molecular compartments (e.g., epigenome, transcriptome, methylome, etc.) were captured by using cis-regulatory quantitative effects in the proposed model. The model, coined Cox-sMBPLS, exhibits superior prediction performance and improved feature selection based on both simulation studies and analysis of data from heart failure patients.

Conclusion

The proposed supervised Cox-sMBPLS model can effectively incorporate prior biological information in the survival prediction system, leading to improved prediction performance and feature selection. It also enables the identification of multi-Omics modules of biomolecules that impact the patients’ survival probability and also provides insights into potential relevant risk factors that merit further investigation.

Prediction of Early Recurrence After Surgery for Liver Tumor (ERASL): An International Validation of the ERASL Risk Models

Background

This study aimed to assess the performance of the pre- and postoperative early recurrence after surgery for liver tumor (ERASL) models at external validation. Prediction of early hepatocellular carcinoma (HCC) recurrence after resection is important for individualized surgical management. Recently, the preoperative (ERASL-pre) and postoperative (ERASL-post) risk models were proposed based on patients from Hong Kong. These models showed good performance although they have not been validated to date by an independent research group.

Methods

This international cohort study included 279 patients from the Netherlands and 392 patients from Japan. The patients underwent first-time resection and showed a diagnosis of HCC on pathology. Performance was assessed according to discrimination (concordance [C] statistic) and calibration (correspondence between observed and predicted risk) with recalibration in a Weibull model.

Results

The discriminatory power of both models was lower in the Netherlands than in Japan (C statistic, 0.57 [95% confidence interval {CI} 0.52–0.62] vs 0.69 [95% CI 0.65–0.73] for the ERASL-pre model and 0.62 [95% CI 0.57–0.67] vs 0.70 [95% CI 0.66–0.74] for the ERASL-post model), whereas their prognostic profiles were similar. The predictions of the ERASL models were systematically too optimistic for both cohorts. Recalibrated ERASL models improved local applicability for both cohorts.

Conclusions

The discrimination of ERASL models was poorer for the Western patients than for the Japanese patients, who showed good performance. Recalibration of the models was performed, which improved the accuracy of predictions. However, in general, a model that explains the East–West difference or one tailored to Western patients still needs to be developed.

Supplementary Information

The online version contains supplementary material available at 10.1245/s10434-021-10235-3.

Review of Statistical Methods for Evaluating the Performance of Survival or Other Time-to-Event Prediction Models (from Conventional to Deep Learning Approaches)

The recent introduction of various high-dimensional modeling methods, such as radiomics and deep learning, has created a much greater diversity in modeling approaches for survival prediction (or, more generally, time-to-event prediction). The newness of the recent modeling approaches and unfamiliarity with the model outputs may confuse some researchers and practitioners about the evaluation of the performance of such models. Methodological literacy to critically appraise the performance evaluation of the models and, ideally, the ability to conduct such an evaluation would be needed for those who want to develop models or apply them in practice. This article intends to provide intuitive, conceptual, and practical explanations of the statistical methods for evaluating the performance of survival prediction models with minimal usage of mathematical descriptions. It covers from conventional to deep learning methods, and emphasis has been placed on recent modeling approaches. This review article includes straightforward explanations of C indices (Harrell's C index, etc.), time-dependent receiver operating characteristic curve analysis, calibration plot, other methods for evaluating the calibration performance, and Brier score.

Development and Validation of Machine Learning-Based Race-Specific Models to Predict 10-Year Risk of Heart Failure: A Multicohort Analysis

BACKGROUND

Heart failure (HF) risk and the underlying risk factors vary by race. Traditional models for HF risk prediction treat race as a covariate in risk prediction and do not account for significant parameters such as cardiac biomarkers. Machine learning (ML) may offer advantages over traditional modeling techniques to develop race-specific HF risk prediction models and to elucidate important contributors of HF development across races.

METHODS

We performed a retrospective analysis of 4 large, community cohort studies (ARIC [Atherosclerosis Risk in Communities], DHS [Dallas Heart Study], JHS [Jackson Heart Study], and MESA [Multi-Ethnic Study of Atherosclerosis]) with adjudicated HF events. The study included participants who were >40 years of age and free of HF at baseline. Race-specific ML models for HF risk prediction were developed in the JHS cohort (for Black race-specific model) and White adults from ARIC (for White race-specific model). The models included 39 candidate variables across demographic, anthropometric, medical history, laboratory, and electrocardiographic domains. The ML models were externally validated and compared with prior established traditional and non-race-specific ML models in race-specific subgroups of the pooled MESA/DHS cohort and Black participants of ARIC. The Harrell C-index and Greenwood-Nam-D'Agostino χ2 tests were used to assess discrimination and calibration, respectively.

RESULTS

The ML models had excellent discrimination in the derivation cohorts for Black (n=4141 in JHS, C-index=0.88) and White (n=7858 in ARIC, C-index=0.89) participants. In the external validation cohorts, the race-specific ML model demonstrated adequate calibration and superior discrimination (Black individuals, C-index=0.80-0.83; White individuals, C-index=0.82) compared with established HF risk models or with non-race-specific ML models derived with race included as a covariate. Among the risk factors, natriuretic peptide levels were the most important predictor of HF risk across both races, followed by troponin levels in Black and ECG-based Cornell voltage in White individuals. Other key predictors of HF risk among Black individuals were glycemic parameters and socioeconomic factors. In contrast, prevalent cardiovascular disease and traditional cardiovascular risk factors were stronger predictors of HF risk in White adults.

CONCLUSIONS

Race-specific and ML-based HF risk models that integrate clinical, laboratory, and biomarker data demonstrated superior performance compared with traditional HF risk and non-race-specific ML models. This approach identifies distinct race-specific contributors of HF.

Machine learning-based long-term outcome prediction in patients undergoing percutaneous coronary intervention

Background

Traditional prognostic risk assessment in patients with coronary artery disease undergoing percutaneous coronary intervention (PCI) is based on a limited selection of clinical and imaging findings. Machine learning (ML) can consider a higher number and complexity of variables and may be useful for characterising cardiovascular risk, predicting outcomes, and identifying biomarkers in large population studies.

Methods

We prospectively enrolled 9,680 consecutive patients with coronary artery disease who underwent PCI at our institution between January 2013 and December 2013. Clinical features were selected and used to train 6 different ML models (support vector machine, decision tree, random forest, gradient boosting decision tree, neural network, and logistic regression) to predict cardiovascular outcomes, 10-fold cross-validation to evaluate the performance of models.

Results

During the 5-year follow-up, 467 (4.82%) patients died. Eighty-seven risk baseline measurements were used to train ML models. Compared with the other models, the random forest model (RF-PCI) exhibited the best performance on predicting all-cause mortality (area under the receiver operating characteristic curve: 0.71±0.04). Calibration plots demonstrated a slight overprediction for patients using the RF-PCI model (Hosmer-Lemeshow test: P>0.05). The top 15 features related to PCI candidates' long-term prognosis, among which 11 were laboratory measures.

Conclusions

ML models improved the prediction of long-term all-cause mortality in patients with coronary artery disease before PCI. The performance of the RF model was better than that of the other models, providing a meaningful stratification.

External validation of a model to identify cardiometabolic predictors of mortality in cancer survivors

PURPOSE

Cancer survivors are at risk of cardiovascular disease because of shared risk factors and effects of treatment. There are few tools to assist in estimating the risk of poor outcomes relating to cardiovascular disease in cancer survivors and identifying those at risk. The purpose of this study was to externally validate a model for predicting the risk of increased mortality in female cancer survivors.

METHODS

A risk prediction model originally developed using data from the general population of older adults from the Australian Longitudinal Study of Ageing was externally validated using data from two Australian Longitudinal Study on Women's Health (ALSWH) cohorts. Three measures of discrimination were calculated. Calibration was assessed by visualising a graph of the model predictions and observed events.

RESULTS

The ALSWH cohorts consisted of 1764 women (aged 73-78 years) and 1833 women (aged 47-52 years). Discrimination was acceptable with the Harrell C-index and the Gonen and Heller K statistic both greater than 0.5. The model explained up to 30% of the variation in mortality. Calibration showed that the recalibrated model performed best in years 8-10 suggesting that the model is better at predicting survival for those with a higher probability of surviving. Overall, model performance was better in the 47-52 years cohort than in the older cohort.

CONCLUSION

We have externally validated a model of cardiometabolic predictors of mortality in female cancer survivors. The model can serve as a basis of clinical tool to assist with decision-making regarding potential risk reduction strategies in this population.

Development and Validation of a Chronic Kidney Disease Prediction Model for Type 2 Diabetes Mellitus in Thailand

OBJECTIVES

The objective of this study was to investigate predictors and develop risk equations for stage-3 chronic kidney disease (CKD) in Thai patients with type 2 diabetes mellitus (DM).

METHODS

A retrospective cohort study was conducted in patients with type 2 DM. The outcome was the development of stage-3 CKD. The data set was randomly split into training and validation data sets. Cox proportional hazard regression was used for model development. Discrimination (Harrell's C statistic) and calibration (the Hosmer-Lemeshow chi-square test and survival probability curve) were applied to evaluate model performance.

RESULTS

In total, 2178 type 2 DM patients without stage-3 CKD, visiting the hospital from January 1, 2008, to December 31, 2017, were recruited, with median follow-up time of 1.29 years (interquartile range, 0.5-2.5 years); 385 (17.68%) subjects had developed stage-3 CKD. The final predictors included age, male sex, urinary albumin to creatinine ratio, estimated glomerular filtration rate, and hemoglobin A1c. Two 3-year stage-3 CKD risk models, model 1 (laboratory model) and model 2 (simplified model), had the C statistic in validation data sets of 0.890 and 0.812, respectively.

CONCLUSIONS

Two 3-year stage-3 CKD risk models were developed for Thai patients with type 2 DM. Both models have good discrimination and calibration. These stage-3 CKD prediction models could equip health providers with tools for clinical management and supporting patient education.

Risk Assessment of Visceral Sarcomas: A Comparative Study of 2698 Cases from the SEER Database

Soft tissue sarcomas arising in visceral organs are rare and lack validated tumor-staging protocols. Clinicopathologic features and clinical outcomes of 2698 visceral sarcomas identified in the Surveillance, Epidemiology, and End Results Program (SEER) database were compared with sarcomas arising in the extremities/trunk (n = 10,237) or retroperitoneum (n = 1067) using standard statistical techniques. Important prognostic criteria for visceral sarcomas, as in other anatomic sites, included tumor size, histologic grade, and presence of metastatic disease. After adjustment for pertinent confounding factors, visceral sarcomas showed cancer-specific survival rates similar to those arising in the retroperitoneum but had worse outcomes than sarcomas in the extremities/trunk. Therefore, the prognostic performance of two different staging algorithms for retroperitoneal sarcomas was evaluated for their use in staging sarcomas of visceral organs. The current AJCC 8th edition and the recently derived Vanderbilt system for staging retroperitoneal sarcoma both showed adequate discrimination, as assessed by multiple clinical concordance indices, and no evidence of miscalibration. Therefore, the authors concluded that previously validated staging systems for retroperitoneal sarcomas based on conventional prognostic factors (histologic grade, tumor size, and presence of metastatic disease) are applicable to visceral sarcomas and should be incorporated into the next edition of the AJCC Cancer Staging Manual.

Development and Validation of Cervical Prediction Models for Patient-Reported Outcomes at 1 Year After Cervical Spine Surgery for Radiculopathy and Myelopathy

STUDY DESIGN

Retrospective analysis of prospectively collected registry data.

OBJECTIVE

To develop and validate prediction models for 12-month patient-reported outcomes of disability, pain, and myelopathy in patients undergoing elective cervical spine surgery.

SUMMARY OF BACKGROUND DATA

Predictive models have the potential to be utilized preoperatively to set expectations, adjust modifiable characteristics, and provide a patient-centered model of care.

METHODS

This study was conducted using data from the cervical module of the Quality Outcomes Database. The outcomes of interest were disability (Neck Disability Index:), pain (Numeric Rating Scale), and modified Japanese Orthopaedic Association score for myelopathy. Multivariable proportional odds ordinal regression models were developed for patients with cervical radiculopathy and myelopathy. Patient demographic, clinical, and surgical covariates as well as baseline patient-reported outcomes scores were included in all models. The models were internally validated using bootstrap resampling to estimate the likely performance on a new sample of patients.

RESULTS

Four thousand nine hundred eighty-eight patients underwent surgery for radiculopathy and 2641 patients for myelopathy. The most important predictor of poor postoperative outcomes at 12-months was the baseline Neck Disability Index score for patients with radiculopathy and modified Japanese Orthopaedic Association score for patients with myelopathy. In addition, symptom duration, workers' compensation, age, employment, and ambulatory and smoking status had a statistically significant impact on all outcomes (P < 0.001). Clinical and surgical variables contributed very little to predictive models, with posterior approach being associated with higher odds of having worse 12-month outcome scores in both the radiculopathy and myelopathy cohorts (P < 0.001). The full models overall discriminative performance ranged from 0.654 to 0.725.

CONCLUSIONS

These predictive models provide individualized risk-adjusted estimates of 12-month disability, pain, and myelopathy outcomes for patients undergoing spine surgery for degenerative cervical disease. Predictive models have the potential to be used as a shared decision-making tool for evidence-based preoperative counselling.

LEVEL OF EVIDENCE

2.

Simulating Progression-Free and Overall Survival for First-Line Doublet Chemotherapy With or Without Bevacizumab in Metastatic Colorectal Cancer Patients Based on Real-World Registry Data

BACKGROUND

Simulation models utilizing real-world data have potential to optimize treatment sequencing strategies for specific patient subpopulations, including when conducting clinical trials is not feasible. We aimed to develop a simulation model to estimate progression-free survival (PFS) and overall survival for first-line doublet chemotherapy with or without bevacizumab for specific subgroups of metastatic colorectal cancer (mCRC) patients based on registry data.

METHODS

Data from 867 patients were used to develop two survival models and one logistic regression model that populated a discrete event simulation (DES). Discrimination and calibration were used for internal validation of these models separately and predicted and observed medians and Kaplan-Meier plots were compared for the integrated DES. Bootstrapping was performed to correct for optimism in the internal validation and to generate correlated sets of model parameters for use in a probabilistic analysis to reflect parameter uncertainty.

RESULTS

The survival models showed good calibration based on the regression slopes and modified Hosmer-Lemeshow statistics at 1 and 2 years, but not for short-term predictions at 0.5 years. Modified C-statistics indicated acceptable discrimination. The simulation estimated that median first-line PFS (95% confidence interval) of 219 (25%) patients could be improved from 175 days (156-199) to 269 days (246-294) if treatment would be targeted based on the highest expected PFS.

CONCLUSIONS

Extensive internal validation showed that DES accurately estimated the outcomes of treatment combination strategies for specific subpopulations, with outcomes suggesting treatment could be optimized. Although results based on real-world data are informative, they cannot replace randomized trials.

Prediction of Survival Among Patients Receiving Transarterial Chemoembolization for Hepatocellular Carcinoma: A Response-Based Approach

BACKGROUND AND AIMS

The heterogeneity of intermediate-stage hepatocellular carcinoma (HCC) and the widespread use of transarterial chemoembolization (TACE) outside recommended guidelines have encouraged the development of scoring systems that predict patient survival. The aim of this study was to build and validate statistical models that offer individualized patient survival prediction using response to TACE as a variable.

APPROACH AND RESULTS

Clinically relevant baseline parameters were collected for 4,621 patients with HCC treated with TACE at 19 centers in 11 countries. In some of the centers, radiological responses (as assessed by modified Response Evaluation Criteria in Solid Tumors [mRECIST]) were also accrued. The data set was divided into a training set, an internal validation set, and two external validation sets. A pre-TACE model ("Pre-TACE-Predict") and a post-TACE model ("Post-TACE-Predict") that included response were built. The performance of the models in predicting overall survival (OS) was compared with existing ones. The median OS was 19.9 months. The factors influencing survival were tumor number and size, alpha-fetoprotein, albumin, bilirubin, vascular invasion, cause, and response as assessed by mRECIST. The proposed models showed superior predictive accuracy compared with existing models (the hepatoma arterial embolization prognostic score and its various modifications) and allowed for patient stratification into four distinct risk categories whose median OS ranged from 7 months to more than 4 years.

CONCLUSIONS

A TACE-specific and extensively validated model based on routinely available clinical features and response after first TACE permitted patient-level prognostication.

Assessment of the risk of developing breast cancer using the Gail model in Asian females: A systematic review

Introduction

Currently, the Breast Cancer Risk Assessment Tool (BCRAT), also known as the Gail model (GM) has been widely recognized and adapted for to study disparity in racial and ethnic groups in America including Asian and Pacific Islander American females. However, its applicability outside America remains uncertain due to diversity in epidemiology and risk factors of breast cancer in populations especially in Asian females. We sought to evaluate the performance of the GM to predict breast cancer risk in Asian countries.

Material and methods

This study identified articles published from 2010 by searching PubMed, MEDLINE, Scopus, Web of Science, Google Scholar and gray literature. The initial search terms were breast cancer, mammary, carcinoma, tumor, neoplasm, risk assessment tool, BCRAT, breast cancer prediction, Gail model, Asia, and Asian.

Results

The search yielded 20 articles, with 7 articles addressing the AUC and/or the expected (E) to observed (O) ratio of predicted breast cancer risk, representing the accuracy of the GM in the Asian population. One publication reported the sensitivity and specificity but no AUC. None of the studies were accepted as the standard for reporting prognostic models. Several studies reported good prognostic testing and likely developed a new model modifying the items in the instrument.

Conclusion

The results are not strong enough to develop breast cancer risk in the setting of Asian countries. Involving the breast cancer risk of the Asian population in developing a prognostic model with good statistical understanding is particularly important and can reduce flawed or biased models. Identifying the best methods to achieve well-suited prognostic models in the Asian population should be a priority.

A Nomogram Based on Apelin-12 for the Prediction of Major Adverse Cardiovascular Events after Percutaneous Coronary Intervention among Patients with ST-Segment Elevation Myocardial Infarction

Objective

This study aimed to establish a clinical prognostic nomogram for predicting major adverse cardiovascular events (MACEs) after primary percutaneous coronary intervention (PCI) among patients with ST-segment elevation myocardial infarction (STEMI).

Methods

Information on 464 patients with STEMI who performed PCI procedures was included. After removing patients with incomplete clinical information, a total of 460 patients followed for 2.5 years were randomly divided into evaluation (n = 324) and validation (n = 324) and validation (

Results

Apelin-12 change rate, apelin-12 level, age, pathological Q wave, myocardial infarction history, anterior wall myocardial infarction, Killip's classification > I, uric acid, total cholesterol, cTnI, and the left atrial diameter were independently associated with MACEs (all P < 0.05). After incorporating these 11 factors, the nomogram achieved good concordance indexes of 0.758 (95%CI = 0.707–0.809) and 0.763 (95%CI = 0.689–0.837) in predicting MACEs in the evaluation and validation cohorts, respectively, and had well-fitted calibration curves. The decision curve analysis (DCA) revealed that the nomogram was clinically useful.

Conclusions

We established and validated a novel nomogram that can provide individual prediction of MACEs for patients with STEMI after PCI procedures in a Chinese population. This practical prognostic nomogram may help clinicians in decision making and enable a more accurate risk assessment.

Predicting donor, recipient and graft survival in living donor kidney transplantation to inform pretransplant counselling: the donor and recipient linked iPREDICTLIVING tool - a retrospective study

Although separate prediction models for donors and recipients were previously published, we identified a need to predict outcomes of donor/recipient simultaneously, as they are clearly not independent of each other. We used characteristics from transplantations performed at the Oslo University Hospital from 1854 live donors and from 837 recipients of a live donor kidney transplant to derive Cox models for predicting donor mortality up to 20 years, and recipient death, and graft loss up to 10 years. The models were developed using the multivariable fractional polynomials algorithm optimizing Akaike’s information criterion, and optimism‐corrected performance was assessed. Age, year of donation, smoking status, cholesterol and creatinine were selected to predict donor mortality (C‐statistic of 0.81). Linear predictors for donor mortality served as summary of donor prognosis in recipient models. Age, sex, year of transplantation, dialysis vintage, primary renal disease, cerebrovascular disease, peripheral vascular disease and HLA mismatch were selected to predict recipient mortality (C‐statistic of 0.77). Age, dialysis vintage, linear predictor of donor mortality, HLA mismatch, peripheral vascular disease and heart disease were selected to predict graft loss (C‐statistic of 0.66). Our prediction models inform decision‐making at the time of transplant counselling and are implemented as online calculators.

Body composition predicts mortality and decompensation in compensated cirrhosis patients: A prospective cohort study

Background & Aims

Body composition, particularly sarcopenia, is associated with mortality in patients with decompensated cirrhosis undergoing transplant evaluation. Similar data are limited for non-transplant eligible or compensated patients.

Methods

A total of 274 patients with cirrhosis were followed prospectively for ≤5 years after a CT scan. We utilized Analytic Morphomics® to measure body composition (fat, muscle, and bone) which was rendered into relative values (percentiles) in relation to a reference population. The model for end-stage liver disease (MELD) score was used as a reference model for survival prediction. We validated our models in a separate cohort.

Results

Our cohort had a mean Child-Pugh score of 7.0 and a mean MELD of 11.3. The median follow-up time was 5.05 years. The proportion of patients alive at 1, 3 and 5 years was 86.5%, 68.0%, and 54.3%; 13 (4.6%) underwent liver transplantation. Child-Pugh B/C (vs. A) cirrhosis was associated with decreased muscle, subcutaneous, and visceral fat area but increased subcutaneous/visceral fat density. Decreased normal density muscle mass was associated with mortality (hazard ratio [HR] 0.984, p <0.001) as well as visceral and subcutaneous fat density (HR 1.013 and 1.014, respectively, p <0.001). Models utilizing these features outperformed MELD alone for mortality discrimination in both the derivation and validation cohort, particularly for those with compensated cirrhosis (C-statistics of 0.74 vs. 0.58). Using competing risk analysis, we found that subcutaneous fat density was most predictive of decompensation (subdistribution HR 1.018, p = 0.0001).

Conclusion

The addition of body composition features to predictive models improves the prospective determination of prognosis in patients with cirrhosis, particularly those with compensated disease. Fat density, a novel feature, is associated with the risk of decompensation.

Lay summary

Am I at high risk of getting sicker and dying? This is the key question on the mind of patients with cirrhosis. The problem is that we have very few tools to help guide our patients, particularly if they have early cirrhosis (without symptoms like confusion or fluid in the belly). We found that how much muscle and fat the patient has and what that muscle or fat looks like on a CT scan provide helpful information. This is important because many patients have CT scans and this information is hiding in plain sight.

•

Features of body composition can predict clinical outcomes in patients with cirrhosis awaiting liver transplantation.

•

Data are lacking regarding long-term outcomes among patients with compensated disease.

•

We show that features of muscle and fat are associated with decompensation and risk of death across the spectrum of cirrhosis.

•

CT scans obtained for unrelated clinical purposes can be analyzed as a digital risk biomarker for patients with compensated cirrhosis.

External validation of the PREDICT tool in Spanish women with breast cancer participating in population-based screening programmes

RATIONALE, AIMS, AND OBJECTIVES

To externally validate the PREDICT tool in a cohort of women participating in a population-based breast cancer screening programme who were diagnosed with breast cancer between 2000 and 2008 in Spain.

METHODS

A total of 535 women were included in the validation study. We calculated predicted 5-year survival using the beta values from the development of the PREDICT model and predicted and observed events for a given risk groups. Model fit, discrimination, and calibration were evaluated. Seeking to improve the model, we also explored the impact on discrimination of the inclusion of additional variables, not in the PREDICT algorithm.

RESULTS

In patients who were oestrogen receptor (ER) positive (negative), PREDICT overestimated (underestimated) the 5-year overall survival in all the subgroups studied. Analysis of model performance showed good calibration but modest discrimination (C-index, 0.697 [ER negative] and 0.768 [ER positive]). When updating the model, no additional variables were found to be significant in ER-negative patients, but for ER-positive patients, concurrent liver disease was a significant factor, its inclusion improving model discrimination (C-index, 0.817).

CONCLUSIONS

The PREDICT tool does not discriminate well in our population considering only the variables of the original algorithm. More accurate tools are needed to obtain a better discrimination.

Development of a Novel Risk Prediction Model for Sudden Cardiac Death in Childhood Hypertrophic Cardiomyopathy (HCM Risk-Kids)

Importance

Sudden cardiac death (SCD) is the most common mode of death in childhood hypertrophic cardiomyopathy (HCM), but there is no validated algorithm to identify those at highest risk.

Objective

To develop and validate an SCD risk prediction model that provides individualized risk estimates.

Design, Setting, and Participants

A prognostic model was developed from a retrospective, multicenter, longitudinal cohort study of 1024 consecutively evaluated patients aged 16 years or younger with HCM. The study was conducted from January 1, 1970, to December 31, 2017.

Exposures

The model was developed using preselected predictor variables (unexplained syncope, maximal left-ventricular wall thickness, left atrial diameter, left-ventricular outflow tract gradient, and nonsustained ventricular tachycardia) identified from the literature and internally validated using bootstrapping.

Main Outcomes and Measures

A composite outcome of SCD or an equivalent event (aborted cardiac arrest, appropriate implantable cardioverter defibrillator therapy, or sustained ventricular tachycardia associated with hemodynamic compromise).

Results

Of the 1024 patients included in the study, 699 were boys (68.3%); mean (interquartile range [IQR]) age was 11 (7-14) years. Over a median follow-up of 5.3 years (IQR, 2.6-8.3; total patient years, 5984), 89 patients (8.7%) died suddenly or had an equivalent event (annual event rate, 1.49; 95% CI, 1.15-1.92). The pediatric model was developed using preselected variables to predict the risk of SCD. The model's ability to predict risk at 5 years was validated; the C statistic was 0.69 (95% CI, 0.66-0.72), and the calibration slope was 0.98 (95% CI, 0.59-1.38). For every 10 implantable cardioverter defibrillators implanted in patients with 6% or more of a 5-year SCD risk, 1 patient may potentially be saved from SCD at 5 years.

Conclusions and Relevance

This new, validated risk stratification model for SCD in childhood HCM may provide individualized estimates of risk at 5 years using readily obtained clinical risk factors. External validation studies are required to demonstrate the accuracy of this model's predictions in diverse patient populations.

A plea for taking all available clinical information into account when assessing the predictive value of omics data

Background

Omics data can be very informative in survival analysis and may improve the prognostic ability of classical models based on clinical risk factors for various diseases, for example breast cancer. Recent research has focused on integrating omics and clinical data, yet has often ignored the need for appropriate model building for clinical variables. Medical literature on classical prognostic scores, as well as biostatistical literature on appropriate model selection strategies for low dimensional (clinical) data, are often ignored in the context of omics research. The goal of this paper is to fill this methodological gap by investigating the added predictive value of gene expression data for models using varying amounts of clinical information.

Methods

We analyze two data sets from the field of survival prognosis of breast cancer patients. First, we construct several proportional hazards prediction models using varying amounts of clinical information based on established medical knowledge. These models are then used as a starting point (i.e. included as a clinical offset) for identifying informative gene expression variables using resampling procedures and penalized regression approaches (model based boosting and the LASSO). In order to assess the added predictive value of the gene signatures, measures of prediction accuracy and separation are examined on a validation data set for the clinical models and the models that combine the two sources of information.

Results

For one data set, we do not find any substantial added predictive value of the omics data when compared to clinical models. On the second data set, we identify a noticeable added predictive value, however only for scenarios where little or no clinical information is included in the modeling process. We find that including more clinical information can lead to a smaller number of selected omics predictors.

Conclusions

New research using omics data should include all available established medical knowledge in order to allow an adequate evaluation of the added predictive value of omics data. Including all relevant clinical information in the analysis might also lead to more parsimonious models. The developed procedure to assess the predictive value of the omics data can be readily applied to other scenarios.

Electronic supplementary material

The online version of this article (10.1186/s12874-019-0802-0) contains supplementary material, which is available to authorized users.

The uncertainty with using risk prediction models for individual decision making: an exemplar cohort study examining the prediction of cardiovascular disease in English primary care

BACKGROUND

Risk prediction models are commonly used in practice to inform decisions on patients' treatment. Uncertainty around risk scores beyond the confidence interval is rarely explored. We conducted an uncertainty analysis of the QRISK prediction tool to evaluate the robustness of individual risk predictions with varying modelling decisions.

METHODS

We derived a cohort of patients eligible for cardiovascular risk prediction from the Clinical Practice Research Datalink (CPRD) with linked hospitalisation and mortality records (N = 3,792,474). Risk prediction models were developed using the methods reported for QRISK2 and 3, before adjusting for additional risk factors, a secular trend, geographical variation in risk and the method for imputing missing data when generating a risk score (model A-model F). Ten-year risk scores were compared across the different models alongside model performance metrics.

RESULTS

We found substantial variation in risk on the individual level across the models. The 95 percentile range of risks in model F for patients with risks between 9 and 10% according to model A was 4.4-16.3% and 4.6-15.8% for females and males respectively. Despite this, the models were difficult to distinguish using common performance metrics (Harrell's C ranged from 0.86 to 0.87). The largest contributing factor to variation in risk was adjusting for a secular trend (HR per calendar year, 0.96 [0.95-0.96] and 0.96 [0.96-0.96]). When extrapolating to the UK population, we found that 3.8 million patients may be reclassified as eligible for statin prescription depending on the model used. A key limitation of this study was that we could not assess the variation in risk that may be caused by risk factors missing from the database (such as diet or physical activity).

CONCLUSIONS

Risk prediction models that use routinely collected data provide estimates strongly dependent on modelling decisions. Despite this large variability in patient risk, the models appear to perform similarly according to standard performance metrics. Decision-making should be supplemented with clinical judgement and evidence of additional risk factors. The largest source of variability, a secular trend in CVD incidence, can be accounted for and should be explored in more detail.

External validation of molecular subtype classifications of colorectal cancer based on microsatellite instability, CIMP, BRAF and KRAS

Background

Competing molecular classification systems have been proposed to complement the TNM staging system for a better prediction of survival in colorectal cancer (CRC). However, validation studies are so far lacking. The aim of this study was to validate and extend previously published molecular classifications of CRC in a large independent cohort of CRC patients.

Methods

CRC patients were recruited into a population-based cohort study (DACHS). Molecular subtypes were categorized based on three previously published classifications. Cox-proportional hazard models, based on the same set of patients and using the same confounders as reported by the original studies, were used to determine overall, cancer-specific, or relapse-free survival for each subtype. Hazard ratios and confidence intervals, as well as Kaplan-Meier plots were compared to those reported by the original studies.

Results

We observed similar patterns of worse survival for the microsatellite stable (MSS)/BRAF-mutated and MSS/KRAS-mutated subtypes in our validation analyses, which were included in two of the validated classifications. Of the two MSI subtypes, one defined by additional presence of CIMP-high and BRAF-mutation and the other by tumors negative for CIMP, BRAF and KRAS-mutations, we could not confirm associations with better prognosis as suggested by one of the classifications. For two of the published classifications, we were able to provide results for additional subgroups not included in the original studies (men, other disease stages, other locations).

Conclusions

External validation of three previously proposed classifications confirmed findings of worse survival for CRC patients with MSS subtypes and BRAF or KRAS mutations. Regarding MSI subtypes, other patient characteristics such as stage of the tumor, may influence the potential survival benefit. Further integration of methylation, genetic, and immunological information is needed to develop and validate a comprehensive classification that will have relevance for use in clinical practice.

Here to stay or go? Connecting turnover research to applied attrition modeling

Attrition modeling is a direct application of extant turnover research that can favorably impact workforce planning and action planning. However, while academic research enables practitioners insights into understanding turnover phenomena, there is no single document that comprehensively translates this work to give guidance as to the many practical decisions that must be made when modeling turnover, as well as how to apply psychological research to messier operational data. This focal article introduces and provides guidance on attrition modeling by outlining early considerations when planning a study, describing how to mesh theory with operational considerations when identifying turnover predictors within organizational settings, highlighting analytical strategies to model turnover, and considering how to appropriately share results. Collectively, this article serves as a guide to conducting attrition modeling within organizations and offers suggestions for future research to inform best practices.

Long-Term Exposure to Elevated Systolic Blood Pressure in Predicting Incident Cardiovascular Disease: Evidence From Large-Scale Routine Electronic Health Records

Background

How measures of long‐term exposure to elevated blood pressure might add to the performance of “current” blood pressure in predicting future cardiovascular disease is unclear. We compared incident cardiovascular disease risk prediction using past, current, and usual systolic blood pressure alone or in combination.

Methods and Results

Using data from UK primary care linked electronic health records, we applied a landmark cohort study design and identified 80 964 people, aged 50 years (derivation cohort=64 772; validation cohort=16 192), who, at study entry, had recorded blood pressure, no prior cardiovascular disease, and no previous antihypertensive or lipid‐lowering prescriptions. We used systolic blood pressure recorded up to 10 years before baseline to estimate past systolic blood pressure (mean, time‐weighted mean, and variability) and usual systolic blood pressure (correcting current values for past time‐dependent blood pressure fluctuations) and examined their prospective relation with incident cardiovascular disease (first hospitalization for or death from coronary heart disease or stroke/transient ischemic attack). We used Cox regression to estimate hazard ratios and applied Bayesian analysis within a machine learning framework in model development and validation. Predictive performance of models was assessed using discrimination (area under the receiver operating characteristic curve) and calibration metrics. We found that elevated past, current, and usual systolic blood pressure values were separately and independently associated with increased incident cardiovascular disease risk. When used alone, the hazard ratio (95% credible interval) per 20–mm Hg increase in current systolic blood pressure was 1.22 (1.18–1.30), but associations were stronger for past systolic blood pressure (mean and time‐weighted mean) and usual systolic blood pressure (hazard ratio ranging from 1.39–1.45). The area under the receiver operating characteristic curve for a model that included current systolic blood pressure, sex, smoking, deprivation, diabetes mellitus, and lipid profile was 0.747 (95% credible interval, 0.722–0.811). The addition of past systolic blood pressure mean, time‐weighted mean, or variability to this model increased the area under the receiver operating characteristic curve (95% credible interval) to 0.750 (0.727–0.811), 0.750 (0.726–0.811), and 0.748 (0.723–0.811), respectively, with all models showing good calibration. Similar small improvements in area under the receiver operating characteristic curve were observed when testing models on the validation cohort, in sex‐stratified analyses, or by using different landmark ages (40 or 60 years).

Conclusions

Using multiple blood pressure recordings from patients’ electronic health records showed stronger associations with incident cardiovascular disease than a single blood pressure measurement, but their addition to multivariate risk prediction models had negligible effects on model performance.

See Editorial Ahmad and Oparil

Comparing the predictive ability of the Revised Minimum Dataset Mortality Risk Index (MMRI-R) with nurses' predictions of mortality among frail older people: a cohort study

OBJECTIVES

to establish the accuracy of community nurses' predictions of mortality among older people with multiple long-term conditions, to compare these with a mortality rating index and to assess the incremental value of nurses' predictions to the prognostic tool.

DESIGN

a prospective cohort study using questionnaires to gather clinical information about patients case managed by community nurses. Nurses estimated likelihood of mortality for each patient on a 5-point rating scale. The dataset was randomly split into derivation and validation cohorts. Cox proportional hazard models were used to estimate risk equations for the Revised Minimum Dataset Mortality Risk Index (MMRI-R) and nurses' predictions of mortality individually and combined. Measures of discrimination and calibration were calculated and compared within the validation cohort.

SETTING

two NHS Trusts in England providing case-management services by nurses for frail older people with multiple long-term conditions.

PARTICIPANTS

867 patients on the caseload of 35 case-management nurses. 433 and 434 patients were assigned to the derivation and validation cohorts, respectively. Patients were followed up for 12 months.

RESULTS

249 patients died (28.72%). In the validation cohort, MMRI-R demonstrated good discrimination (Harrell's c-index 0.71) and nurses' predictions similar discrimination (Harrell's c-index 0.70). There was no evidence of superiority in performance of either method individually (P = 0.83) but the MMRI-R and nurses' predictions together were superior to nurses' predictions alone (P = 0.01).

CONCLUSIONS

patient mortality is associated with higher MMRI-R scores and nurses' predictions of 12-month mortality. The MMRI-R enhanced nurses' predictions and may improve nurses' confidence in initiating anticipatory care interventions.

Development of pre and post-operative models to predict early recurrence of hepatocellular carcinoma after surgical resection

BACKGROUND & AIMS

Resection is the most widely used potentially curative treatment for patients with early hepatocellular carcinoma (HCC). However, recurrence within 2 years occurs in 30-50% of patients, being the major cause of mortality. Herein, we describe 2 models, both based on widely available clinical data, which permit risk of early recurrence to be assessed before and after resection.

METHODS

A total of 3,903 patients undergoing surgical resection with curative intent were recruited from 6 different centres. We built 2 models for early recurrence, 1 using preoperative and 1 using pre and post-operative data, which were internally validated in the Hong Kong cohort. The models were then externally validated in European, Chinese and US cohorts. We developed 2 online calculators to permit easy clinical application.

RESULTS

Multivariable analysis identified male gender, large tumour size, multinodular tumour, high albumin-bilirubin (ALBI) grade and high serum alpha-fetoprotein as the key parameters related to early recurrence. Using these variables, a preoperative model (ERASL-pre) gave 3 risk strata for recurrence-free survival (RFS) in the entire cohort - low risk: 2-year RFS 64.8%, intermediate risk: 2-year RFS 42.5% and high risk: 2-year RFS 20.7%. Median survival in each stratum was similar between centres and the discrimination between the 3 strata was enhanced in the post-operative model (ERASL-post) which included 'microvascular invasion'.

CONCLUSIONS

Statistical models that can predict the risk of early HCC recurrence after resection have been developed, extensively validated and shown to be applicable in the international setting. Such models will be valuable in guiding surveillance follow-up and in the design of post-resection adjuvant therapy trials.

LAY SUMMARY

The most effective treatment of hepatocellular carcinoma is surgical removal of the tumour but there is often recurrence. In this large international study, we develop a statistical method that allows clinicians to estimate the risk of recurrence in an individual patient. This facility enhances communication with the patient about the likely success of the treatment and will help in designing clinical trials that aim to find drugs that decrease the risk of recurrence.

Breslow Density Is a Novel Prognostic Feature That Adds Value to Melanoma Staging

Histomorphologic prognostic biomarkers that can be measured using only an hematoxylin and eosin stain are very attractive because they are simple and cheap. We conceived an entirely novel biomarker of this type, the Breslow density (BD), which measures invasive melanoma cell density at the site where Breslow thickness (BT) is measured. This study assessed BD's prognostic value. In this study, BD was measured in 1329 melanoma patients. Measurement accuracy and precision was assessed using intraclass correlation coefficient (ICC). Survival was assessed with a primary end-point of melanoma-specific survival (MSS) and also overall survival and metastasis-free survival. We found that BD measurement was accurate compared with gold standard image analysis (ICC, 0.84). Precision was excellent for 3 observers with different experience (ICC, 0.93) and for an observer using only written instructions (ICC, 0.93). BD was a highly significant predictor in multivariable analysis for overall survival, MSS, and metastasis-free survival (each, P<0.001) and it explained MSS better than BT, but BT and BD together had best explanatory capability. A BD cut point of ≥65% was trained in 970 melanomas and validated in 359. This cut point showed promise as a novel way to upstage melanoma from T stage "a" to "b." BD was combined with BT to create a targeted burden score. This was a validated as an adjunct to American Joint Committee on Cancer stage. In summary, BD can be measured accurately and precisely. It demonstrated independent prognostic value and explained MSS better than BT alone. Notably, we demonstrated ways that BD could be used with American Joint Committee on Cancer version 8 staging.