Lessons learnt when accounting for competing events in the external validation of time-to-event prognostic models

Development of a risk prediction model for subsequent infection after colonization with carbapenem-resistant Enterobacterales: a retrospective cohort study

BACKGROUND

Colonization of carbapenem-resistant Enterobacterale (CRE) is considered as one of vital preconditions for infection, with corresponding high morbidity and mortality. It is important to construct a reliable prediction model for those CRE carriers with high risk of infection.

METHODS

A retrospective cohort study was conducted in two Chinese tertiary hospitals for patients with CRE colonization from 2011 to 2021. Univariable analysis and the Fine-Gray sub-distribution hazard model were utilized to identify potential predictors for CRE-colonized infection, while death was the competing event. A nomogram was established to predict 30-day and 60-day risk of CRE-colonized infection.

RESULTS

879 eligible patients were enrolled in our study and divided into training (n = 761) and validation (n = 118) group, respectively. There were 196 (25.8%) patients suffered from subsequent CRE infection. The median duration of subsequent infection after identification of CRE colonization was 20 (interquartile range [IQR], 14-32) days. Multisite colonization, polymicrobial colonization, catheterization and receiving albumin after colonization, concomitant respiratory diseases, receiving carbapenems and antimicrobial combination therapy before CRE colonization within 90 days were included in final model. Model discrimination and calibration were acceptable for predicting the probability of 60-day CRE-colonized infection in both training (area under the curve [AUC], 74.7) and validation dataset (AUC, 81.1). Decision-curve analysis revealed a significantly better net benefit in current model. Our prediction model is freely available online at https://ken-zheng.shinyapps.io/PredictingModelofCREcolonizedInfection/ .

CONCLUSIONS

Our nomogram has a good predictive performance and could contribute to early identification of CRE carriers with a high-risk of subsequent infection, although external validation would be required.

Predicting the risks of kidney failure and death in adults with moderate to severe chronic kidney disease: multinational, longitudinal, population based, cohort study

OBJECTIVE

To train and test a super learner strategy for risk prediction of kidney failure and mortality in people with incident moderate to severe chronic kidney disease (stage G3b to G4).

DESIGN

Multinational, longitudinal, population based, cohort study.

SETTINGS

Linked population health data from Canada (training and temporal testing), and Denmark and Scotland (geographical testing).

PARTICIPANTS

People with newly recorded chronic kidney disease at stage G3b-G4, estimated glomerular filtration rate (eGFR) 15-44 mL/min/1.73 m2.

MODELLING

The super learner algorithm selected the best performing regression models or machine learning algorithms (learners) based on their ability to predict kidney failure and mortality with minimised cross-validated prediction error (Brier score, the lower the better). Prespecified learners included age, sex, eGFR, albuminuria, with or without diabetes, and cardiovascular disease. The index of prediction accuracy, a measure of calibration and discrimination calculated from the Brier score (the higher the better) was used to compare KDpredict with the benchmark, kidney failure risk equation, which does not account for the competing risk of death, and to evaluate the performance of KDpredict mortality models.

RESULTS

67 942 Canadians, 17 528 Danish, and 7740 Scottish residents with chronic kidney disease at stage G3b to G4 were included (median age 77-80 years; median eGFR 39 mL/min/1.73 m2). Median follow-up times were five to six years in all cohorts. Rates were 0.8-1.1 per 100 person years for kidney failure and 10-12 per 100 person years for death. KDpredict was more accurate than kidney failure risk equation in prediction of kidney failure risk: five year index of prediction accuracy 27.8% (95% confidence interval 25.2% to 30.6%) versus 18.1% (15.7% to 20.4%) in Denmark and 30.5% (27.8% to 33.5%) versus 14.2% (12.0% to 16.5%) in Scotland. Predictions from kidney failure risk equation and KDpredict differed substantially, potentially leading to diverging treatment decisions. An 80-year-old man with an eGFR of 30 mL/min/1.73 m2 and an albumin-to-creatinine ratio of 100 mg/g (11 mg/mmol) would receive a five year kidney failure risk prediction of 10% from kidney failure risk equation (above the current nephrology referral threshold of 5%). The same man would receive five year risk predictions of 2% for kidney failure and 57% for mortality from KDpredict. Individual risk predictions from KDpredict with four or six variables were accurate for both outcomes. The KDpredict models retrained using older data provided accurate predictions when tested in temporally distinct, more recent data.

CONCLUSIONS

KDpredict could be incorporated into electronic medical records or accessed online to accurately predict the risks of kidney failure and death in people with moderate to severe CKD. The KDpredict learning strategy is designed to be adapted to local needs and regularly revised over time to account for changes in the underlying health system and care processes.

Predictive models in chronic kidney disease: essential tools in clinical practice

PURPOSE OF REVIEW

The integration of risk prediction in managing chronic kidney disease (CKD) is universally considered a key point of routine clinical practice to guide time-sensitive choices, such as dialysis access planning or counseling on kidney transplant options. Several prognostic models have been developed and validated to provide individualized evaluation of kidney failure risk in CKD patients. This review aims to analyze the current evidence on existing predictive models and evaluate the different advantages and disadvantages of these tools.

RECENT FINDINGS

Since Tangri et al. introduced the Kidney Failure Risk Equation in 2011, the nephrological scientific community focused its interest in enhancing available algorithms and finding new prognostic equations. Although current models can predict kidney failure with high discrimination, different questions remain unsolved. Thus, this field is open to new possibilities and discoveries.

SUMMARY

Accurately informing patients of their prognoses can result in tailored therapy with important clinical and psychological implications. Over the last 5 years, the number of disease-modifying therapeutic options has considerably increased, providing possibilities to not only prevent the kidney failure onset in patients with advanced CKD but also delay progression from early stages in at-risk individuals.

Multicentre external validation of the prognostic model kidney failure risk equation in patients with CKD stages 3 and 4 in Peru: a retrospective cohort study

OBJECTIVES

To externally validate the four-variable kidney failure risk equation (KFRE) in the Peruvian population for predicting kidney failure at 2 and 5 years.

DESIGN

A retrospective cohort study.

SETTING

17 primary care centres from the Health's Social Security of Peru.

PARTICIPANTS

Patients older than 18 years, diagnosed with chronic kidney disease stage 3a-3b-4 and 3b-4, between January 2013 and December 2017. Patients were followed until they developed kidney failure, died, were lost, or ended the study (31 December 2019), whichever came first.

PRIMARY AND SECONDARY OUTCOME MEASURES

Performance of the KFRE model was assessed based on discrimination and calibration measures considering the competing risk of death.

RESULTS

We included 7519 patients in stages 3a-4 and 2798 patients in stages 3b-4. The estimated cumulative incidence of kidney failure, accounting for competing event of death, at 2 years and 5 years, was 1.52% and 3.37% in stages 3a-4 and 3.15% and 6.86% in stages 3b-4. KFRE discrimination at 2 and 5 years was high, with time-dependent area under the curve and C-index >0.8 for all populations. Regarding calibration in-the-large, the observed to expected ratio and the calibration intercept indicated that KFRE underestimates the overall risk at 2 years and overestimates it at 5 years in all populations.

CONCLUSIONS

The four-variable KFRE models have good discrimination but poor calibration in the Peruvian population. The model underestimates the risk of kidney failure in the short term and overestimates it in the long term. Further research should focus on updating or recalibrating the KFRE model to better predict kidney failure in the Peruvian context before recommending its use in clinical practice.

Around the EQUATOR with Clin-STAR: Prediction modeling opportunities and challenges in aging research

The 2015 Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement was published to improve reporting transparency for prediction modeling studies. The objective of this review is to highlight methodologic challenges that aging-focused researchers will encounter when designing and reporting studies involving prediction models for older adults and provide guidance for addressing these challenges. In following the 22-item TRIPOD checklist, researchers must consider the representativeness of cohorts used (e.g., whether older adults with frailty, cognitive impairment, and social isolation were included), strategies for incorporating common geriatric predictors (e.g., age, comorbidities, functional status, and frailty), methods for handling missing data and competing risk of death, and assessment of model performance heterogeneity across important subgroups (e.g., age, sex, race, and ethnicity). We provide guidance to help aging-focused researchers develop, validate, and report models that can inform and improve patient care, which we label "TRIPOD-65."

An external validation of the Kidney Donor Risk Index in the UK transplant population in the presence of semi-competing events

BACKGROUND

Transplantation represents the optimal treatment for many patients with end-stage kidney disease. When a donor kidney is available to a waitlisted patient, clinicians responsible for the care of the potential recipient must make the decision to accept or decline the offer based upon complex and variable information about the donor, the recipient and the transplant process. A clinical prediction model may be able to support clinicians in their decision-making. The Kidney Donor Risk Index (KDRI) was developed in the United States to predict graft failure following kidney transplantation. The survival process following transplantation consists of semi-competing events where death precludes graft failure, but not vice-versa.

METHODS

We externally validated the KDRI in the UK kidney transplant population and assessed whether validation under a semi-competing risks framework impacted predictive performance. Additionally, we explored whether the KDRI requires updating. We included 20,035 adult recipients of first, deceased donor, single, kidney-only transplants between January 1, 2004, and December 31, 2018, collected by the UK Transplant Registry and held by NHS Blood and Transplant. The outcomes of interest were 1- and 5-year graft failure following transplantation. In light of the semi-competing events, recipient death was handled in two ways: censoring patients at the time of death and modelling death as a competing event. Cox proportional hazard models were used to validate the KDRI when censoring graft failure by death, and cause-specific Cox models were used to account for death as a competing event.

RESULTS

The KDRI underestimated event probabilities for those at higher risk of graft failure. For 5-year graft failure, discrimination was poorer in the semi-competing risks model (0.625, 95% CI 0.611 to 0.640;0.611, 95% CI 0.597 to 0.625), but predictions were more accurate (Brier score 0.117, 95% CI 0.112 to 0.121; 0.114, 95% CI 0.109 to 0.118). Calibration plots were similar regardless of whether the death was modelled as a competing event or not. Updating the KDRI worsened calibration, but marginally improved discrimination.

CONCLUSIONS

Predictive performance for 1-year graft failure was similar between death-censored and competing event graft failure, but differences appeared when predicting 5-year graft failure. The updated index did not have superior performance and we conclude that updating the KDRI in the present form is not required.

Using the kidney failure risk equation to predict end-stage kidney disease in CKD patients of South Asian ethnicity: an external validation study

BACKGROUND

The kidney failure risk equation (KFRE) predicts the 2- and 5-year risk of needing kidney replacement therapy (KRT) using four risk factors - age, sex, urine albumin-to-creatinine ratio (ACR) and creatinine-based estimated glomerular filtration rate (eGFR). Although the KFRE has been recalibrated in a UK cohort, this did not consider minority ethnic groups. Further validation of the KFRE in different ethnicities is a research priority. The KFRE also does not consider the competing risk of death, which may lead to overestimation of KRT risk. This study externally validates the KFRE for patients of South Asian ethnicity and compares methods for accounting for ethnicity and the competing event of death.

METHODS

Data were gathered from an established UK cohort containing 35,539 individuals diagnosed with chronic kidney disease. The KFRE was externally validated and updated in several ways taking into account ethnicity, using recognised methods for time-to-event data, including the competing risk of death. A clinical impact assessment compared the updated models through consideration of referrals made to secondary care.

RESULTS

The external validation showed the risk of KRT differed by ethnicity. Model validation performance improved when incorporating ethnicity and its interactions with ACR and eGFR as additional risk factors. Furthermore, accounting for the competing risk of death improved prediction. Using criteria of 5 years ≥ 5% predicted KRT risk, the competing risks model resulted in an extra 3 unnecessary referrals (0.59% increase) but identified an extra 1 KRT case (1.92% decrease) compared to the previous best model. Hybrid criteria of predicted risk using the competing risks model and ACR ≥ 70 mg/mmol should be used in referrals to secondary care.

CONCLUSIONS

The accuracy of KFRE prediction improves when updated to consider South Asian ethnicity and to account for the competing risk of death. This may reduce unnecessary referrals whilst identifying risks of KRT and could further individualise the KFRE and improve its clinical utility. Further research should consider other ethnicities.

Utility of the Kidney Failure Risk Equation and Estimated GFR for Estimating Time to Kidney Failure in Advanced CKD

RATIONALE & OBJECTIVE

The Kidney Failure Risk Equation (KFRE) predicts the 2-year risk of kidney failure for patients with chronic kidney disease (CKD). Translating KFRE-predicted risk or estimated glomerular filtration rate (eGFR) into time to kidney failure could inform decision making for patients approaching kidney failure.

STUDY DESIGN

Retrospective cohort.

SETTING & PARTICIPANTS

CKD Outcomes and Practice Patterns Study (CKDOPPS) cohort of patients with an eGFR<60mL/min/1.73m2 from 34 US nephrology practices (2013-2021).

EXPOSURE

2-year KFRE risk or eGFR.

OUTCOME

Kidney failure defined as initiation of dialysis or kidney transplantation.

ANALYTICAL APPROACH

Accelerated failure time (Weibull) models used to estimate the median, 25th, and 75th percentile times to kidney failure starting from KFRE values of 20%, 40%, and 50%, and from eGFR values of 20, 15, and 10mL/min/1.73m2. We examined variability in time to kidney failure by age, sex, race, diabetes status, albuminuria, and blood pressure.

RESULTS

Overall, 1,641 participants were included (mean age 69±13 years; median eGFR of 28mL/min/1.73m2 [IQR 20-37mL/min/1.73 m2]). Over a median follow-up period of 19 months (IQR, 12-30 months), 268 participants developed kidney failure, and 180 died before reaching kidney failure. The median estimated time to kidney failure was widely variable across patient characteristics from an eGFR of 20mL/min/1.73m2 and was shorter for younger age, male sex, Black (versus non-Black), diabetes (vs no diabetes), higher albuminuria, and higher blood pressure. Estimated times to kidney failure were comparably less variable across these characteristics for KFRE thresholds and eGFR of 15 or 10mL/min/1.73m2.

LIMITATIONS

Inability to account for competing risks when estimating time to kidney failure.

CONCLUSIONS

Among those with eGFR<15mL/min/1.73m2 or KFRE risk>40%), both KFRE risk and eGFR showed similar relationships with time to kidney failure. Our results demonstrate that estimating time to kidney failure in advanced CKD can inform clinical decisions and patient counseling on prognosis, regardless of whether estimates are based on eGFR or the KFRE.

PLAIN-LANGUAGE SUMMARY

Clinicians often talk to patients with advanced chronic kidney disease about the level of kidney function expressed as the estimated glomerular filtration rate (eGFR) and about the risk of developing kidney failure, which can be estimated using the Kidney Failure Risk Equation (KFRE). In a cohort of patients with advanced chronic kidney disease, we examined how eGFR and KFRE risk predictions corresponded to the time patients had until reaching kidney failure. Among those with eGFR<15mL/min/1.73m2 or KFRE risk > 40%), both KFRE risk and eGFR showed similar relationships with time to kidney failure. Estimating time to kidney failure in advanced CKD using either eGFR or KFRE can inform clinical decisions and patient counseling on prognosis.

Validation of a Salivary miRNA Signature of Endometriosis - Interim Data

Salivary miRNA Signature of EndometriosisThis interim analysis of the prospective, multicenter, external validation ENDOmiRNA Saliva Test study, confirms the diagnostic performance and reproducibility of the saliva miRNA signature for endometriosis. At a population prevalence of ∼80%, the miRNA signature had a sensitivity of 96.2%, specificity of 95.1%, and area under the curve of 0.96.

External validation and updating of prediction models of bleeding risk in patients with cancer receiving anticoagulants

OBJECTIVE

Patients with cancer are at increased bleeding risk, and anticoagulants increase this risk even more. Yet, validated bleeding risk models for prediction of bleeding risk in patients with cancer are lacking. The aim of this study is to predict bleeding risk in anticoagulated patients with cancer.

METHODS

We performed a study using the routine healthcare database of the Julius General Practitioners' Network. Five bleeding risk models were selected for external validation. Patients with a new cancer episode during anticoagulant treatment or those initiating anticoagulation during active cancer were included. The outcome was the composite of major bleeding and clinically relevant non-major (CRNM) bleeding. Next, we internally validated an updated bleeding risk model accounting for the competing risk of death.

RESULTS

The validation cohort consisted of 1304 patients with cancer, mean age 74.0±10.9 years, 52.2% males. In total 215 (16.5%) patients developed a first major or CRNM bleeding during a mean follow-up of 1.5 years (incidence rate; 11.0 per 100 person-years (95% CI 9.6 to 12.5)). The c-statistics of all selected bleeding risk models were low, around 0.56. Internal validation of an updated model accounting for death as competing risk showed a slightly improved c-statistic of 0.61 (95% CI 0.54 to 0.70). On updating, only age and a history of bleeding appeared to contribute to the prediction of bleeding risk.

CONCLUSIONS

Existing bleeding risk models cannot accurately differentiate bleeding risk between patients. Future studies may use our updated model as a starting point for further development of bleeding risk models in patients with cancer.

Removing race from the CKD-EPI equation and its impact on prognosis in a predominantly White European population

BACKGROUND

While American nephrology societies recommend using the 2021 Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) estimated glomerular filtration rate (eGFR) equation without a Black race coefficient, it is unknown how this would impact disease distribution, prognosis and kidney failure risk prediction in predominantly White non-US populations.

METHODS

We studied 1.6 million Stockholm adults with serum/plasma creatinine measurements between 2007 and 2019. We calculated changes in eGFR and reclassification across KDIGO GFR categories when changing from the 2009 to 2021 CKD-EPI equation; estimated associations between eGFR and the clinical outcomes kidney failure with replacement therapy (KFRT), (cardiovascular) mortality and major adverse cardiovascular events using Cox regression; and investigated prognostic accuracy (discrimination and calibration) of both equations within the Kidney Failure Risk Equation.

RESULTS

Compared with the 2009 equation, the 2021 equation yielded a higher eGFR by a median [interquartile range (IQR)] of 3.9 (2.9-4.8) mL/min/1.73 m2, which was larger at older age and for men. Consequently, 9.9% of the total population and 36.2% of the population with CKD G3a-G5 was reclassified to a higher eGFR category. Reclassified individuals exhibited a lower risk of KFRT, but higher risks of all-cause/cardiovascular death and major adverse cardiovascular events, compared with non-reclassified participants of similar eGFR. eGFR by both equations strongly predicted study outcomes, with equal discrimination and calibration for the Kidney Failure Risk Equation.

CONCLUSIONS

Implementing the 2021 CKD-EPI equation in predominantly White European populations would raise eGFR by a modest amount (larger at older age and in men) and shift a major proportion of CKD patients to a higher eGFR category. eGFR by both equations strongly predicted outcomes.