How to develop a more accurate risk prediction model when there are few events

Anxiety after completion of treatment for early-stage breast cancer: a systematic review to identify candidate predictors and evaluate multivariable model development

Purpose

The purpose of this review was to identify potential candidate predictors of anxiety in women with early-stage breast cancer (BC) after adjuvant treatments and evaluate methodological development of existing multivariable models to inform the future development of a predictive risk stratification model (PRSM).

Methods

Databases (MEDLINE, Web of Science, CINAHL, CENTRAL and PsycINFO) were searched from inception to November 2015. Eligible studies were prospective, recruited women with stage 0–3 BC, used a validated anxiety outcome ≥3 months post-treatment completion and used multivariable prediction models. Internationally accepted quality standards were used to assess predictive risk of bias and strength of evidence.

Results

Seven studies were identified: five were observational cohorts and two secondary analyses of RCTs. Variability of measurement and selective reporting precluded meta-analysis. Twenty-one candidate predictors were identified in total. Younger age and previous mental health problems were identified as risk factors in ≥3 studies. Clinical variables (e.g. treatment, tumour grade) were not identified as predictors in any studies. No studies adhered to all quality standards.

Conclusions

Pre-existing vulnerability to mental health problems and younger age increased the risk of anxiety after completion of treatment for BC survivors, but there was no evidence that chemotherapy was a predictor. Multiple predictors were identified but many lacked reproducibility or were not measured across studies, and inadequate reporting did not allow full evaluation of the multivariable models. The use of quality standards in the development of PRSM within supportive cancer care would improve model quality and performance, thereby allowing professionals to better target support for patients.

Electronic supplementary material

The online version of this article (doi:10.1007/s00520-017-3688-6) contains supplementary material, which is available to authorized users.

Predicting the onset of hazardous alcohol drinking in primary care: development and validation of a simple risk algorithm

BACKGROUND

Little is known about the risk of progressing to hazardous alcohol use in abstinent or low-risk drinkers.

AIM

To develop and validate a simple brief risk algorithm for the onset of hazardous alcohol drinking (HAD) over 12 months for use in primary care.

DESIGN AND SETTING

Prospective cohort study in 32 health centres from six Spanish provinces, with evaluations at baseline, 6 months, and 12 months.

METHOD

Forty-one risk factors were measured and multilevel logistic regression and inverse probability weighting were used to build the risk algorithm. The outcome was new occurrence of HAD during the study, as measured by the AUDIT.

RESULTS

From the lists of 174 GPs, 3954 adult abstinent or low-risk drinkers were recruited. The 'predictAL-10' risk algorithm included just nine variables (10 questions): province, sex, age, cigarette consumption, perception of financial strain, having ever received treatment for an alcohol problem, childhood sexual abuse, AUDIT-C, and interaction AUDIT-C*Age. The c-index was 0.886 (95% CI = 0.854 to 0.918). The optimal cutoff had a sensitivity of 0.83 and specificity of 0.80. Excluding childhood sexual abuse from the model (the 'predictAL-9'), the c-index was 0.880 (95% CI = 0.847 to 0.913), sensitivity 0.79, and specificity 0.81. There was no statistically significant difference between the c-indexes of predictAL-10 and predictAL-9.

CONCLUSION

The predictAL-10/9 is a simple and internally valid risk algorithm to predict the onset of hazardous alcohol drinking over 12 months in primary care attendees; it is a brief tool that is potentially useful for primary prevention of hazardous alcohol drinking.

Normal Tissue Complication Probability (NTCP) Modelling of Severe Acute Mucositis using a Novel Oral Mucosal Surface Organ at Risk

AIMS

A normal tissue complication probability (NTCP) model of severe acute mucositis would be highly useful to guide clinical decision making and inform radiotherapy planning. We aimed to improve upon our previous model by using a novel oral mucosal surface organ at risk (OAR) in place of an oral cavity OAR.

MATERIALS AND METHODS

Predictive models of severe acute mucositis were generated using radiotherapy dose to the oral cavity OAR or mucosal surface OAR and clinical data. Penalised logistic regression and random forest classification (RFC) models were generated for both OARs and compared. Internal validation was carried out with 100-iteration stratified shuffle split cross-validation, using multiple metrics to assess different aspects of model performance. Associations between treatment covariates and severe mucositis were explored using RFC feature importance.

RESULTS

Penalised logistic regression and RFC models using the oral cavity OAR performed at least as well as the models using mucosal surface OAR. Associations between dose metrics and severe mucositis were similar between the mucosal surface and oral cavity models. The volumes of oral cavity or mucosal surface receiving intermediate and high doses were most strongly associated with severe mucositis.

CONCLUSIONS

The simpler oral cavity OAR should be preferred over the mucosal surface OAR for NTCP modelling of severe mucositis. We recommend minimising the volume of mucosa receiving intermediate and high doses, where possible.

Risk prediction of pulmonary tuberculosis using genetic and conventional risk factors in adult Korean population

A complex interplay among host, pathogen, and environmental factors is believed to contribute to the risk of developing pulmonary tuberculosis (PTB). The lack of replication of published genome-wide association study (GWAS) findings limits the clinical utility of reported single nucleotide polymorphisms (SNPs). We conducted a GWAS using 467 PTB cases and 1,313 healthy controls obtained from two community-based cohorts in Korea. We evaluated the performance of PTB risk models based on different combinations of genetic and nongenetic factors and validated the results in an independent Korean population comprised of 179 PTB cases and 500 healthy controls. We demonstrated the polygenic nature of PTB and nongenetic factors such as age, sex, and body mass index (BMI) were strongly associated with PTB risk. None of the SNPs achieved genome-wide significance; instead, we were able to replicate the associations between PTB and ten SNPs near or in the genes, CDCA7, GBE1, GADL1, SPATA16, C6orf118, KIAA1432, DMRT2, CTR9, CCDC67, and CDH13, which may play roles in the immune and inflammatory pathways. Among the replicated SNPs, an intergenic SNP, rs9365798, located downstream of the C6orf118 gene showed the most significant association under the dominant model (OR = 1.59, 95% CI 1.32–1.92, P = 2.1×10⁻⁶). The performance of a risk model combining the effects of ten replicated SNPs and six nongenetic factors (i.e., age, sex, BMI, cigarette smoking, systolic blood pressure, and hemoglobin) were validated in the replication set (AUC = 0.80, 95% CI 0.76–0.84). The strategy of combining genetic and nongenetic risk factors ultimately resulted in better risk prediction for PTB in the adult Korean population.

A scoring system to predict breast cancer mortality at 5 and 10 years

Although predictive models exist for mortality in breast cancer (BC) (generally all cause-mortality), they are not applicable to all patients and their statistical methodology is not the most powerful to develop a predictive model. Consequently, we developed a predictive model specific for BC mortality at 5 and 10 years resolving the above issues. This cohort study included 287 patients diagnosed with BC in a Spanish region in 2003–2016. Main outcome variable: time-to-BC death. Secondary variables: age, personal history of breast surgery, personal history of any cancer/BC, premenopause, postmenopause, grade, estrogen receptor, progesterone receptor, c-erbB2, TNM stage, multicentricity/multifocality, diagnosis and treatment. A points system was constructed to predict BC mortality at 5 and 10 years. The model was internally validated by bootstrapping. The points system was integrated into a mobile application for Android. Mean follow-up was 8.6 ± 3.5 years and 55 patients died of BC. The points system included age, personal history of BC, grade, TNM stage and multicentricity. Validation was satisfactory, in both discrimination and calibration. In conclusion, we constructed and internally validated a scoring system for predicting BC mortality at 5 and 10 years. External validation studies are needed for its use in other geographical areas.

A Clinical Prediction Score to Guide Referral of Elderly Dialysis Patients for Kidney Transplant Evaluation

INTRODUCTION

Dialysis patients aged ≥70 years derive improved life expectancy through kidney transplantation compared to their waitlisted counterparts, but guidelines are not clear about how to identify appropriate transplantation candidates. We developed a clinical prediction score to identify elderly dialysis patients with expected 5-year survival appropriate for kidney transplantation (>5 years).

METHODS

Incident dialysis patients in 2006-2009 aged ≥70 were identified from the United States Renal Data System database and divided into derivation and validation cohorts. Using the derivation cohort, candidate variables with a significant crude association with 5-year all-cause mortality were included in a multivariable logistic regression model to generate a scoring system. The scoring system was tested in the validation cohort and a cohort of elderly transplant recipients.

RESULTS

Characteristics most predictive of 5-year mortality included age >80, body mass index (BMI) <18, the presence of congestive heart failure (CHF), chronic obstructive pulmonary disease (COPD), immobility, and being institutionalized. Factors associated with increased 5-year survival were non-white race, a primary cause of end stage renal disease (ESRD) other than diabetes, employment within 6 months of dialysis initiation, and dialysis start via arteriovenous fistula (AVF). 5-year mortality was 47% for the lowest risk score group (3.6% of the validation cohort) and >90% for the highest risk cohort (42% of the validation cohort).

CONCLUSION

This clinical prediction score could be useful for physicians to identify potentially suitable candidates for kidney transplantation.

Enhancing risk stratification for use in integrated care: a cluster analysis of high-risk patients in a retrospective cohort study

OBJECTIVE

To show how segmentation can enhance risk stratification tools for integrated care, by providing insight into different care usage patterns within the high-risk population.

DESIGN

A retrospective cohort study. A risk score was calculated for each person using a logistic regression, which was then used to select the top 5% high-risk individuals. This population was segmented based on the usage of different care settings using a k-means cluster analysis. Data from 2008 to 2011 were used to create the risk score and segments, while 2012 data were used to understand the predictive abilities of the models.

SETTING AND PARTICIPANTS

Data were collected from administrative data sets covering primary and secondary care for a random sample of 300 000 English patients.

MAIN MEASURES

The high-risk population was segmented based on their usage of 4 different care settings: emergency acute care, elective acute care, outpatient care and GP care.

RESULTS

While the risk strata predicted care usage at a high level, within the high-risk population, usage varied significantly. 4 different groups of high-risk patients could be identified. These 4 segments had distinct usage patterns across care settings, reflecting different levels and types of care needs. The 2008-2011 usage patterns of the 4 segments were consistent with the 2012 patterns.

DISCUSSION

Cluster analyses revealed that the high-risk population is not homogeneous, as there exist 4 groups of patients with different needs across the care continuum. Since the patterns were predictive of future care use, they can be used to develop integrated care programmes tailored to these different groups.

CONCLUSIONS

Usage-based segmentation augments risk stratification by identifying patient groups with different care needs, around which integrated care programmes can be designed.

Development of a prognostic model based on demographic, environmental and lifestyle information for predicting incidences of symptomatic respiratory or gastrointestinal infection in adult office workers

Background

Occurrence of respiratory tract infection (RTI) or gastrointestinal tract infection (GTI) is known to vary between individuals and may be a confounding factor in the analysis of the results of intervention trials. We aimed at developing a prognostic model for predicting individual incidences of RTI and GTI on the basis of data collected in a hand-hygiene intervention trial among adult office workers, and comprising a prior-to-onset questionnaire on potential infection-risk factors and weekly electronic follow-up reports on occurrence of symptoms of, and on exposures to RTI or GTI.

Methods

A mixed-effect negative binomial regression model was used to calculate a predictor-specific incidence rate ratio for each questionnaire variable and for each of the four endpoints, and predicted individual incidences for symptoms of and exposures to RTI and GTI. In the fitting test these were then compared with the observed incidences.

Results

Out of 1270 eligible employees of six enterprises, 683 volunteered to participate in the trial. Ninety-two additional participants were recruited during the follow-up. Out of the 775 registered participants, 717 returned the questionnaire with data on potential predictor variables and follow-up reports for determination of outcomes. Age and gender were the strongest predictors of both exposure to, and symptoms of RTI or GTI, although no gender difference was seen in the RTI incidence. In addition, regular use of public transport, and history of seasonal influenza vaccination increased the risk of RTI. The individual incidence values predicted by the model showed moderate correlation with those observed in each of the four categories. According to the Cox-Snell multivariate formula the model explained 11.2% of RTI and 3.3% of GTI incidences. Resampling revealed mean and 90% confidence interval values of 10.9 (CI 6.9–14.5)% for RTI and 2.4 (0.6–4.4)% for GTI.

Conclusion

The model created explained a relatively small proportion of the occurrence of RTI or GTI. Unpredictable exposure to disease agents, and individual susceptibility factors are likely to be key determinants of disease emergence. Yet, the model might be useful in prerandomization stratification of study population in RTI intervention trials where the expected difference between trial arms is relatively small.

Trial registration

Registered at ClinicalTrials.gov with Identifier NCT00821509 on 12 March 2009.

Electronic supplementary material

The online version of this article (doi:10.1186/s13063-016-1668-7) contains supplementary material, which is available to authorized users.

Functional Data Analysis Applied to Modeling of Severe Acute Mucositis and Dysphagia Resulting From Head and Neck Radiation Therapy

Purpose

Current normal tissue complication probability modeling using logistic regression suffers from bias and high uncertainty in the presence of highly correlated radiation therapy (RT) dose data. This hinders robust estimates of dose-response associations and, hence, optimal normal tissue—sparing strategies from being elucidated. Using functional data analysis (FDA) to reduce the dimensionality of the dose data could overcome this limitation.

Methods and Materials

FDA was applied to modeling of severe acute mucositis and dysphagia resulting from head and neck RT. Functional partial least squares regression (FPLS) and functional principal component analysis were used for dimensionality reduction of the dose-volume histogram data. The reduced dose data were input into functional logistic regression models (functional partial least squares—logistic regression [FPLS-LR] and functional principal component—logistic regression [FPC-LR]) along with clinical data. This approach was compared with penalized logistic regression (PLR) in terms of predictive performance and the significance of treatment covariate—response associations, assessed using bootstrapping.

Results

The area under the receiver operating characteristic curve for the PLR, FPC-LR, and FPLS-LR models was 0.65, 0.69, and 0.67, respectively, for mucositis (internal validation) and 0.81, 0.83, and 0.83, respectively, for dysphagia (external validation). The calibration slopes/intercepts for the PLR, FPC-LR, and FPLS-LR models were 1.6/−0.67, 0.45/0.47, and 0.40/0.49, respectively, for mucositis (internal validation) and 2.5/−0.96, 0.79/−0.04, and 0.79/0.00, respectively, for dysphagia (external validation). The bootstrapped odds ratios indicated significant associations between RT dose and severe toxicity in the mucositis and dysphagia FDA models. Cisplatin was significantly associated with severe dysphagia in the FDA models. None of the covariates was significantly associated with severe toxicity in the PLR models. Dose levels greater than approximately 1.0 Gy/fraction were most strongly associated with severe acute mucositis and dysphagia in the FDA models.

Conclusions

FPLS and functional principal component analysis marginally improved predictive performance compared with PLR and provided robust dose-response associations. FDA is recommended for use in normal tissue complication probability modeling.

Early Response in Cellulitis: A Prospective Study of Dynamics and Predictors

In this prospective study of cellulitis, several nonpharmacological factors were associated with lack of early response. Such early nonresponse was rarely related to inappropriate therapy but strongly predictive of early treatment escalation, suggesting that broadening antibiotic treatment often may be premature.

Background. Skin and soft tissue infections are common reasons for medical care. Use of broad-spectrum therapy and costs have increased. Assessment of early treatment response has been given a central role both in clinical trials and everyday practice. However, there is a paucity of data on the dynamics of response, causes of early nonresponse, and how early nonresponse affects resource use and predicts outcome.

Methods. We prospectively enrolled 216 patients hospitalized with cellulitis. Clinical and biochemical response data during the first 3 days of treatment were analyzed in relation to baseline factors, antibiotic use, surgery, and outcome. Multivariable analysis included logistic lasso regression.

Results. Clinical or biochemical response was observed in the majority of patients the day after treatment initiation. Concordance between clinical and biochemical response was strongest at days 2 and 3. Female sex, cardiovascular disease, higher body mass index, shorter duration of symptoms, and cellulitis other than typical erysipelas were predictors of nonresponse at day 3. In contrast, baseline factors were not predictive of clinical failure assessed posttreatment. Among cases with antibiotic treatment escalation by day 2, 90% (37/41) had nonresponse at day 1, but only 5% (2/40) had inappropriate initial therapy. Nonresponse at day 3 was a predictor of treatment duration >14 days, but not of clinical failure.

Conclusions. Nonpharmacological factors had a major impact on early response dynamics. Delayed response was rarely related to inappropriate therapy but strongly predictive of early treatment escalation, suggesting that broadening antibiotic treatment may often be premature.

Normal tissue complication probability (NTCP) modelling using spatial dose metrics and machine learning methods for severe acute oral mucositis resulting from head and neck radiotherapy

BACKGROUND AND PURPOSE

Severe acute mucositis commonly results from head and neck (chemo)radiotherapy. A predictive model of mucositis could guide clinical decision-making and inform treatment planning. We aimed to generate such a model using spatial dose metrics and machine learning.

MATERIALS AND METHODS

Predictive models of severe acute mucositis were generated using radiotherapy dose (dose-volume and spatial dose metrics) and clinical data. Penalised logistic regression, support vector classification and random forest classification (RFC) models were generated and compared. Internal validation was performed (with 100-iteration cross-validation), using multiple metrics, including area under the receiver operating characteristic curve (AUC) and calibration slope, to assess performance. Associations between covariates and severe mucositis were explored using the models.

RESULTS

The dose-volume-based models (standard) performed equally to those incorporating spatial information. Discrimination was similar between models, but the RFCstandard had the best calibration. The mean AUC and calibration slope for this model were 0.71 (s.d.=0.09) and 3.9 (s.d.=2.2), respectively. The volumes of oral cavity receiving intermediate and high doses were associated with severe mucositis.

CONCLUSIONS

The RFCstandard model performance is modest-to-good, but should be improved, and requires external validation. Reducing the volumes of oral cavity receiving intermediate and high doses may reduce mucositis incidence.

The Development of Statistical Models for Predicting Surgical Site Infections in Japan: Toward a Statistical Model–Based Standardized Infection Ratio

OBJECTIVE

To develop and internally validate a surgical site infection (SSI) prediction model for Japan.

DESIGN

Retrospective observational cohort study.

METHODS

We analyzed surveillance data submitted to the Japan Nosocomial Infections Surveillance system for patients who had undergone target surgical procedures from January 1, 2010, through December 31, 2012. Logistic regression analyses were used to develop statistical models for predicting SSIs. An SSI prediction model was constructed for each of the procedure categories by statistically selecting the appropriate risk factors from among the collected surveillance data and determining their optimal categorization. Standard bootstrapping techniques were applied to assess potential overfitting. The C-index was used to compare the predictive performances of the new statistical models with those of models based on conventional risk index variables.

RESULTS

The study sample comprised 349,987 cases from 428 participant hospitals throughout Japan, and the overall SSI incidence was 7.0%. The C-indices of the new statistical models were significantly higher than those of the conventional risk index models in 21 (67.7%) of the 31 procedure categories (P<.05). No significant overfitting was detected.

CONCLUSIONS

Japan-specific SSI prediction models were shown to generally have higher accuracy than conventional risk index models. These new models may have applications in assessing hospital performance and identifying high-risk patients in specific procedure categories.

Infect. Control Hosp. Epidemiol. 2016;37(3):260–271