Improving Lung Cancer Screening Selection: The HUNT Lung Cancer Risk Model for Ever-Smokers Versus the NELSON and 2021 United States Preventive Services Task Force Criteria in the Cohort of Norway: A Population-Based Prospective Study

Background

Improving the method for selecting participants for lung cancer (LC) screening is an urgent need. Here, we compared the performance of the Helseundersøkelsen i Nord-Trøndelag (HUNT) Lung Cancer Model (HUNT LCM) versus the Dutch-Belgian lung cancer screening trial (Nederlands-Leuvens Longkanker Screenings Onderzoek (NELSON)) and 2021 United States Preventive Services Task Force (USPSTF) criteria regarding LC risk prediction and efficiency.

Methods

We used linked data from 10 Norwegian prospective population-based cohorts, Cohort of Norway. The study included 44,831 ever-smokers, of which 686 (1.5%) patients developed LC; the median follow-up time was 11.6 years (0.01-20.8 years).

Results

Within 6 years, 222 (0.5%) individuals developed LC. The NELSON and 2021 USPSTF criteria predicted 37.4% and 59.5% of the LC cases, respectively. By considering the same number of individuals as the NELSON and 2021 USPSTF criteria selected, the HUNT LCM increased the LC prediction rate by 41.0% and 12.1%, respectively. The HUNT LCM significantly increased sensitivity (p < 0.001 and p = 0.028), and reduced the number needed to predict one LC case (29 versus 40, p < 0.001 and 36 versus 40, p = 0.02), respectively. Applying the HUNT LCM 6-year 0.98% risk score as a cutoff (14.0% of ever-smokers) predicted 70.7% of all LC, increasing LC prediction rate with 89.2% and 18.9% versus the NELSON and 2021 USPSTF, respectively (both p < 0.001).

Conclusions

The HUNT LCM was significantly more efficient than the NELSON and 2021 USPSTF criteria, improving the prediction of LC diagnosis, and may be used as a validated clinical tool for screening selection.

Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review

BACKGROUND

A comprehensive overview of artificial intelligence (AI) for cardiovascular disease (CVD) prediction and a screening tool of AI models (AI-Ms) for independent external validation are lacking. This systematic review aims to identify, describe, and appraise AI-Ms of CVD prediction in the general and special populations and develop a new independent validation score (IVS) for AI-Ms replicability evaluation.

METHODS

PubMed, Web of Science, Embase, and IEEE library were searched up to July 2021. Data extraction and analysis were performed for the populations, distribution, predictors, algorithms, etc. The risk of bias was evaluated with the prediction risk of bias assessment tool (PROBAST). Subsequently, we designed IVS for model replicability evaluation with five steps in five items, including transparency of algorithms, performance of models, feasibility of reproduction, risk of reproduction, and clinical implication, respectively. The review is registered in PROSPERO (No. CRD42021271789).

RESULTS

In 20,887 screened references, 79 articles (82.5% in 2017-2021) were included, which contained 114 datasets (67 in Europe and North America, but 0 in Africa). We identified 486 AI-Ms, of which the majority were in development (n = 380), but none of them had undergone independent external validation. A total of 66 idiographic algorithms were found; however, 36.4% were used only once and only 39.4% over three times. A large number of different predictors (range 5-52,000, median 21) and large-span sample size (range 80-3,660,000, median 4466) were observed. All models were at high risk of bias according to PROBAST, primarily due to the incorrect use of statistical methods. IVS analysis confirmed only 10 models as "recommended"; however, 281 and 187 were "not recommended" and "warning," respectively.

CONCLUSION

AI has led the digital revolution in the field of CVD prediction, but is still in the early stage of development as the defects of research design, report, and evaluation systems. The IVS we developed may contribute to independent external validation and the development of this field.

Development of a machine learning-based model for predicting risk of early postoperative recurrence of hepatocellular carcinoma

BACKGROUND

Surgical resection is the primary treatment for hepatocellular carcinoma (HCC). However, studies indicate that nearly 70% of patients experience HCC recurrence within five years following hepatectomy. The earlier the recurrence, the worse the prognosis. Current studies on postoperative recurrence primarily rely on postoperative pathology and patient clinical data, which are lagging. Hence, developing a new pre-operative prediction model for postoperative recurrence is crucial for guiding individualized treatment of HCC patients and enhancing their prognosis.

AIM

To identify key variables in pre-operative clinical and imaging data using machine learning algorithms to construct multiple risk prediction models for early postoperative recurrence of HCC.

METHODS

The demographic and clinical data of 371 HCC patients were collected for this retrospective study. These data were randomly divided into training and test sets at a ratio of 8:2. The training set was analyzed, and key feature variables with predictive value for early HCC recurrence were selected to construct six different machine learning prediction models. Each model was evaluated, and the best-performing model was selected for interpreting the importance of each variable. Finally, an online calculator based on the model was generated for daily clinical practice.

RESULTS

Following machine learning analysis, eight key feature variables (age, intratumoral arteries, alpha-fetoprotein, pre-operative blood glucose, number of tumors, glucose-to-lymphocyte ratio, liver cirrhosis, and pre-operative platelets) were selected to construct six different prediction models. The XGBoost model outperformed other models, with the area under the receiver operating characteristic curve in the training, validation, and test datasets being 0.993 (95% confidence interval: 0.982-1.000), 0.734 (0.601-0.867), and 0.706 (0.585-0.827), respectively. Calibration curve and decision curve analysis indicated that the XGBoost model also had good predictive performance and clinical application value.

CONCLUSION

The XGBoost model exhibits superior performance and is a reliable tool for predicting early postoperative HCC recurrence. This model may guide surgical strategies and postoperative individualized medicine.

Comparison of various prediction models in the effect of Roux-en-Y gastric bypass on type 2 diabetes in the Chinese population 5 years after surgery

BACKGROUND

Various prediction models of type 2 diabetes (T2D) remission have been externally verified internationally. However, long-term validated results after Roux-en-Y gastric bypass (RYGB) surgery are lacking. The best model for the Chinese population is also unknown.

OBJECTIVES

To evaluate the prediction effect of various prediction models on the long-term diabetes remission after RYGB in the Chinese population and to provide reference for clinical use.

SETTING

A retrospective clinical study at a university hospital.

METHODS

We retrospectively analyzed Chinese population data 5 years after RYGB and externally validated 11 predictive models to evaluate the predictive effect of each model on long-term T2D remission after RYGB.

RESULTS

We enrolled 84 patients. The mean body mass index was 41 kg/m2, and the percentage of excess weight loss (%EWL) was 72.3%. The mean glycated hemoglobin level was 8.4% preoperatively and decreased to 5.9% after 5 years. The 5-year postoperative complete and partial remission rates of T2D were 31% and 70.2%, respectively. The ABCD scoring model (sensitivity, 84%; specificity, 76%; area under the curve [AUC], .866) and the Panuzi et al. [34] study (sensitivity, 84%; specificity, 81%; AUC, .842) showed excellent results. In the Hosmer-Lemeshow goodness-of-fit test, calibration values for ABCD and Panuzi et al. [34] were .14 and .21, respectively. The predicted-to-observed ratios of ABCD and Panuzi et al. [34] were .83 and .88, respectively.

CONCLUSIONS

T2D was relieved to varying degrees 5 years after RYGB in patients with obesity. The prediction models in ABCD and the Panuzi et al. [34] studies showed the best prediction effects. ABCD was recommended for clinical use because of excellent predictive performance, good statistical test results, and simple and practical design features.

The performance of FIT-based and other risk prediction models for colorectal neoplasia in symptomatic patients: a systematic review

Background

Colorectal cancer (CRC) incidence and mortality are increasing internationally. Endoscopy services are under significant pressure with many overwhelmed. Faecal immunochemical testing (FIT) has been advocated to identify a high-risk population of symptomatic patients requiring definitive investigation by colonoscopy. Combining FIT with other factors in a risk prediction model could further improve performance in identifying those requiring investigation most urgently. We systematically reviewed performance of models predicting risk of CRC and/or advanced colorectal polyps (ACP) in symptomatic patients, with a particular focus on those models including FIT.

Methods

The review protocol was published on PROSPERO (CRD42022314710). Searches were conducted from database inception to April 2023 in MEDLINE, EMBASE, Cochrane libraries, SCOPUS and CINAHL. Risk of bias of each study was assessed using The Prediction study Risk Of Bias Assessment Tool. A narrative synthesis based on the guidelines for Synthesis Without Meta-Analysis was performed due to study heterogeneity.

Findings

We included 62 studies; 23 included FIT (n = 22) or guaiac Faecal Occult Blood Testing (n = 1) combined with one or more other variables. Twenty-one studies were conducted solely in primary care. Generally, prediction models including FIT consistently had good discriminatory ability for CRC/ACP (i.e. AUC >0.8) and performed better than models without FIT although some models without FIT also performed well. However, many studies did not present calibration and internal and external validation were limited. Two studies were rated as low risk of bias; neither model included FIT.

Interpretation

Risk prediction models, including and not including FIT, show promise for identifying those most at risk of colorectal neoplasia. Substantial limitations in evidence remain, including heterogeneity, high risk of bias, and lack of external validation. Further evaluation in studies adhering to gold standard methodology, in appropriate populations, is required before widespread adoption in clinical practice.

Funding

National Institute for Health and Care Research (NIHR) [Health Technology Assessment Programme (HTA) Programme (Project number 133852).

Risk prediction models for type 2 diabetes using either fasting plasma glucose or HbA1c in Chinese, Malay, and Indians: Results from three multi-ethnic Singapore cohorts

AIMS

To assess three well-established type 2 diabetes (T2D) risk prediction models based on fasting plasma glucose (FPG) in Chinese, Malays, and Indians, and to develop simplified risk models based on either FPG or HbA1c.

METHODS

We used a prospective multiethnic Singapore cohort to evaluate the established models and develop simplified models. 6,217 participants without T2D at baseline were included, with an average follow-up duration of 8.3 years. The simplified risk models were validated in two independent multiethnic Singapore cohorts (N = 12,720).

RESULTS

The established risk models had moderate-to-good discrimination (area under the receiver operating characteristic curves, AUCs 0.762 - 0.828) but a lack of fit (P-values < 0.05). Simplified risk models that included fewer predictors (age, BMI, systolic blood pressure, triglycerides, and HbA1c or FPG) showed good discrimination in all cohorts (AUCs ≥ 0.810), and sufficiently captured differences between the ethnic groups. While recalibration improved fit the simplified models in validation cohorts, there remained evidence of miscalibration in Chinese (p ≤ 0.012).

CONCLUSIONS

Simplified risk models including HbA1c or FPG had good discrimination in predicting incidence of T2D in three major Asian ethnic groups. Risk functions with HbA1c performed as well as those with FPG.

There is no such thing as a validated prediction model

BACKGROUND

Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context?

MAIN BODY

We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models.

CONCLUSION

Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.

Quantifying the utility of islet autoantibody levels in the prediction of type 1 diabetes in children

AIMS/HYPOTHESIS

The aim of this study was to explore the utility of islet autoantibody (IAb) levels for the prediction of type 1 diabetes in autoantibody-positive children.

METHODS

Prospective cohort studies in Finland, Germany, Sweden and the USA followed 24,662 children at increased genetic or familial risk of developing islet autoimmunity and diabetes. For the 1403 who developed IAbs (523 of whom developed diabetes), levels of autoantibodies against insulin (IAA), glutamic acid decarboxylase (GADA) and insulinoma-associated antigen-2 (IA-2A) were harmonised for analysis. Diabetes prediction models using multivariate logistic regression with inverse probability censored weighting (IPCW) were trained using 10-fold cross-validation. Discriminative power for disease was estimated using the IPCW concordance index (C index) with 95% CI estimated via bootstrap.

RESULTS

A baseline model with covariates for data source, sex, diabetes family history, HLA risk group and age at seroconversion with a 10-year follow-up period yielded a C index of 0.61 (95% CI 0.58, 0.63). The performance improved after adding the IAb positivity status for IAA, GADA and IA-2A at seroconversion: C index 0.72 (95% CI 0.71, 0.74). Using the IAb levels instead of positivity indicators resulted in even better performance: C index 0.76 (95% CI 0.74, 0.77). The predictive power was maintained when using the IAb levels alone: C index 0.76 (95% CI 0.75, 0.76). The prediction was better for shorter follow-up periods, with a C index of 0.82 (95% CI 0.81, 0.83) at 2 years, and remained reasonable for longer follow-up periods, with a C index of 0.76 (95% CI 0.75, 0.76) at 11 years. Inclusion of the results of a third IAb test added to the predictive power, and a suitable interval between seroconversion and the third test was approximately 1.5 years, with a C index of 0.78 (95% CI 0.77, 0.78) at 10 years follow-up.

CONCLUSIONS/INTERPRETATION

Consideration of quantitative patterns of IAb levels improved the predictive power for type 1 diabetes in IAb-positive children beyond qualitative IAb positivity status.

Application of machine learning in predicting the risk of postpartum depression: A systematic review

BACKGROUND

Postpartum depression (PPD) presents a serious health problem among women and their families. Machine learning (ML) is a rapidly advancing field with increasing utility in predicting PPD risk. We aimed to synthesize and evaluate the quality of studies on application of ML techniques in predicting PPD risk.

METHODS

We conducted a systematic search of eight databases, identifying English and Chinese studies on ML techniques for predicting PPD risk and ML techniques with performance metrics. Quality of the studies involved was evaluated using the Prediction Model Risk of Bias Assessment Tool.

RESULTS

Seventeen studies involving 62 prediction models were included. Supervised learning was the main ML technique employed and the common ML models were support vector machine, random forest and logistic regression. Five studies (30 %) reported both internal and external validation. Two studies involved model translation, but none were tested clinically. All studies showed a high risk of bias, and more than half showed high application risk.

LIMITATIONS

Including Chinese articles slightly reduced the reproducibility of the review. Model performance was not quantitatively analyzed owing to inconsistent metrics and the absence of methods for correlation meta-analysis.

CONCLUSIONS

Researchers have paid more attention to model development than to validation, and few have focused on improvement and innovation. Models for predicting PPD risk continue to emerge. However, few have achieved the acceptable quality standards. Therefore, ML techniques for successfully predicting PPD risk are yet to be deployed in clinical environments.

External validation of models for predicting risk of colorectal cancer using the China Kadoorie Biobank

BACKGROUND

In China, colorectal cancer (CRC) incidence and mortality have been steadily increasing over the last decades. Risk models to predict incident CRC have been developed in various populations, but they have not been systematically externally validated in a Chinese population.  This study aimed to assess the performance of risk scores in predicting CRC using the China Kadoorie Biobank (CKB), one of the largest and geographically diverse prospective cohort studies in China.

METHODS

Nine models were externally validated in 512,415 participants in CKB and included 2976 cases of CRC. Model discrimination was assessed, overall and by sex, age, site, and geographic location, using the area under the receiver operating characteristic curve (AUC). Model discrimination of these nine models was compared to a model using age alone. Calibration was assessed for five models, and they were re-calibrated in CKB.

RESULTS

The three models with the highest discrimination (Ma (Cox model) AUC 0.70 [95% CI 0.69-0.71]; Aleksandrova 0.70 [0.69-0.71]; Hong 0.69 [0.67-0.71]) included the variables age, smoking, and alcohol. These models performed significantly better than using a model based on age alone (AUC of 0.65 [95% CI 0.64-0.66]). Model discrimination was generally higher in younger participants, males, urban environments, and for colon cancer. The two models (Guo and Chen) developed in Chinese populations did not perform better than the others. Among the 10% of participants with the highest risk, the three best performing models identified 24-26% of participants that went on to develop CRC.

CONCLUSIONS

Several risk models based on easily obtainable demographic and modifiable lifestyle factor have good discrimination in a Chinese population. The three best performing models have a higher discrimination than using a model based on age alone.

Patient expectations of and experiences with a suicide risk identification algorithm in clinical practice

BACKGROUND

Suicide risk prediction models derived from electronic health records (EHR) and insurance claims are a novel innovation in suicide prevention but patient perspectives on their use have been understudied.

METHODS

In this qualitative study, between March and November 2020, 62 patients were interviewed from three health systems: one anticipating implementation of an EHR-derived suicide risk prediction model and two others piloting different implementation approaches. Site-tailored interview guides focused on patients' perceptions of this technology, concerns, and preferences for and experiences with suicide risk prediction model implementation in clinical practice. A constant comparative analytic approach was used to derive themes.

RESULTS

Interview participants were generally supportive of suicide risk prediction models derived from EHR data. Concerns included apprehension about inducing anxiety and suicidal thoughts, or triggering coercive treatment, particularly among those who reported prior negative experiences seeking mental health care. Participants who were engaged in mental health care or case management expected to be asked about their suicide risk and largely appreciated suicide risk conversations, particularly by clinicians comfortable discussing suicidality.

CONCLUSION

Most patients approved of suicide risk models that use EHR data to identify patients at-risk for suicide. As health systems proceed to implement such models, patient-centered care would involve dialogue initiated by clinicians experienced with assessing suicide risk during virtual or in person care encounters. Health systems should proactively monitor for negative consequences that result from risk model implementation to protect patient trust.

Predicting unplanned hospital visits in older home care recipients: a cross-country external validation study

Background

Accurate identification of older persons at risk of unplanned hospital visits can facilitate preventive interventions. Several risk scores have been developed to identify older adults at risk of unplanned hospital visits. It is unclear whether risk scores developed in one country, perform as well in another. This study validates seven risk scores to predict unplanned hospital admissions and emergency department (ED) visits in older home care recipients from six countries.

Methods

We used the IBenC sample (n = 2446), a cohort of older home care recipients from six countries (Belgium, Finland, Germany, Iceland, Italy and The Netherlands) to validate four specific risk scores (DIVERT, CARS, EARLI and previous acute admissions) and three frailty indicators (CHESS, Fried Frailty Criteria and Frailty Index). Outcome measures were unplanned hospital admissions, ED visits or any unplanned hospital visits after 6 months. Missing data were handled by multiple imputation. Performance was determined by assessing calibration and discrimination (area under receiver operating characteristic curve (AUC)).

Results

Risk score performance varied across countries. In Iceland, for any unplanned hospital visits DIVERT and CARS reached a fair predictive value (AUC 0.74 [0.68–0.80] and AUC 0.74 [0.67–0.80]), respectively). In Finland, DIVERT had fair performance predicting ED visits (AUC 0.72 [0.67–0.77]) and any unplanned hospital visits (AUC 0.73 [0.67–0.77]). In other countries, AUCs did not exceed 0.70.

Conclusions

Geographical validation of risk scores predicting unplanned hospital visits in home care recipients showed substantial variations of poor to fair performance across countries. Unplanned hospital visits seem considerably dependent on healthcare context. Therefore, risk scores should be validated regionally before applied to practice. Future studies should focus on identification of more discriminative predictors in order to develop more accurate risk scores.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12877-021-02521-2.

Modeling risks of cardiovascular and cancer mortality following a diagnosis of loco-regional breast cancer

Background

Many women with breast cancer also have a high likelihood of cardiovascular mortality, and while there are several cardiovascular risk prediction models, none have been validated in a cohort of breast cancer patients. We first compared the performance of commonly-used cardiovascular models, and then derived a new model where breast cancer and cardiovascular mortality were modeled simultaneously, to account for the competing risk endpoints and commonality of risk factors between the two events.

Methods

We included 20,462 women diagnosed with stage I–III breast cancer between 2000 and 2010 in Kaiser Permanente Northern California (KPNC) with follow-up through April 30, 2015, and examined the performance of the Framingham, CORE and SCOREOP cardiovascular risk models by area under the receiver operating characteristic curve (AUC), and observed-to -expected (O/E) ratio. We developed a multi-state model based on cause-specific hazards (CSH) to jointly model the causes of mortality.

Results

The extended models including breast cancer characteristics (grade, tumor size, nodal involvement) with CVD risk factors had better discrimination at 5-years with AUCs of 0.85 (95% CI 0.83, 0.86) for cardiovascular death and 0.80 (95% CI 0.78, 0.87) for breast cancer death compared with the existing cardiovascular models evaluated at 5 years AUCs ranging 0.71–0.78. Five-year calibration for breast and cardiovascular mortality from our multi-state model was also excellent (O/E = 1.01, 95% CI 0.91–1.11).

Conclusion

A model incorporating cardiovascular risk factors, breast cancer characteristics, and competing events, outperformed traditional models of cardiovascular disease by simultaneously estimating cancer and cardiovascular mortality risks.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13058-021-01469-w.

Clinical characteristics and risk factors of fatal patients with COVID-19: a retrospective cohort study in Wuhan, China

BACKGROUND

The coronavirus disease 2019 (COVID-19) has caused a global pandemic, resulting in considerable mortality. The risk factors, clinical treatments, especially comprehensive risk models for COVID-19 death are urgently warranted.

METHODS

In this retrospective study, 281 non-survivors and 712 survivors with propensity score matching by age, sex, and comorbidities were enrolled from January 13, 2020 to March 31, 2020.

RESULTS

Higher SOFA, qSOFA, APACHE II and SIRS scores, hypoxia, elevated inflammatory cytokines, multi-organ dysfunction, decreased immune cell subsets, and complications were significantly associated with the higher COVID-19 death risk. In addition to traditional predictors for death risk, including APACHE II (AUC = 0.83), SIRS (AUC = 0.75), SOFA (AUC = 0.70) and qSOFA scores (AUC = 0.61), another four prediction models that included immune cells subsets (AUC = 0.90), multiple organ damage biomarkers (AUC = 0.89), complications (AUC = 0.88) and inflammatory-related indexes (AUC = 0.75) were established. Additionally, the predictive accuracy of combining these risk factors (AUC = 0.950) was also significantly higher than that of each risk group alone, which was significant for early clinical management for COVID-19.

CONCLUSIONS

The potential risk factors could help to predict the clinical prognosis of COVID-19 patients at an early stage. The combined model might be more suitable for the death risk evaluation of COVID-19.

Management of adnexal mass: A comparison of five national guidelines

OBJECTIVES

General gynecologists are often the first to face a newly diagnosed adnexal mass. Bothering mass symptoms, fertility issues, and the effect of a possible surgical intervention on fertility in term of mechanical factor and ovarian follicular reserve are all considerations that should be accounted for. This study summarizes and compares five different adnexal mass management guidelines, enabling clinicians to peruse consensus and controversy issues, thus choosing the optimal management method.

DESIGN

We retrieved, reviewed and compared the most recent national guidelines of adnexal mass management from the national societies of the United States (American College of Obstetricians and Gynecologists), England (the Royal College of Obstetricians and Gynecologists), Canada (the Society of Obstetricians and Gynaecologists of Canada), Australia (the Royal Australian College of General Practitioners), and France (French College of Gynaecologists and Obstetricians).

RESULTS

There is a broad consensus regarding the role of transvaginal ultrasound as part of the initial evaluation of an adnexal mass and the radiological characteristics suggesting it being malignant. The role of transabdominal ultrasound or doppler mode is controversial. The use of MRI in cases of indeterminate adnexal masses is widely accepted. Ultrasound-guided aspiration is generally not recommended. There is a broad consensus that CA-125 should not be used as an ovarian cancer disease screening tool, though its role in the initial evaluation of adnexal masses is controversial. Risk prediction models are generally accepted, particularly the 'International Ovarian Tumor Analysis simple rules' and the 'Risk of Malignancy Index'.

CONCLUSION

Adnexal mass management national guidelines, though similar, had noticeable variations in the content, references cited, and recommendations made. While this variation might raise a concern as to the reproducibility of synthesizing literature, it can help practitioners present all spectra of recommendations and available data.

Validation and comparison of 28 risk prediction models for coronary artery disease

AIMS

Risk prediction models (RPMs) for coronary artery disease (CAD), using variables to calculate CAD risk, are potentially valuable tools in prevention strategies. However, their use in the clinical practice is limited by a lack of poor model description, external validation, and head-to-head comparisons.

METHODS AND RESULTS

CAD RPMs were identified through Tufts PACE CPM Registry and a systematic PubMed search. Every RPM was externally validated in the three cohorts (the UK Biobank, LifeLines, and PREVEND studies) for the primary endpoint myocardial infarction (MI) and secondary endpoint CAD, consisting of MI, percutaneous coronary intervention, and coronary artery bypass grafting. Model discrimination (C-index), calibration (intercept and regression slope), and accuracy (Brier score) were assessed and compared head-to-head between RPMs. Linear regression analysis was performed to evaluate predictive factors to estimate calibration ability of an RPM. Eleven articles containing 28 CAD RPMs were included. No single best-performing RPM could be identified across all cohorts and outcomes. Most RPMs yielded fair discrimination ability: mean C-index of RPMs was 0.706 ± 0.049, 0.778 ± 0.097, and 0.729 ± 0.074 (P < 0.01) for prediction of MI in UK Biobank, LifeLines, and PREVEND, respectively. Endpoint incidence in the original development cohorts was identified as a significant predictor for external validation performance.

CONCLUSION

Performance of CAD RPMs was comparable upon validation in three large cohorts, based on which no specific RPM can be recommended for predicting CAD risk.

Development and Validation of a Risk Prediction Tool for Second Primary Lung Cancer

Background

With advancing therapeutics, lung cancer (LC) survivors are rapidly increasing in number. Although mounting evidence suggests LC survivors have high risk of second primary lung cancer (SPLC), there is no validated prediction model available for clinical use to identify high-risk LC survivors for SPLC.

Methods

Using data from 6325 ever-smokers in the Multiethnic Cohort (MEC) study diagnosed with initial primary lung cancer (IPLC) in 1993-2017, we developed a prediction model for 10-year SPLC risk after IPLC diagnosis using cause-specific Cox regression. We evaluated the model’s clinical utility using decision curve analysis and externally validated it using 2 population-based data—Prostate, Lung, Colorectal, and Ovarian Cancer Screening Trial (PLCO) and National Lung Screening Trial (NLST)—that included 2963 and 2844 IPLC (101 and 93 SPLC cases), respectively.

Results

Over 14 063 person-years, 145 (2.3%) ever-smoking IPLC patients developed SPLC in MEC. Our prediction model demonstrated a high predictive accuracy (Brier score = 2.9, 95% confidence interval [CI] = 2.4 to 3.3) and discrimination (area under the receiver operating characteristics [AUC] = 81.9%, 95% CI = 78.2% to 85.5%) based on bootstrap validation in MEC. Stratification by the estimated risk quartiles showed that the observed SPLC incidence was statistically significantly higher in the 4th vs 1st quartile (9.5% vs 0.2%; P < .001). Decision curve analysis indicated that in a wide range of 10-year risk thresholds from 1% to 20%, the model yielded a larger net-benefit vs hypothetical all-screening or no-screening scenarios. External validation using PLCO and NLST showed an AUC of 78.8% (95% CI = 74.6% to 82.9%) and 72.7% (95% CI = 67.7% to 77.7%), respectively.

Conclusions

We developed and validated a SPLC prediction model based on large population-based cohorts. The proposed prediction model can help identify high-risk LC patients for SPLC and can be incorporated into clinical decision making for SPLC surveillance and screening.

Longitudinal Studies 5: Development of Risk Prediction Models for Patients with Chronic Disease

Chronic diseases are now the major cause of ill health in both developed and developing countries. Chronic diseases evolve, over decades, from an early reversible phase, to a late stage of irreversible organ damage. Importantly, the trajectory of individual patients with a chronic disease is highly variable. This uncertainty causes substantial stress and difficulty for patients, care providers, and health systems. Clinical risk prediction models address this uncertainty by incorporating multiple variables to more precisely estimate the risk of adverse events for an individual patient. In the current chapter, we describe the general approach to developing a risk prediction model. We then illustrate how these methods are applied in the development and validation of the kidney failure risk equation (KFRE), which accurately predicts the risk of kidney failure in patients with chronic kidney disease stages 3-5.

Nonclinical Features in Predictive Modeling of Cardiovascular Diseases: A Machine Learning Approach

BACKGROUND

In the broader healthcare domain, the prediction bears more value than an explanation considering the cost of delays in its services. There are various risk prediction models for cardiovascular diseases (CVDs) in the literature for early risk assessment. However, the substantial increase in CVDs-related mortality is challenging global health systems, especially in developing countries. This situation allows researchers to improve CVDs prediction models using new features and risk computing methods. This study aims to assess nonclinical features that can be easily available in any healthcare systems, in predicting CVDs using advanced and flexible machine learning (ML) algorithms.

METHODS

A gender-matched case-control study was conducted in the largest public sector cardiac hospital of Pakistan, and the data of 460 subjects were collected. The dataset comprised of eight nonclinical features. Four supervised ML algorithms were used to train and test the models to predict the CVDs status by considering traditional logistic regression (LR) as the baseline model. The models were validated through the train-test split (70:30) and tenfold cross-validation approaches.

RESULTS

Random forest (RF), a nonlinear ML algorithm, performed better than other ML algorithms and LR. The area under the curve (AUC) of RF was 0.851 and 0.853 in the train-test split and tenfold cross-validation approach, respectively. The nonclinical features yielded an admissible accuracy (minimum 71%) through the LR and ML models, exhibiting its predictive capability in risk estimation.

CONCLUSION

The satisfactory performance of nonclinical features reveals that these features and flexible computational methodologies can reinforce the existing risk prediction models for better healthcare services.

Utility of risk prediction models to detect atrial fibrillation in screened participants

AIMS

Atrial fibrillation (AF) is associated with higher risk of stroke. While the prevalence of AF is low in the general population, risk prediction models might identify individuals for selective screening of AF. We aimed to systematically identify and compare the utility of established models to predict prevalent AF.

METHODS AND RESULTS

Systematic search of PubMed and EMBASE for risk prediction models for AF. We adapted established risk prediction models and assessed their predictive performance using data from 2.5M individuals who attended vascular screening clinics in the USA and the UK and in the subset of 1.2M individuals with CHA2DS2-VASc ≥2. We assessed discrimination using area under the receiver operating characteristic (AUROC) curves and agreement between observed and predicted cases using calibration plots. After screening 6959 studies, 14 risk prediction models were identified. In our cohort, 10 464 (0.41%) participants had AF. For discrimination, six prediction model had AUROC curves of 0.70 or above in all individuals and those with CHA2DS2-VASc ≥2. In these models, calibration plots showed very good concordance between predicted and observed risks of AF. The two models with the highest observed prevalence in the highest decile of predicted risk, CHARGE-AF and MHS, showed an observed prevalence of AF of 1.6% with a number needed to screen of 63. Selective screening of the 10% highest risk identified 39% of cases with AF.

CONCLUSION

Prediction models can reliably identify individuals at high risk of AF. The best performing models showed an almost fourfold higher prevalence of AF by selective screening of individuals in the highest decile of risk compared with systematic screening of all cases.

REGISTRATION

This systematic review was registered (PROSPERO CRD42019123847).

Comparison of Four Risk Prediction Models for Diabetes Remission after Roux-en-Y Gastric Bypass Surgery in Obese Chinese Patients with Type 2 Diabetes Mellitus

BACKGROUND

Roux-en-Y gastric bypass (RYGB) is a major type of bariatric surgery. Various models have been established for facilitating clinical decision-making and predicting outcomes after RYGB; the ABCD, DiaRem, advanced-DiaRem (Ad-DiaRem), and DiaBetter scores are among the most commonly used risk prediction models. However, these models were primarily developed based on retrospective analyses of patients from Western countries at 1 year after RYGB. The present study was to assess the performance of these models and identify the optimal model, for predicting postoperative diabetes remission in diabetic Chinese patients.

METHODS

The present study included a total of 253 RYGB patients; 214 completed a 1-year follow-up and 131 completed a 3-year follow-up. The assessments and comparisons of the predictive performance of the four models were based on both discrimination and calibration measures. Discrimination was assessed according to the area under the receiver operating characteristic curve (AUC), and calibration was evaluated by Hosmer-Lemeshow goodness-of-fit tests and predicted-to-observed ratios.

RESULTS

One hundred thirteen (52.8%) in the 1-year follow-up group and 59 (45.0%) in the 3-year follow-up group achieved complete diabetes remission. Although all models showed similar discriminatory capacity and good calibration, the DiaBetter model exhibited the best predictive performance (1-year follow-up, AUC 0.760, 95% confidence interval [CI] 0.697-0.815, predicted-to-observed ratio 1.04; 3-year follow-up, AUC 0.804, 95% CI 0.726-0.868, predicted-to-observed ratio 0.95).

CONCLUSIONS

The present results indicated that the DiaBetter model is the optimal model for predicting postoperative diabetes remission in diabetic Chinese individuals, due to its excellent predictive accuracy and ready availability for use in clinical practice.

Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small

OBJECTIVES

When developing a clinical prediction model, penalization techniques are recommended to address overfitting, as they shrink predictor effect estimates toward the null and reduce mean-square prediction error in new individuals. However, shrinkage and penalty terms ('tuning parameters') are estimated with uncertainty from the development data set. We examined the magnitude of this uncertainty and the subsequent impact on prediction model performance.

STUDY DESIGN AND SETTING

This study comprises applied examples and a simulation study of the following methods: uniform shrinkage (estimated via a closed-form solution or bootstrapping), ridge regression, the lasso, and elastic net.

RESULTS

In a particular model development data set, penalization methods can be unreliable because tuning parameters are estimated with large uncertainty. This is of most concern when development data sets have a small effective sample size and the model's Cox-Snell R2 is low. The problem can lead to considerable miscalibration of model predictions in new individuals.

CONCLUSION

Penalization methods are not a 'carte blanche'; they do not guarantee a reliable prediction model is developed. They are more unreliable when needed most (i.e., when overfitting may be large). We recommend they are best applied with large effective sample sizes, as identified from recent sample size calculations that aim to minimize the potential for model overfitting and precisely estimate key parameters.

Assessing the risk of venous thromboembolism (VTE) in ambulatory patients with cancer: Rationale and implementation of a pharmacist-led VTE risk assessment program in an ambulatory cancer centre

Objectives

The objectives of this paper were to identify and compare clinical prediction models used to assess the risk of venous thromboembolism (VTE) in ambulatory patients with cancer, as well as review the rationale and implementation of a pharmacist-led VTE screening program using the Khorana Risk Score model in an ambulatory oncology centre in Sault Ste. Marie, Ontario, Canada.

Data Sources

PubMed was used to identify clinical practice guidelines and review articles discussing risk prediction models used to assess VTE risk in ambulatory patients with cancer.

Data Summary

Three commonly used VTE risk prediction models in ambulatory patients with cancer: the Khorana Risk Score, Vienna Cancer and Thrombosis Study (CATS) and Protecht Score, were identified via literature review. After considering guideline recommendations, site-specific factors (i.e. laboratory costs, time pharmacists spent calculating VTE risk) and evidence from the CASSINI and AVERT trials, a novel pharmacist-led VTE risk assessment program using the Khorana Risk Score was developed during a fourth-year PharmD clinical rotation at the Algoma District Cancer Program (ADCP) [ambulatory cancer care centre]. ADCP patients with a Khorana Risk Score of ≥2 were referred to the hematologist for a full VTE workup. Considering limitations, inclusion and exclusion criteria of the CASSINI and AVERT trials, the hematologist and pharmacy team decided on appropriate initiation of thromboprophylaxis with a direct oral anticoagulant (DOAC).

Conclusions

The Khorana Risk Score was the chosen model used for the pharmacist-led VTE risk assessment program due to its user-friendly scoring algorithm, evidence from validation studies and clinical trials, as well as ease of integration into pharmacy workflow. More research is needed to determine if pharmacist-led VTE risk assessment programs will impact patient outcomes, such as morbidity and mortality, secondary to cancer-associated thrombosis.

Is It Possible to Predict Weight Loss After Bariatric Surgery?-External Validation of Predictive Models

Background

Bariatric surgery is the most effective obesity treatment. Weight loss varies among patients, and not everyone achieves desired outcome. Identification of predictive factors for weight loss after bariatric surgery resulted in several prediction tools proposed. We aimed to validate the performance of available prediction models for weight reduction 1 year after surgical treatment.

Materials and Methods

The retrospective analysis included patients after Roux-en-Y gastric bypass (RYGB) or sleeve gastrectomy (SG) who completed 1-year follow-up. Postoperative body mass index (BMI) predicted by 12 models was calculated for each patient. The correlation between predicted and observed BMI was assessed using linear regression. Accuracy was evaluated by squared Pearson’s correlation coefficient (R²). Goodness-of-fit was assessed by standard error of estimate (SE) and paired sample t test between estimated and observed BMI.

Results

Out of 760 patients enrolled, 509 (67.00%) were women with median age 42 years. Of patients, 65.92% underwent SG and 34.08% had RYGB. Median BMI decreased from 45.19 to 32.53kg/m² after 1 year. EWL amounted to 62.97%. All models presented significant relationship between predicted and observed BMI in linear regression (correlation coefficient between 0.29 and 1.22). The best predictive model explained 24% variation of weight reduction (adjusted R²=0.24). Majority of models overestimated outcome with SE 5.03 to 5.13kg/m².

Conclusion

Although predicted BMI had reasonable correlation with observed values, none of evaluated models presented acceptable accuracy. All models tend to overestimate the outcome. Accurate tool for weight loss prediction should be developed to enhance patient’s assessment.

Graphical abstract

Supplementary Information

The online version contains supplementary material available at 10.1007/s11695-021-05341-w.

Cardiovascular diseases risk prediction in patients with diabetes: Posthoc analysis from a matched case-control study in Bangladesh

Purpose

This study aimed to investigate the estimated 10-year predicted risk of developing cardiovascular diseases (CVD) among participants with and without diabetes in Bangladesh.

Methods

We performed posthoc analysis from a matched case-control study conducted among 1262 participants. A total of 631 participants with diabetes (case) were recruited from a tertiary hospital, and 631 age, sex and residence matched participants (control) were recruited from the community in Dhaka, Bangladesh. Socioeconomic anthropometric, clinical and CVD risk factor data were collected from the participants. The 10-year estimated CVD risk was calculated using the Framingham Risk Score, which has reasonable validity in South Asians.

Results

The mean (SD) age of the participants were 51 (10) years. Total 52.3% of cases and 17.2% of controls were at high risk for CVD. The 10-year risk of CVD increased by age and was higher among males in both groups. Among the control group, high CVD risk was more prevalent among higher education and income groups. More than 85% of the tobacco smokers and 70% of chewing tobacco users in the case group were at high risk of CVD. Prevalence of high CVD risk among non-smokers cases was 8.6%. About 35% of hypertensive participants in the control group were at high risk of CVD.

Conclusion

Bangladeshi patients with diabetes showed a significant burden of CVD risk at a relatively younger age. Strategies for reducing tobacco use and improving BP control in people with diabetes is needed for lowering future CVD risks.

Comparison among different multiparametric scores for risk stratification in heart failure patients with reduced ejection fraction

Heart failure is a serious condition with high prevalence (about 2% in the adult population in developed countries, and more than 8% in patients older than 75 years). About 3–5% of hospital admissions are linked with heart failure incidents. The guidelines of the European Society of Cardiology for the diagnosis and treatment of acute and chronic heart failure have identified individual markers in patients with heart failure, including demographic data, aetiology, comorbidities, clinical, radiological, haemodynamic, echocardiographic and biochemical parameters. Several scoring systems have been proposed to identify adverse events, such as destabilizations, re-hospitalizations and mortality. This article reviews scoring systems for heart failure prognostication, with particular mention of those models with exercise tolerance objective definition. Although most of the models include readily available clinical information, quite a few of them comprise circulating levels of natriuretic peptides and a more objective evaluation of exercise tolerance. A literature review was also conducted to (a) identify heart failure risk-prediction models, (b) assess statistical approach, and (c) identify common variables.

ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models

OBJECTIVES

Receiver operating characteristic (ROC) curves show how well a risk prediction model discriminates between patients with and without a condition. We aim to investigate how ROC curves are presented in the literature and discuss and illustrate their potential limitations.

STUDY DESIGN AND SETTING

We conducted a pragmatic literature review of contemporary publications that externally validated clinical prediction models. We illustrated limitations of ROC curves using a testicular cancer case study and simulated data.

RESULTS

Of 86 identified prediction modeling studies, 52 (60%) presented ROC curves without thresholds and one (1%) presented an ROC curve with only a few thresholds. We illustrate that ROC curves in their standard form withhold threshold information have an unstable shape even for the same area under the curve (AUC) and are problematic for comparing model performance conditional on threshold. We compare ROC curves with classification plots, which show sensitivity and specificity conditional on risk thresholds.

CONCLUSION

ROC curves do not offer more information than the AUC to indicate discriminative ability. To assess the model's performance for decision-making, results should be provided conditional on risk thresholds. Therefore, if discriminatory ability must be visualized, classification plots are attractive.

The impact of frailty on admission to home care services and nursing homes: eight-year follow-up of a community-dwelling, older adult, Spanish cohort

Background

Frailty in older adults is a common multidimensional clinical entity, a state of vulnerability to stressors that increases the risk of adverse outcomes such as functional decline, institutionalization or death. The aim of this study is to identify the factors that anticipate the future inclusion of community-dwelling individuals aged ≥70 years in home care programmes (HC) and nursing homes (NH), and to develop the corresponding prediction models.

Methods

A prospective cohort study was conducted in 23 primary healthcare centers located in Catalonia, Spain, with an eight-year follow-up (2005–2013). The cohort was made up of 616 individuals. Data collection included a baseline multidimensional assessment carried out by primary health care professionals. Outcome variables were collected during follow-up by consulting electronic healthcare records, and the Central Registry of Catalonia for mortality. A prognostic index for a HC and NH at 8 years was estimated for each patient. Death prior to these events was considered a competing risk event, and Fine–Gray regression models were used.

Results

At baseline, mean age was 76.4 years and 55.5% were women. During follow-up, 19.2% entered a HC program, 8.2% a NH, and 15.4% died without presenting an event. Of those who entered a NH, 31.5% had previously been in a HC program.

Multivariate models for a HC and NH showed that the risk of a HC entry was associated with older age, dependence on the Instrumental Activities of Daily Living, and slow gait measured by Timed-up-and-go test. An increased risk of being admitted to a NH was associated with older age, dependence on the Instrumental Activities of Daily Living, number of prescriptions, and the presence of social risk.

Conclusions

Prognostic models based on comprehensive geriatric assessments can predict the need for the commencement of HC and NH admission in community-dwelling older adults. Our findings underline the necessity to measure functional capacity, mobility, number of prescriptions, and social aspects of older adults in primary healthcare centers. In such a setting they can be offered longitudinal holistic assessments so as to benefit from preventive actions in order to remain independent in the community for as long as possible.

Risk Stratification Strategies for Colorectal Cancer Screening: From Logistic Regression to Artificial Intelligence

Risk stratification is a system by which clinically meaningful separation of risk is achieved in a group of otherwise similar persons. Although parametric logistic regression dominates risk prediction, use of nonparametric and semiparametric methods, including artificial neural networks, is increasing. These statistical-learning and machine-learning methods, along with simple rules, are collectively referred to as "artificial intelligence" (AI). AI requires knowledge of study validity, understanding of model metrics, and determination of whether and to what extent the model can and should be applied to the patient or population under consideration. Further investigation is needed, especially in model validation and impact assessment.

Risk factors and risk prediction models for colorectal cancer metastasis and recurrence: an umbrella review of systematic reviews and meta-analyses of observational studies

BACKGROUND

There is a clear need for systematic appraisal of models/factors predicting colorectal cancer (CRC) metastasis and recurrence because clinical decisions about adjuvant treatment are taken on the basis of such variables.

METHODS

We conducted an umbrella review of all systematic reviews of observational studies (with/without meta-analysis) that evaluated risk factors of CRC metastasis and recurrence. We also generated an updated synthesis of risk prediction models for CRC metastasis and recurrence. We cross-assessed individual risk factors and risk prediction models.

RESULTS

Thirty-four risk factors for CRC metastasis and 17 for recurrence were investigated. Twelve of 34 and 4/17 risk factors with p < 0.05 were estimated to change the odds of the outcome at least 3-fold. Only one risk factor (vascular invasion for lymph node metastasis [LNM] in pT1 CRC) presented convincing evidence. We identified 24 CRC risk prediction models. Across 12 metastasis models, six out of 27 unique predictors were assessed in the umbrella review and four of them changed the odds of the outcome at least 3-fold. Across 12 recurrence models, five out of 25 unique predictors were assessed in the umbrella review and only one changed the odds of the outcome at least 3-fold.

CONCLUSIONS

This study provides an in-depth evaluation and cross-assessment of 51 risk factors and 24 prediction models. Our findings suggest that a minority of influential risk factors are employed in prediction models, which indicates the need for a more rigorous and systematic model construction process following evidence-based methods.

Evaluating Cardiovascular Disease (CVD) risk scores for participants with known CVD and non-CVD in a multiracial/ethnic Caribbean sample

Background

Cardiovascular Disease (CVD) risk prediction models have been useful in estimating if individuals are at low, intermediate, or high risk, of experiencing a CVD event within some established time frame, usually 10 years. Central to this is the concern in Trinidad and Tobago of using pre-existing CVD risk prediction methods, based on populations in the developed world (e.g. ASSIGN, Framingham and QRISK^®2), to establish risk for its multiracial/ethnic Caribbean population. The aim of this study was to determine which pre-existing CVD risk method is best suited for predicting CVD risk for individuals in this population.

Method

A survey was completed by 778 participants, 526 persons with no prior CVD, and 252 who previously reported a CVD event. Lifestyle and biometric data was collected from non-CVD participants, while for CVD participants, medical records were used to collect data at the first instance of CVD. The performances of three CVD risk prediction models (ASSIGN, Framingham and QRISK^®2) were evaluated using their calculated risk scores.

Results

All three models (ASSIGN, Framingham and QRISK^®2) identified less than 62% of cases (CVD participants) with a high proportion of false-positive predictions to true predictions as can be seen by positive predictabilities ranging from 78% (ASSIGN and Framingham) to 87% (QRISK^®2). Further, for all three models, individuals whose scores fell into the misclassification range were 2X more likely to be individuals who had experienced a prior CVD event as opposed to healthy individuals.

Conclusion

The ASSIGN, Framingham and QRISK^®2 models should be utilised with caution on a Trinidad and Tobago population of intermediate and high risk for CVD since these models were found to have underestimated the risk for individuals with CVD up to 2.5 times more often than they overestimated the risk for healthy persons.

Validating Risk Prediction Models of Diabetes Remission After Sleeve Gastrectomy

INTRODUCTION

Many risk prediction models of diabetes remission after bariatric and metabolic surgery have been proposed. Most models have been created using Roux-en-Y gastric bypass cohorts. However, validation of these models in sleeve gastrectomy (SG) is limited. The objective of our study is to validate the performance of risk prediction models of diabetes remission in obese patients with diabetes who underwent SG.

METHOD

This retrospective cohort study included 128 patients who underwent SG with at least 1 year follow-up from Dec 2011 to Sep 2016 as the validation cohort. A literature review revealed total 11 models with 2 categories (scoring system and logistic regression), which were validated by our study dataset. Discrimination was evaluated by area under the receiver operating characteristic (AUC) while calibration by Hosmer-Lemeshow test and predicted versus observed remission ratio.

RESULTS

At 1 year after surgery, 71.9% diabetes remission (HbA1c < 6.0 off medication) and 61.4% excess weight loss were observed. Individual metabolic surgery, ABCD, DiaRem, Advanced-DiaRem, DiaBetter, Ana et al., and Dixon et al. models showed excellent discrimination power (AUC > 0.8). In calibration, all models overestimated diabetes remission from 5 to 30% but did not lose their goodness of fit.

CONCLUSION

This is the first comprehensive external validation of current risk prediction models of diabetes remission at 1 year after SG. Seven models showed excellent predicting power, and scoring models were recommended more because of their easy utility.

Calibration: the Achilles heel of predictive analytics

BACKGROUND

The assessment of calibration performance of risk prediction models based on regression or more flexible machine learning algorithms receives little attention.

MAIN TEXT

Herein, we argue that this needs to change immediately because poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making. We summarize how to avoid poor calibration at algorithm development and how to assess calibration at algorithm validation, emphasizing balance between model complexity and the available sample size. At external validation, calibration curves require sufficiently large samples. Algorithm updating should be considered for appropriate support of clinical practice.

CONCLUSION

Efforts are required to avoid poor calibration when developing prediction models, to evaluate calibration when validating models, and to update models when indicated. The ultimate aim is to optimize the utility of predictive analytics for shared decision-making and patient counseling.

Patient-Reported Symptoms Improve Performance of Risk Prediction Models for Emergency Department Visits Among Patients With Cancer: A Population-Wide Study in Ontario Using Administrative Data

CONTEXT

Prior work shows measurements of symptom severity using the Edmonton Symptom Assessment System (ESAS) which are associated with emergency department (ED) visits in patients with cancer; however, it is not known if symptom severity improves the ability to predict ED visits.

OBJECTIVES

To determine whether information on symptom severity improves the ability to predict ED visits among patients with cancer.

METHODS

This was a population-based study of patients who were diagnosed with cancer and had at least one ESAS assessment completed between 2007 and 2015 in Ontario, Canada. After splitting the cohort into training and test sets, two ED visit risk prediction models using logistic regression were developed on the training cohort, one without ESAS and one with ESAS. The predictive performance of each risk model was assessed on the test cohort and compared with respect to area under the curve and calibration.

RESULTS

The full cohort consisted of 212,615 unique patients with a total of 1,267,294 ESAS assessments. The risk prediction model including ESAS was superior in sensitivity, specificity, accuracy, and discrimination. The area under the curve was 73.7% under the model with ESAS, whereas it was 70.1% under the model without ESAS. The model with ESAS was also better calibrated. This improvement in calibration was particularly noticeable among patients in the higher deciles of predicted risk.

CONCLUSION

This study demonstrates the importance of incorporating symptom measurements when developing an ED visit risk calculator for patients with cancer. Improved predictive models for ED visits using measurements of symptom severity may serve as an important clinical tool to prompt timely interventions by the cancer care team before an ED visit is necessary.

Using machine learning techniques to develop risk prediction models to predict graft failure following kidney transplantation: protocol for a retrospective cohort study

Background: A mechanism to predict graft failure before the actual kidney transplantation occurs is crucial to clinical management of chronic kidney disease patients.  Several kidney graft outcome prediction models, developed using machine learning methods, are available in the literature.  However, most of those models used small datasets and none of the machine learning-based prediction models available in the medical literature modelled time-to-event (survival) information, but instead used the binary outcome of failure or not. The objective of this study is to develop two separate machine learning-based predictive models to predict graft failure following live and deceased donor kidney transplant, using time-to-event data in a large national dataset from Australia.   Methods: The dataset provided by the Australia and New Zealand Dialysis and Transplant Registry will be used for the analysis. This retrospective dataset contains the cohort of patients who underwent a kidney transplant in Australia from January 1 st, 2007, to December 31 st, 2017.  This included 3,758 live donor transplants and 7,365 deceased donor transplants.  Three machine learning methods (survival tree, random survival forest and survival support vector machine) and one traditional regression method, Cox proportional regression, will be used to develop the two predictive models.  The best predictive model will be selected based on the model's performance. Discussion: This protocol describes the development of two separate machine learning-based predictive models to predict graft failure following live and deceased donor kidney transplant, using a large national dataset from Australia.   Furthermore, these two models will be the most comprehensive kidney graft failure predictive models that have used survival data to model using machine learning techniques.  Thus, these models are expected to provide valuable insight into the complex interactions between graft failure and donor and recipient characteristics.

Using machine learning techniques to develop risk prediction models to predict graft failure following kidney transplantation: protocol for a retrospective cohort study

Background: A mechanism to predict graft failure before the actual kidney transplantation occurs is crucial to clinical management of chronic kidney disease patients.  Several kidney graft outcome prediction models, developed using machine learning methods, are available in the literature.  However, most of those models used small datasets and none of the machine learning-based prediction models available in the medical literature modelled time-to-event (survival) information, but instead used the binary outcome of failure or not. The objective of this study is to develop two separate machine learning-based predictive models to predict graft failure following live and deceased donor kidney transplant, using time-to-event data in a large national dataset from Australia.   Methods: The dataset provided by the Australia and New Zealand Dialysis and Transplant Registry will be used for the analysis. This retrospective dataset contains the cohort of patients who underwent a kidney transplant in Australia from January 1 st, 2007, to December 31 st, 2017. This included 3,758 live donor transplants and 7,365 deceased donor transplants. Three machine learning methods (survival tree, random survival forest and survival support vector machine) and one traditional regression method, Cox proportional regression, will be used to develop the two predictive models (for live donor and deceased donor transplants). The best predictive model will be selected based on the model's performance. Discussion: This protocol describes the development of two separate machine learning-based predictive models to predict graft failure following live and deceased donor kidney transplant, using a large national dataset from Australia. Furthermore, these two models will be the most comprehensive kidney graft failure predictive models that have used survival data to model using machine learning techniques. Thus, these models are expected to provide valuable insight into the complex interactions between graft failure and donor and recipient characteristics.

Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators

CONTEXT

Urologists regularly develop clinical risk prediction models to support clinical decisions. In contrast to traditional performance measures, decision curve analysis (DCA) can assess the utility of models for decision making. DCA plots net benefit (NB) at a range of clinically reasonable risk thresholds.

OBJECTIVE

To provide recommendations on interpreting and reporting DCA when evaluating prediction models.

EVIDENCE ACQUISITION

We informally reviewed the urological literature to determine investigators' understanding of DCA. To illustrate, we use data from 3616 patients to develop risk models for high-grade prostate cancer (n=313, 9%) to decide who should undergo a biopsy. The baseline model includes prostate-specific antigen and digital rectal examination; the extended model adds two predictors based on transrectal ultrasound (TRUS).

EVIDENCE SYNTHESIS

We explain risk thresholds, NB, default strategies (treat all, treat no one), and test tradeoff. To use DCA, first determine whether a model is superior to all other strategies across the range of reasonable risk thresholds. If so, that model appears to improve decisions irrespective of threshold. Second, consider if there are important extra costs to using the model. If so, obtain the test tradeoff to check whether the increase in NB versus the best other strategy is worth the additional cost. In our case study, addition of TRUS improved NB by 0.0114, equivalent to 1.1 more detected high-grade prostate cancers per 100 patients. Hence, adding TRUS would be worthwhile if we accept subjecting 88 patients to TRUS to find one additional high-grade prostate cancer or, alternatively, subjecting 10 patients to TRUS to avoid one unnecessary biopsy.

CONCLUSIONS

The proposed guidelines can help researchers understand DCA and improve application and reporting.

PATIENT SUMMARY

Decision curve analysis can identify risk models that can help us make better clinical decisions. We illustrate appropriate reporting and interpretation of decision curve analysis.

The ANDROMEDA prospective cohort study: predictive value of combined criteria to tailor breast cancer screening and new opportunities from circulating markers: study protocol

Background

In recent years growing interest has been posed on alternative ways to screen women for breast cancer involving different imaging techniques or adjusting screening interval by breast cancer risk estimates. A new research area is studying circulating microRNAs as molecular biomarkers potentially useful for non invasive early detection together with the analysis of single-nucleotide polymorphisms (SNPs).

The Andromeda study is a prospective cohort study on women attending breast cancer screening in a northern Italian area. The aims of the study are: 1) to define appropriate women risk-based stratifications for personalized screening considering different factors (reproductive, family and biopsy history, breast density, lifestyle habits); 2) to evaluate the diagnostic accuracy of selected circulating microRNAs in a case-control study nested within the above mentioned cohort.

Methods

About 21,000 women aged 46–67 years compliant to screening mammography are expected to be enrolled. At enrolment, information on well-known breast cancer risk factors and life-styles habits are collected through self-admistered questionnaires. Information on breast density and anthropometric measurements (height, weight, body composition, and waist circumference) are recorded. In addition, women are requested to provide a blood sample for serum, plasma and buffy-coat storing for subsequent molecular analyses within the nested case-control study. This investigation will be performed on approximately 233 cases (screen-detected) and 699 matched controls to evaluate SNPs and circulating microRNAs. The whole study will last three years and the cohort will be followed up for ten years to observe the onset of new breast cancer cases.

Discussion

Nowadays women undergo the same screening protocol, independently of their breast density and their individual risk to develop breast cancer. New criteria to better stratify women in risk groups could enable the screening strategies to target high-risk women while reducing interventions in those at low-risk. In this frame the present study will contribute in identifying the feasibility and impact of implementing personalized breast cancer screening.

Trial registration

NCT02618538 (retrospectively registered on 27–11-2015.)

Electronic supplementary material

The online version of this article (10.1186/s12885-017-3784-5) contains supplementary material, which is available to authorized users.

Applying Risk Prediction Models to Optimize Lung Cancer Screening: Current Knowledge, Challenges, and Future Directions

Purpose of review

Risk prediction models may be useful for facilitating effective and high-quality decision-making at critical steps in the lung cancer screening process. This review provides a current overview of published lung cancer risk prediction models and their applications to lung cancer screening and highlights both challenges and strategies for improving their predictive performance and use in clinical practice.

Recent findings

Since the 2011 publication of the National Lung Screening Trial results, numerous prediction models have been proposed to estimate the probability of developing or dying from lung cancer or the probability that a pulmonary nodule is malignant. Respective models appear to exhibit high discriminatory accuracy in identifying individuals at highest risk of lung cancer or differentiating malignant from benign pulmonary nodules. However, validation and critical comparison of the performance of these models in independent populations are limited. Little is also known about the extent to which risk prediction models are being applied in clinical practice and influencing decision-making processes and outcomes related to lung cancer screening.

Summary

Current evidence is insufficient to determine which lung cancer risk prediction models are most clinically useful and how to best implement their use to optimize screening effectiveness and quality. To address these knowledge gaps, future research should be directed toward validating and enhancing existing risk prediction models for lung cancer and evaluating the application of model-based risk calculators and its corresponding impact on screening processes and outcomes.

Risk Prediction Models in CKD

Chronic kidney disease (CKD) currently affects 20 million Americans and is associated with increased morbidity and mortality. Resource-efficient and appropriate treatment of CKD benefits the patient and provides improved resource allocation for the health care system. Prediction models can be useful in efficiently allocating resources, and currently are being used at the bedside for several important clinical decisions. There is a paucity of prediction models in use in nephrology, but one such model, the Kidney Failure Risk Equation, uses routinely collected laboratory values and can inform clinical decisions related to the following: (1) triage of nephrology referrals, (2) evaluating the need for more intensive interdisciplinary clinic care, (3) determining the timing of modality education, and (4) dialysis access planning. The development of new models that predict survival and quality of life on dialysis, success on home modalities, failure of arteriovenous fistulas, and risk of cardiovascular disease in patients with CKD is needed.

Microarray-based SNP genotyping to identify genetic risk factors of triple-negative breast cancer (TNBC) in South Indian population

In the view of aggressive nature of Triple-Negative Breast cancer (TNBC) due to the lack of receptors (ER, PR, HER2) and high incidence of drug resistance associated with it, a case-control association study was conducted to identify the contributing genetic risk factors for Triple-negative breast cancer (TNBC). A total of 30 TNBC patients and 50 age and gender-matched controls of Indian origin were screened for 9,00,000 SNP markers using microarray-based SNP genotyping approach. The initial PLINK association analysis (p < 0.01, MAF 0.14-0.44, OR 10-24) identified 28 non-synonymous SNPs and one stop gain mutation in the exonic region as possible determinants of TNBC risk. All the 29 SNPs were annotated using ANNOVAR. The interactions between these markers were evaluated using Multifactor dimensionality reduction (MDR) analysis. The interactions were in the following order: exm408776 > exm1278309 > rs316389 > rs1651654 > rs635538 > exm1292477. Recursive partitioning analysis (RPA) was performed to construct decision tree useful in predicting TNBC risk. As shown in this analysis, rs1651654 and exm585172 SNPs are found to be determinants of TNBC risk. Artificial neural network model was used to generate the Receiver operating characteristic curves (ROC), which showed high sensitivity and specificity (AUC-0.94) of these markers. To conclude, among the 9,00,000 SNPs tested, CCDC42 exm1292477, ANXA3 exm408776, SASH1 exm585172 are found to be the most significant genetic predicting factors for TNBC. The interactions among exm408776, exm1278309, rs316389, rs1651654, rs635538, exm1292477 SNPs inflate the risk for TNBC further. Targeted analysis of these SNPs and genes alone also will have similar clinical utility in predicting TNBC.

Uncertainty quantification in breast cancer risk prediction models using self-reported family health history

Introduction. Family health history (FHx) is an important factor in breast and ovarian cancer risk assessment. As such, multiple risk prediction models rely strongly on FHx data when identifying a patient’s risk. These models were developed using verified information and when translated into a clinical setting assume that a patient’s FHx is accurate and complete. However, FHx information collected in a typical clinical setting is known to be imprecise and it is not well understood how this uncertainty may affect predictions in clinical settings. Methods. Using Monte Carlo simulations and existing measurements of uncertainty of self-reported FHx, we show how uncertainty in FHx information can alter risk classification when used in typical clinical settings. Results. We found that various models ranged from 52% to 64% for correct tier-level classification of pedigrees under a set of contrived uncertain conditions, but that significant misclassification are not negligible. Conclusions. Our work implies that (i) uncertainty quantification needs to be considered when transferring tools from a controlled research environment to a more uncertain environment (i.e, a health clinic) and (ii) better FHx collection methods are needed to reduce uncertainty in breast cancer risk prediction in clinical settings.

Derivation and Evaluation of a Risk-Scoring Tool to Predict Participant Attrition in a Lifestyle Intervention Project

Participant attrition in clinical trials and community-based interventions is a serious, common, and costly problem. In order to develop a simple predictive scoring system that can quantify the risk of participant attrition in a lifestyle intervention project, we analyzed data from the Special Diabetes Program for Indians Diabetes Prevention Program (SDPI-DP), an evidence-based lifestyle intervention to prevent diabetes in 36 American Indian and Alaska Native communities. SDPI-DP participants were randomly divided into a derivation cohort (n = 1600) and a validation cohort (n = 801). Logistic regressions were used to develop a scoring system from the derivation cohort. The discriminatory power and calibration properties of the system were assessed using the validation cohort. Seven independent factors predicted program attrition: gender, age, household income, comorbidity, chronic pain, site's user population size, and average age of site staff. Six factors predicted long-term attrition: gender, age, marital status, chronic pain, site's user population size, and average age of site staff. Each model exhibited moderate to fair discriminatory power (C statistic in the validation set: 0.70 for program attrition, and 0.66 for long-term attrition) and excellent calibration. The resulting scoring system offers a low-technology approach to identify participants at elevated risk for attrition in future similar behavioral modification intervention projects, which may inform appropriate allocation of retention resources. This approach also serves as a model for other efforts to prevent participant attrition.

Risk prediction models for hepatocellular carcinoma in different populations

Hepatocellular carcinoma (HCC) is a malignant disease with limited therapeutic options due to its aggressive progression. It places heavy burden on most low and middle income countries to treat HCC patients. Nowadays accurate HCC risk predictions can help making decisions on the need for HCC surveillance and antiviral therapy. HCC risk prediction models based on major risk factors of HCC are useful and helpful in providing adequate surveillance strategies to individuals who have different risk levels. Several risk prediction models among cohorts of different populations for estimating HCC incidence have been presented recently by using simple, efficient, and ready-to-use parameters. Moreover, using predictive scoring systems to assess HCC development can provide suggestions to improve clinical and public health approaches, making them more cost-effective and effort-effective, for inducing personalized surveillance programs according to risk stratification. In this review, the features of risk prediction models of HCC across different populations were summarized, and the perspectives of HCC risk prediction models were discussed as well.

A calibration hierarchy for risk models was defined: from utopia to empirical data

OBJECTIVE

Calibrated risk models are vital for valid decision support. We define four levels of calibration and describe implications for model development and external validation of predictions.

STUDY DESIGN AND SETTING

We present results based on simulated data sets.

RESULTS

A common definition of calibration is "having an event rate of R% among patients with a predicted risk of R%," which we refer to as "moderate calibration." Weaker forms of calibration only require the average predicted risk (mean calibration) or the average prediction effects (weak calibration) to be correct. "Strong calibration" requires that the event rate equals the predicted risk for every covariate pattern. This implies that the model is fully correct for the validation setting. We argue that this is unrealistic: the model type may be incorrect, the linear predictor is only asymptotically unbiased, and all nonlinear and interaction effects should be correctly modeled. In addition, we prove that moderate calibration guarantees nonharmful decision making. Finally, results indicate that a flexible assessment of calibration in small validation data sets is problematic.

CONCLUSION

Strong calibration is desirable for individualized decision support but unrealistic and counter productive by stimulating the development of overly complex models. Model development and external validation should focus on moderate calibration.

Perceptions of prognostic risks in chronic kidney disease: a national survey

Background

Predicting the clinical trajectories of chronic kidney disease (CKD) to discern personalized care remains a complex challenge in nephrology. Understanding the appropriate risk thresholds and time frame associated with predicting risks of key outcomes (kidney failure, cardiovascular (CV) events, and death) is critical in facilitating decision-making. As part of an exploratory research and practice support needs assessment, we aimed to determine the importance of the time frames for predicting key outcomes, and to assess the perceived demand for risk prediction tools among Canadian nephrologists.

Methods

A web-based survey was developed by a pan-Canadian expert panel of practitioners. Upon pre-test for clarity and ease of completion, the final survey was nationally deployed to Canadian nephrologists. Anonymous responses were gathered over a 4-month period. The results were analyzed using descriptive statistics.

Results

One hundred eleven nephrologists responded to our survey. The majority of the respondents described prediction of events over time frames of 1–5 years as being “extremely important” or “very important” to decision-making on a 5-point Likert scale. To plan for arteriovenous fistula referral, the respondents deemed thresholds which would predict probability of kidney failure between >30 and >50 % at 1 year, as useful, while many commented that the rate of progression should be included for decision-making. Over 80 % of the respondents were not satisfied with their current ability to predict the progression to kidney failure, CV events, and death. Most of them indicated that they would value and use validated risk scores for decision-making.

Conclusions

Our national survey of nephrologists shows that the risk prediction for major adverse clinical outcomes is valuable in CKD at multiple time frames and risk thresholds. Further research is required in developing relevant and meaningful risk prediction models for clinical decision-making in patient-centered CKD care.

Electronic supplementary material

The online version of this article (doi:10.1186/s40697-015-0088-z) contains supplementary material, which is available to authorized users.

Development of a shared decision-making tool to assist patients and clinicians with decisions on oral anticoagulant treatment for atrial fibrillation

OBJECTIVE

Decision aids (DAs) are increasingly used to operationalize shared decision-making (SDM) but their development is not often described. Decisions about oral anticoagulants (OACs) for atrial fibrillation (AF) involve a trade-off between lowering stroke risk and increasing OAC-associated bleeding risk, and consideration of how treatment affects lifestyle. The benefits and risks of OACs hinge upon a patient's risk factors for stroke and bleeding and how they value these outcomes. We present the development of a DA about AF that estimates patients' risks for stroke and bleeding and assesses their preferences for outcomes.

RESEARCH DESIGN AND METHODS

Based on a literature review and expert discussions, we identified stroke and major bleeding risk prediction models and embedded them into risk assessment modules. We identified the most important factors in choosing OAC treatment (warfarin used as the default reference OAC) through focus group discussions with AF patients who had used warfarin and clinician interviews. We then designed preference assessment and introductory modules accordingly. We integrated these modules into a prototype AF SDM tool and evaluated its usability through interviews.

RESULTS

Our tool included four modules: (1) introduction to AF and OAC treatment risks and benefits; (2) stroke risk assessment; (3) bleeding risk assessment; and (4) preference assessment. Interactive risk calculators estimated patient-specific stroke and bleeding risks; graphics were developed to communicate these risks. After cognitive interviews, the content was improved. The final AF tool calculates patient-specific risks and benefits of OAC treatment and couples these estimates with patient preferences to improve clinical decision-making.

CONCLUSIONS

The AF SDM tool may help patients choose whether OAC treatment is best for them and represents a patient-centered, integrative approach to educate patients on the benefits and risks of OAC treatment. Future research is needed to evaluate this tool in a real-world setting. The development process presented can be applied to similar SDM tools.

The Right Ventricular Function After Left Ventricular Assist Device (RVF-LVAD) study: rationale and preliminary results

AIMS

Despite improved outcomes and lower right ventricular failure (RVF) rates with continuous-flow left ventricular assist devices (LVADs), RVF still occurs in 20-40% of LVAD recipients and leads to worse clinical and patient-centred outcomes and higher utilization of healthcare resources. Preoperative quantification of RV function with echocardiography has only recently been considered for RVF prediction, and RV mechanics have not been prospectively evaluated.

METHODS AND RESULTS

In this single-centre prospective cohort study, we plan to enroll a total of 120 LVAD candidates to evaluate standard and mechanics-based echocardiographic measures of RV function, obtained within 7 days of planned LVAD surgery, for prediction of (i) RVF within 90 days; (ii) quality of life (QoL) at 90 days; and (iii) RV function recovery at 90 days post-LVAD. Our primary hypothesis is that an RV echocardiographic score will predict RVF with clinically relevant discrimination (C >0.85) and positive and negative predictive values (>80%). Our secondary hypothesis is that the RV score will predict QoL and RV recovery by 90 days. We expect that RV mechanics will provide incremental prognostic information for these outcomes. The preliminary results of an interim analysis are encouraging.

CONCLUSION

The results of this study may help improve LVAD outcomes and reduce resource utilization by facilitating shared decision-making and selection for LVAD implantation, provide insights into RV function recovery, and potentially inform reassessment of LVAD timing in patients at high risk for RVF.