Calibration: the Achilles heel of predictive analytics

Development, evaluation and validation of machine learning models to predict hospitalizations of patients with coronary artery disease within the next 12 months

BACKGROUND

Improved survival of patients after acute coronary syndromes, population growth, and overall life expectancy rise have led to a significant increase in the proportion of patients with stable coronary artery disease (CAD), creating a significant load on the entire healthcare system. The disease often progresses with the development of many complications while significantly increasing the likelihood of hospitalization. Developing and applying a machine learning model for predicting hospitalizations of patients with CAD to an inpatient medical facility will allow for close monitoring of high-risk patients, early preventive interventions, and optimized medical care.

AIMS

Development and external validation of personalized models for predicting the preventable hospitalizations of patients with stable CAD and its complications using ML algorithms and data of real-world clinical practice.

METHODS

135,873 depersonalized electronic health records of 49,103 patients with stable CAD were included in the study. Anthropometric measurements, physical examination results, laboratory, instrumental, anamnestic, and socio-demographic data, widely used in routine medical practice, were considered as potential predictors, a total of 73 features. Logistic regression, decision tree-based methods including gradient boosting (AdaBoost, LightGBM, XGBoost, CatBoost) and bagging (RandomForest and ExtraTrees), discriminant analysis (LinearDiscriminant, QuadraticDiscriminant), and naive Bayes classifier were compared. External validation was performed on the data of a separate region.

RESULTS

The best results and stability to external validation data were shown by the CatBoost model with an AUC of 0.875 (95% CI 0.865-0.885) for the internal testing and 0.872 (95% CI 0.856-0.886) for the external validation. The best model showed good performance evaluated through AUROC, Brier score and standardized net benefit (for the target NPV threshold) for the validation dataset that was only slightly similar to the train data.

CONCLUSION

The metrics of the best model were superior to previously published studies. The results of external validation demonstrated the relative stability of the model to new data from another region that confirms the possibility of the model's application in real clinical practice.

Understanding random resampling techniques for class imbalance correction and their consequences on calibration and discrimination of clinical risk prediction models

OBJECTIVE

Class imbalance is sometimes considered a problem when developing clinical prediction models and assessing their performance. To address it, correction strategies involving manipulations of the training dataset, such as random undersampling or oversampling, are frequently used. The aim of this article is to illustrate the consequences of these class imbalance correction strategies on clinical prediction models' internal validity in terms of calibration and discrimination performances.

METHODS

We used both heuristic intuition and formal mathematical reasoning to characterize the relations between conditional probabilities of interest and probabilities targeted when using random undersampling or oversampling. We propose a plug-in estimator that represents a natural correction for predictions obtained from models that have been trained on artificially balanced datasets ("naïve" models). We conducted a Monte Carlo simulation with two different data generation processes and present a real-world example using data from the International Stroke Trial database to empirically demonstrate the consequences of applying random resampling techniques for class imbalance correction on calibration and discrimination (in terms of Area Under the ROC, AUC) for logistic regression and tree-based prediction models.

RESULTS

Across our simulations and in the real-world example, calibration of the naïve models was very poor. The models using the plug-in estimator generally outperformed the models relying on class imbalance correction in terms of calibration while achieving the same discrimination performance.

CONCLUSION

Random resampling techniques for class imbalance correction do not generally improve discrimination performance (i.e., AUC), and their use is hard to justify when aiming at providing calibrated predictions. Improper use of such class imbalance correction techniques can lead to suboptimal data usage and less valid risk prediction models.

Converting IMPROVE bleeding and VTE risk assessment models into a fast-and-frugal decision tree for optimal hospital VTE prophylaxis

ABSTRACT

Current hospital venous thromboembolism (VTE) prophylaxis for medical patients is characterized by both underuse and overuse. The American Society of Hematology (ASH) has endorsed the use of risk assessment models (RAMs) as an approach to individualize VTE prophylaxis by balancing overuse (excessive risk of bleeding) and underuse (risk of avoidable VTE). ASH has endorsed IMPROVE (International Medical Prevention Registry on Venous Thromboembolism) risk assessment models, the only RAMs to assess short-term bleeding and VTE risk in acutely ill medical inpatients. ASH, however, notes that no RAMs have been thoroughly analyzed for their effect on patient outcomes. We aimed to validate the IMPROVE models and adapt them into a simple, fast-and-frugal (FFT) decision tree to evaluate the impact of VTE prevention on health outcomes and costs. We used 3 methods: the "best evidence" from ASH guidelines, a "learning health system paradigm" combining guideline and real-world data from the Medical University of South Carolina (MUSC), and a "real-world data" approach based solely on MUSC data retrospectively extracted from electronic records. We found that the most effective VTE prevention strategy used the FFT decision tree based on an IMPROVE VTE score of ≥2 or ≥4 and a bleeding score of <7. This method could prevent 45% of unnecessary treatments, saving ∼$5 million annually for patients such as the MUSC cohort. We recommend integrating IMPROVE models into hospital electronic medical records as a point-of-care tool, thereby enhancing VTE prevention in hospitalized medical patients.

Using Tumor Marker Gene Variants to Improve the Diagnostic Accuracy of DUPAN-2 and Carbohydrate Antigen 19-9 for Pancreatic Cancer

PURPOSE

Circulating carbohydrate antigen 19-9 (CA19-9) levels reflect FUT3 and FUT2 fucosyltransferase activity. Measuring the related glycan, DUPAN-2, can be useful in individuals unable to synthesize CA19-9. We hypothesized that similar to CA19-9, FUT functional groups determined by variants in FUT3 and FUT2 influence DUPAN-2 levels, and having tumor marker reference ranges for each functional group would improve diagnostic performance.

MATERIALS AND METHODS

Using a training/validation study design, FUT2/FUT3 genotypes were determined in 938 individuals from Johns Hopkins Hospital: 607 Cancer of the Pancreas Screening (CAPS) study subjects with unremarkable pancreata and 331 with pancreatic ductal adenocarcinoma (PDAC). Serum DUPAN-2 and CA19-9 levels were measured by immunoassay.

RESULTS

In controls, three functional FUT groups were identified with significant differences in DUPAN-2 levels: FUT3-intact, FUT3-null/FUT2-intact, and FUT3-null/FUT2-null. DUPAN-2 training set diagnostic cutoffs for each FUT group yielded higher diagnostic sensitivity in the validation set for patients with stage I/II PDAC than uniform cutoffs (60.4% [95% CI, 50.2 to 70.0] v 39.8% [30.0 to 49.8]), at approximately 99% (96.7 to 99.6) specificity. Combining FUT/CA19-9 and FUT/DUPAN-2 tests yielded 78.4% (72.3 to 83.7) sensitivity for stage I/II PDAC, at 97.7% (95.3 to 99.1) specificity in the combined sets, with higher AUC (stage I/II: 0.960 v 0.935 for CA19-9 + DUPAN-2 without the FUT test; P < .001); for stage I PDAC, sensitivity was 62.0% (49.1 to 73.2; AUC, 0.919 v 0.883; P = .03). CA19-9 levels in FUT3-null/FUT2-null PDAC subjects were higher than in FUT3-null/FUT2-intact subjects (median/IQR; 24.9/57.4 v <1/2.3 U/mL; P = .0044). In a simulated CAPS cohort, AUC precision recall (AUCPR) scores were 0.51 for CA19-9 alone, 0.64 for FUT/CA19-9, 0.73 for CA19-9/DUPAN-2, and 0.84 for FUT/CA19-9/DUPAN-2.

CONCLUSION

Using a tumor marker gene test to individualize CA19-9 and DUPAN-2 reference ranges achieves high diagnostic performance for stage I/II pancreatic cancer.

Prediction of sustained opioid use in children and adolescents using machine learning

BACKGROUND

Opioid misuse in the paediatric population is understudied. This study aimed to develop a machine learning classifier to differentiate between occasional and sustained opioid users among children and adolescents in outpatient settings.

METHODS

Data for 29,335 patients under 19 yr with recorded opioid purchases were collected from medical records. Machine learning methods were applied to predict sustained opioid use within 1, 2, or 3 yr after first opioid use, using sociodemographic information, medical history, and healthcare usage variables collected near the time of first prescription fulfilment. The models' performance was evaluated with classification and calibration metrics, and a decision curve analysis. An online tool was deployed for model self-exploration and visualisation.

RESULTS

The models demonstrated good performance, with a 1-yr follow-up model achieving a sensitivity of 0.772, a specificity of 0.703, and an ROC-AUC of 0.792 on an independent test set, with calibration intercept and slope of 0.00 and 1.02, respectively. Decision curve analysis revealed the clinical benefit of using the model relative to other strategies. SHAP analysis (SHapley Additive exPlanations) identified influential variables, including the number of diagnoses, medical images, laboratory tests, and type of opioid used.

CONCLUSIONS

Our model showed promising performance in predicting sustained opioid use among paediatric patients. The online risk prediction tool can facilitate compliance to such tools by clinicians. This study presents the potential of machine learning in identifying at-risk paediatric populations for sustained opioid use, potentially contributing to secondary prevention of opioid abuse.

Assessing calibration and bias of a deployed machine learning malnutrition prediction model within a large healthcare system

Malnutrition is a frequently underdiagnosed condition leading to increased morbidity, mortality, and healthcare costs. The Mount Sinai Health System (MSHS) deployed a machine learning model (MUST-Plus) to detect malnutrition upon hospital admission. However, in diverse patient groups, a poorly calibrated model may lead to misdiagnosis, exacerbating health care disparities. We explored the model's calibration across different variables and methods to improve calibration. Data from adult patients admitted to five MSHS hospitals from January 1, 2021 - December 31, 2022, were analyzed. We compared MUST-Plus prediction to the registered dietitian's formal assessment. Hierarchical calibration was assessed and compared between the recalibration sample (N = 49,562) of patients admitted between January 1, 2021 - December 31, 2022, and the hold-out sample (N = 17,278) of patients admitted between January 1, 2023 - September 30, 2023. Statistical differences in calibration metrics were tested using bootstrapping with replacement. Before recalibration, the overall model calibration intercept was -1.17 (95% CI: -1.20, -1.14), slope was 1.37 (95% CI: 1.34, 1.40), and Brier score was 0.26 (95% CI: 0.25, 0.26). Both weak and moderate measures of calibration were significantly different between White and Black patients and between male and female patients. Logistic recalibration significantly improved calibration of the model across race and gender in the hold-out sample. The original MUST-Plus model showed significant differences in calibration between White vs. Black patients. It also overestimated malnutrition in females compared to males. Logistic recalibration effectively reduced miscalibration across all patient subgroups. Continual monitoring and timely recalibration can improve model accuracy.

Competing risks of monomorphic vs. non-monomorphic ventricular arrhythmias in primary prevention implantable cardioverter-defibrillator recipients: Global Electrical Heterogeneity and Clinical Outcomes (GEHCO) study

AIMS

Ablation of monomorphic ventricular tachycardia (MMVT) has been shown to reduce shock frequency and improve survival. We aimed to compare cause-specific risk factors for MMVT and polymorphic ventricular tachycardia (PVT)/ventricular fibrillation (VF) and to develop predictive models.

METHODS AND RESULTS

The multicentre retrospective cohort study included 2668 patients (age 63.1 ± 13.0 years; 23% female; 78% white; 43% non-ischaemic cardiomyopathy; left ventricular ejection fraction 28.2 ± 11.1%). Cox models were adjusted for demographic characteristics, heart failure severity and treatment, device programming, and electrocardiogram metrics. Global electrical heterogeneity was measured by spatial QRS-T angle (QRSTa), spatial ventricular gradient elevation (SVGel), azimuth, magnitude (SVGmag), and sum absolute QRST integral (SAIQRST). We compared the out-of-sample performance of the lasso and elastic net for Cox proportional hazards and the Fine-Gray competing risk model. During a median follow-up of 4 years, 359 patients experienced their first sustained MMVT with appropriate implantable cardioverter-defibrillator (ICD) therapy, and 129 patients had their first PVT/VF with appropriate ICD shock. The risk of MMVT was associated with wider QRSTa [hazard ratio (HR) 1.16; 95% confidence interval (CI) 1.01-1.34], larger SVGel (HR 1.17; 95% CI 1.05-1.30), and smaller SVGmag (HR 0.74; 95% CI 0.63-0.86) and SAIQRST (HR 0.84; 95% CI 0.71-0.99). The best-performing 3-year competing risk Fine-Gray model for MMVT [time-dependent area under the receiver operating characteristic curve (ROC(t)AUC) 0.728; 95% CI 0.668-0.788] identified high-risk (> 50%) patients with 75% sensitivity and 65% specificity, and PVT/VF prediction model had ROC(t)AUC 0.915 (95% CI 0.868-0.962), both satisfactory calibration.

CONCLUSION

We developed and validated models to predict the competing risks of MMVT or PVT/VF that could inform procedural planning and future randomized controlled trials of prophylactic ventricular tachycardia ablation.

CLINICAL TRIAL REGISTRATION

URL:www.clinicaltrials.gov Unique identifier:NCT03210883.

Risk calculators for the detection of prostate cancer: a systematic review

BACKGROUND

Prostate cancer (PCa) (early) detection poses significant challenges, including unnecessary testing and the risk of potential overdiagnosis. The European Association of Urology therefore suggests an individual risk-adapted approach, incorporating risk calculators (RCs) into the PCa detection pathway. In the context of 'The PRostate Cancer Awareness and Initiative for Screening in the European Union' (PRAISE-U) project ( https://uroweb.org/praise-u ), we aim to provide an overview of the currently available clinical RCs applicable in an early PCa detection algorithm.

METHODS

We performed a systematic review to identify RCs predicting detection of clinically significant PCa at biopsy. A search was performed in the databases Medline ALL, Embase, Web of Science Core Collection, Cochrane Central Register of Controlled Trials and Google Scholar for publications between January 2010 and July 2023. We retrieved relevant literature by using the terms "prostate cancer", "screening/diagnosis" and "predictive model". Inclusion criteria included systematic reviews, meta-analyses, and clinical trials. Exclusion criteria applied to studies involving pre-targeted high-risk populations, diagnosed PCa patients, or a sample sizes under 50 men.

RESULTS

We identified 6474 articles, of which 140 were included after screening abstracts and full texts. In total, we identified 96 unique RCs. Among these, 45 underwent external validation, with 28 validated in multiple cohorts. Of the externally validated RCs, 17 are based on clinical factors, 19 incorporate clinical factors along with MRI details, 4 were based on blood biomarkers alone or in combination with clinical factors, and 5 included urinary biomarkers. The median AUC of externally validated RCs ranged from 0.63 to 0.93.

CONCLUSIONS

This systematic review offers an extensive analysis of currently available RCs, their variable utilization, and performance within validation cohorts. RCs have consistently demonstrated their capacity to mitigate the limitations associated with early detection and have been integrated into modern practice and screening trials. Nevertheless, the lack of external validation data raises concerns about numerous RCs, and it is crucial to factor in this omission when evaluating whether a specific RC is applicable to one's target population.

Risk factors and predictive model for prenatal depression: A large retrospective study in China

BACKGROUND

Prenatal depression, associated with adverse effects on mothers and fetuses, has received little attention. We conducted a large-sample study to investigate the risk factors of, and develop a predictive model for, prenatal depression in the Chinese population.

METHODS

This study enrolled 14,329 pregnant women who delivered at the West China Second University Hospital, Sichuan University from January 2017 to December 2020. Participants were divided into a training or validation cohort. Multiple variables were collected and selected using univariate logistic regression and least absolute shrinkage and selection operator penalty regression. After multivariate logistic analysis, a predictive model was developed and validated internally and externally.

RESULTS

Nine variables (employment, planned pregnancy, pregnancy number, conception methods, gestational diabetes mellitus, twin pregnancy, placenta previa, umbilical cord encirclement, and educational attainment) were identified as independent risk factors for prenatal depression. Receiver operating characteristic curves in both the training and validation cohorts showed excellent discrimination of the predictive model (the area under the curve: 0.746 and 0.732, respectively).

LIMITATIONS

The results of this retrospective study may be affected by confounding and information bias. Some important variables were excluded, such as family history of mental disorders. The study was conducted in China; its results may not be generalizable to other regions.

CONCLUSION

Our study identified nine significant risk factors for prenatal depression and constructed an accurate predictive model. This model could be applied as a clinical decision aid for individualized risk estimates and prevention of prenatal depression.

GRADE concept paper 8: judging the certainty of discrimination performance estimates of prognostic models in a body of validation studies

BACKGROUND

Prognostic models incorporate multiple prognostic factors to estimate the likelihood of future events for individual patients based on their prognostic factor values. Evaluating these models crucially involves conducting studies to assess their predictive performance, like discrimination. Systematic reviews and meta-analyses of these validation studies play an essential role in selecting models for clinical practice.

METHODS

In this paper, we outline 3 thresholds to determine the target for certainty rating in the discrimination of prognostic models, as observed across a body of validation studies.

RESULTS AND CONCLUSION

We propose 3 thresholds when rating the certainty of evidence about a prognostic model's discrimination. The first threshold amounts to rating certainty in the model's ability to classify better than random chance. The other 2 approaches involve setting thresholds informed by other mechanisms for classification: clinician intuition or an alternative prognostic model developed for the same disease area and outcome. The choice of threshold will vary based on the context. Instead of relying on arbitrary discrimination cut-offs, our approach positions the observed discrimination within an informed spectrum, potentially aiding decisions about a prognostic model's practical utility.

Development and Validation of the Scoring System of Appendicitis Severity 2.0

Importance

When considering nonoperative treatment in a patient with acute appendicitis, it is crucial to accurately rule out complicated appendicitis. The Atema score, also referred to as the Scoring System of Appendicitis Severity (SAS), has been designed to differentiate between uncomplicated and complicated appendicitis but has not been prospectively externally validated.

Objective

To externally validate the SAS and, in case of failure, to develop an improved SAS (2.0) for estimating the probability of complicated appendicitis.

Design, Setting, and Participants

This prospective study included adult patients who underwent operations for suspected acute appendicitis at 11 hospitals in the Netherlands between January 2020 and August 2021.

Main Outcomes and Measures

Appendicitis severity was predicted according to the SAS in 795 patients and its sensitivity and negative predictive value (NPV) for complicated appendicitis were calculated. Since the predefined targets of 95% for both were not met, the SAS 2.0 was developed using the same cohort. This clinical prediction model was developed with multivariable regression using clinical, biochemical, and imaging findings. The SAS 2.0 was externally validated in a temporal validation cohort consisting of 565 patients.

Results

In total, 1360 patients were included, 463 of whom (34.5%) had complicated appendicitis. Validation of the SAS resulted in a sensitivity of 83.6% (95% CI, 78.8-87.6) and an NPV of 85.0% (95% CI, 80.6-88.8), meaning that the predefined targets were not achieved. Therefore, the SAS 2.0 was developed, internally validated (C statistic, 0.87; 95% CI, 0.84-0.89), and subsequently externally validated (C statistic, 0.86; 95% CI, 0.82-0.89). The SAS 2.0 was designed to calculate a patient's individual probability of having complicated appendicitis along with a 95% CI.

Conclusions and Relevance

In this study, external validation of the SAS fell short in accurately distinguishing complicated from uncomplicated appendicitis. The newly developed and externally validated SAS 2.0 was able to assess an individual patient's probability of having complicated appendicitis with high accuracy in patients with acute appendicitis. Use of this patient-specific risk assessment tool can be helpful when considering and discussing nonoperative treatment of acute appendicitis with patients.

Systematic Review of Prognosis Models in Predicting Tooth Loss in Periodontitis

This study reviews and appraises the methodological and reporting quality of prediction models for tooth loss in periodontitis patients, including the use of regression and machine learning models. Studies involving prediction modeling for tooth loss in periodontitis patients were screened. A search was performed in MEDLINE via PubMed, Embase, and CENTRAL up to 12 February 2022, with citation chasing. Studies exploring model development or external validation studies for models assessing tooth loss in periodontitis patients for clinical use at any time point, with all prediction horizons in English, were considered. Studies were excluded if models were not developed for use in periodontitis patients, were not developed or validated on any data set, predicted outcomes other than tooth loss, or were prognostic factor studies. The CHARMS checklist was used for data extraction, TRIPOD to assess reporting quality, and PROBAST to assess the risk of bias. In total, 4,661 records were screened, and 45 studies were included. Only 26 studies reported any kind of performance measure. The median C-statistic reported was 0.671 (range, 0.57-0.97). All studies were at a high risk of bias due to inappropriate handling of missing data (96%), inappropriate evaluation of model performance (92%), and lack of accounting for model overfitting in evaluating model performance (68%). Many models predicting tooth loss in periodontitis are available, but studies evaluating these models are at a high risk of bias. Model performance measures are likely to be overly optimistic and might not be replicated in clinical use. While this review is unable to recommend any model for clinical practice, it has collated the existing models and their model performance at external validation and their associated sample sizes, which would be helpful to identify promising models for future external validation studies.

KLIC Score Does Not Predict Failure After Early Prosthetic Joint Infection: An External Validation With 153 Knees and 130 Hips

BACKGROUND

A preoperative risk score, the KLIC score (chronic renal failure [K], liver cirrhosis [L], indication of the index surgery [I], cemented prosthesis [C], and C-reactive protein >115 mg/L), has been developed to predict the risk of treatment failure after early prosthetic joint infection (PJI). This study aimed to validate the KLIC score for the debridement, antibiotics, and implant retention (DAIR) procedure and one-stage revisions in a Northern European cohort.

METHODS

Revisions due to early PJI of the hip or knee between January 1, 2008, and September 12, 2021, were identified retrospectively. The primary outcome was early failure, which was considered when the patient needed an unscheduled surgery, the patient died, or the patient was prescribed long-term suppressive antibiotics. To examine the association between KLIC score and failure risk, univariable logistic regression with area under the curve (AUC) was used. In addition, models were calibrated to assess prognostic ability and clinical utility was examined with decision-curve analyses.

RESULTS

An increase in KLIC score had a moderate predictive value for early failure after DAIR (odds ratio [OR] 1.45; confidence interval [CI] 1.13 to 1.90). For one-stage revision, it was only slightly predictive of failure (OR 1.20; CI 0.93 to 1.56). After 60 days, the AUC for DAIR was 0.63 (CI 0.55 to 0.72) and 0.56 (CI 0.46 to 0.66) for one-stage revisions, indicating poor discriminative ability. The decision-curve analyses revealed that the model did not offer a remarkable net benefit across a range of threshold probabilities.

CONCLUSIONS

We demonstrated that the KLIC score is not a reliable predictor of early failure after early PJI in a Northern European cohort. Using the model to guide treatment decisions does not provide any additional clinical utility beyond the baseline strategies.

External validation of the CholeS conversion from laparoscopic to open cholecystectomy (CLOC) risk score in Aotearoa New Zealand: a validation study

BACKGROUND

Conversion of laparoscopic cholecystectomy to open is uncommon, but is associated with longer hospital stay and recovery. Prognosticating conversion may aid service planning and provision. We therefore aimed to assess the external validity of the largest risk score for operative conversion.

METHODS

CHOLENZ was a multicentre, prospective, national cohort study of cholecystectomy for benign biliary disease conducted by STRATA, a trainee-led collaborative network. Data were collected from patients undergoing cholecystectomy in New Zealand hospitals between 1 August and 30 October 2021 with 30-day follow-up. The Conversion from Laparoscopic to Open Cholecystectomy (CLOC) score from the CholeS study was assessed for external validity by interrogating its accuracy and calibration in the CHOLENZ dataset.

RESULTS

Of 1162 cholecystectomies started laparoscopically, 20 (1.7%) were converted to open in the CHOLENZ dataset. The CLOC score predicted 2.9% (IQR 1.3%-8.1%) would be converted. Area under the curve was 0.65 (95% 0.51-0.79) and calibration was acceptable with a Hosmer-Lemeshow p value of 0.45; with evidence of tendency to overestimate with interrogation of calibration across a continuous risk profile (intercept 1.27, slope 0.4). Sensitivity analysis with imputed data improved accuracy. Recalibration with the addition of body mass index, and preoperative bilirubin also improved accuracy to 0.86 (95% CI 0.78-0.95).

CONCLUSIONS

The CLOC score in its original form is not generalisable to the Aotearoa New Zealand setting and is therefore not suitable for clinical use in our local setting.

Predicting peritoneal recurrence after radical gastrectomy for gastric cancer: Validation of a prediction model (PERI-Gastric 1 and PERI-Gastric 2) on a Korean database

BACKGROUND

Peritoneal recurrence is a significant cause of treatment failure after radical gastrectomy for gastric cancer. The prediction of metachronous peritoneal recurrence would have a significantly impact risk stratification and tailored treatment planning. This study aimed to externally validate the previously established PERI-Gastric 1 and 2 models to assess their generalizability in an independent population.

METHODS

Retrospective external validation was conducted on a cohort of 8564 patients who underwent elective gastrectomy for stage Ib-IIIc gastric cancer between 1998 and 2018 at the Yonsei Cancer Center. Discrimination was tested using the area under the receiver operating characteristic curves (AUROC). Accuracy was tested by plotting observations against the predicted risk of peritoneal recurrence and analyzing the resulting calibration plots. Clinical usefulness was tested with a decision curve analysis.

RESULTS

In the validation cohort, PERI-Gastric 1 and PERI-Gastric 2 exhibited an AUROC of 0.766 (95 % C.I. 0.752-0.778) and 0.767 (95 % C.I. 0.755-0.780), a calibration-in-the-large of 0.935 and 0.700, a calibration belt with a 95 % C.I. over the bisector in the risk range of 24%-33 % and 35%-47 %. The decision curve analysis revealed a positive net benefit in the risk range of 10%-42 % and 15%-45 %, respectively.

CONCLUSIONS

This study presents the external validation of the PERI-Gastric 1 and 2 scores in an Eastern population. The models demonstrated fair discrimination and satisfactory calibration for predicting the risk of peritoneal recurrence after radical gastrectomy, even in Eastern patients. PERI-Gastric 1 and 2 scores could also be applied to predict the risk of metachronous peritoneal recurrence in Eastern populations.

Multicentre validation of a machine learning model for predicting respiratory failure after noncardiac surgery

BACKGROUND

Postoperative respiratory failure is a serious complication that could benefit from early accurate identification of high-risk patients. We developed and validated a machine learning model to predict postoperative respiratory failure, defined as prolonged (>48 h) mechanical ventilation or reintubation after surgery.

METHODS

Easily extractable electronic health record (EHR) variables that do not require subjective assessment by clinicians were used. From EHR data of 307,333 noncardiac surgical cases, the model, trained with a gradient boosting algorithm, utilised a derivation cohort of 99,025 cases from Seoul National University Hospital (2013-9). External validation was performed using three separate cohorts A-C from different hospitals comprising 208,308 cases. Model performance was assessed by area under the receiver operating characteristic (AUROC) curve and area under the precision-recall curve (AUPRC), a measure of sensitivity and precision at different thresholds.

RESULTS

The model included eight variables: serum albumin, age, duration of anaesthesia, serum glucose, prothrombin time, serum creatinine, white blood cell count, and body mass index. Internally, the model achieved an AUROC of 0.912 (95% confidence interval [CI], 0.908-0.915) and AUPRC of 0.113. In external validation cohorts A, B, and C, the model achieved AUROCs of 0.879 (95% CI, 0.876-0.882), 0.872 (95% CI, 0.870-0.874), and 0.931 (95% CI, 0.925-0.936), and AUPRCs of 0.029, 0.083, and 0.124, respectively.

CONCLUSIONS

Utilising just eight easily extractable variables, this machine learning model demonstrated excellent discrimination in both internal and external validation for predicting postoperative respiratory failure. The model enables personalised risk stratification and facilitates data-driven clinical decision-making.

Multi-Ancestry Polygenic Risk Score for Coronary Heart Disease Based on an Ancestrally Diverse Genome-Wide Association Study and Population-Specific Optimization

BACKGROUND

Predictive performance of polygenic risk scores (PRS) varies across populations. To facilitate equitable clinical use, we developed PRS for coronary heart disease (CHD; PRSCHD) for 5 genetic ancestry groups.

METHODS

We derived ancestry-specific and multi-ancestry PRSCHD based on pruning and thresholding (PRSPT) and ancestry-based continuous shrinkage priors (PRSCSx) applied to summary statistics from the largest multi-ancestry genome-wide association study meta-analysis for CHD to date, including 1.1 million participants from 5 major genetic ancestry groups. Following training and optimization in the Million Veteran Program, we evaluated the best-performing PRSCHD in 176,988 individuals across 9 diverse cohorts.

RESULTS

Multi-ancestry PRSPT and PRSCSx outperformed ancestry-specific PRSPT and PRSCSx across a range of tuning values. Two best-performing multi-ancestry PRSCHD (ie, PRSPTmult and PRSCSxmult) and 1 ancestry-specific (PRSCSxEUR) were taken forward for validation. PRSPTmult demonstrated the strongest association with CHD in individuals of South Asian ancestry and European ancestry (odds ratio per 1 SD [95% CI, 2.75 [2.41-3.14], 1.65 [1.59-1.72]), followed by East Asian ancestry (1.56 [1.50-1.61]), Hispanic/Latino ancestry (1.38 [1.24-1.54]), and African ancestry (1.16 [1.11-1.21]). PRSCSxmult showed the strongest associations in South Asian ancestry (2.67 [2.38-3.00]) and European ancestry (1.65 [1.59-1.71]), lower in East Asian ancestry (1.59 [1.54-1.64]), Hispanic/Latino ancestry (1.51 [1.35-1.69]), and the lowest in African ancestry (1.20 [1.15-1.26]).

CONCLUSIONS

The use of summary statistics from a large multi-ancestry genome-wide meta-analysis improved the performance of PRSCHD in most ancestry groups compared with single-ancestry methods. Despite the use of one of the largest and most diverse sets of training and validation cohorts to date, improvement of predictive performance was limited in African ancestry. This highlights the need for larger genome-wide association study datasets of underrepresented populations to enhance the performance of PRSCHD.

Development and Internal Validation of a Multivariable Prediction Model for Mortality After Hip Fracture with Machine Learning Techniques

In order to estimate the likelihood of 1, 3, 6 and 12 month mortality in patients with hip fractures, we applied a variety of machine learning methods using readily available, preoperative data. We used prospectively collected data from a single university hospital in Copenhagen, Denmark for consecutive patients with hip fractures, aged 60 years and older, treated between September 2008 to September 2010 (n = 1186). Preoperative biochemical and anamnestic data were used as predictors and outcome was survival at 1, 3, 6 and 12 months after the fracture. After feature selection for each timepoint a stratified split was done (70/30) before training and validating Random Forest models, extreme gradient boosting (XGB) and Generalized Linear Models. We evaluated and compared each model using receiver operator characteristic (ROC), calibration slope and intercept, Spiegelhalter's z- test and Decision Curve Analysis. Using combinations of between 10 and 13 anamnestic and biochemical parameters we were able to successfully estimate the likelihood of mortality with an area under the curve on ROC curves of 0.79, 0.80, 0.79 and 0.81 for 1, 3, 6 and 12 month, respectively. The XGB was the overall best calibrated and most promising model. The XGB model most successfully estimated the likelihood of mortality postoperatively. An easy-to-use model could be helpful in perioperative decisions concerning level of care, focused research and information to patients. External validation is necessary before widespread use and is currently underway, an online tool has been developed for educational/experimental purposes ( https://hipfx.shinyapps.io/hipfx/ ).

Machine learning models for predicting blood pressure phenotypes by combining multiple polygenic risk scores

We construct non-linear machine learning (ML) prediction models for systolic and diastolic blood pressure (SBP, DBP) using demographic and clinical variables and polygenic risk scores (PRSs). We developed a two-model ensemble, consisting of a baseline model, where prediction is based on demographic and clinical variables only, and a genetic model, where we also include PRSs. We evaluate the use of a linear versus a non-linear model at both the baseline and the genetic model levels and assess the improvement in performance when incorporating multiple PRSs. We report the ensemble model's performance as percentage variance explained (PVE) on a held-out test dataset. A non-linear baseline model improved the PVEs from 28.1 to 30.1% (SBP) and 14.3% to 17.4% (DBP) compared with a linear baseline model. Including seven PRSs in the genetic model computed based on the largest available GWAS of SBP/DBP improved the genetic model PVE from 4.8 to 5.1% (SBP) and 4.7 to 5% (DBP) compared to using a single PRS. Adding additional 14 PRSs computed based on two independent GWASs further increased the genetic model PVE to 6.3% (SBP) and 5.7% (DBP). PVE differed across self-reported race/ethnicity groups, with primarily all non-White groups benefitting from the inclusion of additional PRSs. In summary, non-linear ML models improves BP prediction in models incorporating diverse populations.

EEG connectivity and network analyses predict outcome in patients with disorders of consciousness - A systematic review and meta-analysis

Outcome prediction in prolonged disorders of consciousness (DOC) remains challenging. This can result in either inappropriate withdrawal of treatment or unnecessary prolongation of treatment. Electroencephalography (EEG) is a cheap, portable, and non-invasive device with various opportunities for complex signal analysis. Computational EEG measures, such as EEG connectivity and network metrics, might be ideal candidates for the investigation of DOC, but their capacity in prognostication is still undisclosed. We conducted a meta-analysis aiming to compare the prognostic power of the widely used clinical scale, Coma Recovery Scale-Revised - CRS-R and EEG connectivity and network metrics. We found that the prognostic power of the CRS-R scale was moderate (AUC: 0.67 (0.60-0.75)), but EEG connectivity and network metrics predicted outcome with significantly (p = 0.0071) higher accuracy (AUC:0.78 (0.70-0.86)). We also estimated the prognostic capacity of EEG spectral power, which was not significantly (p = 0.3943) inferior to that of the EEG connectivity and graph-theory measures (AUC:0.75 (0.70-0.80)). Multivariate automated outcome prediction tools seemed to outperform clinical and EEG markers.

Machine Learning Algorithms Versus Classical Regression Models in Pre-Eclampsia Prediction: A Systematic Review

PURPOSE OF REVIEW

Machine learning (ML) approaches are an emerging alternative for healthcare risk prediction. We aimed to synthesise the literature on ML and classical regression studies exploring potential prognostic factors and to compare prediction performance for pre-eclampsia.

RECENT FINDINGS

From 9382 studies retrieved, 82 were included. Sixty-six publications exclusively reported eighty-four classical regression models to predict variable timing of onset of pre-eclampsia. Another six publications reported purely ML algorithms, whilst another 10 publications reported ML algorithms and classical regression models in the same sample with 8 of 10 findings that ML algorithms outperformed classical regression models. The most frequent prognostic factors were age, pre-pregnancy body mass index, chronic medical conditions, parity, prior history of pre-eclampsia, mean arterial pressure, uterine artery pulsatility index, placental growth factor, and pregnancy-associated plasma protein A. Top performing ML algorithms were random forest (area under the curve (AUC) = 0.94, 95% confidence interval (CI) 0.91-0.96) and extreme gradient boosting (AUC = 0.92, 95% CI 0.90-0.94). The competing risk model had similar performance (AUC = 0.92, 95% CI 0.91-0.92) compared with a neural network. Calibration performance was not reported in the majority of publications. ML algorithms had better performance compared to classical regression models in pre-eclampsia prediction. Random forest and boosting-type algorithms had the best prediction performance. Further research should focus on comparing ML algorithms to classical regression models using the same samples and evaluation metrics to gain insight into their performance. External validation of ML algorithms is warranted to gain insights into their generalisability.

Development and validation of a SCORing systEm for pre-thrombectomy diagnosis of IntraCranial Atherosclerotic Disease (Score-ICAD)

BACKGROUND

Early identification of intracranial atherosclerotic disease (ICAD) may impact the management of patients undergoing mechanical thrombectomy (MT). We sought to develop and validate a scoring system for pre-thrombectomy diagnosis of ICAD in anterior circulation large vessel/distal medium vessel occlusion strokes (LVOs/DMVOs).

METHODS

Retrospective analysis of two prospectively maintained comprehensive stroke center databases including patients with anterior circulation occlusions spanning 2010-22 (development cohort) and 2018-22 (validation cohort). ICAD cases were matched for age and sex (1:1) to non-ICAD controls.

RESULTS

Of 2870 MTs within the study period, 348 patients were included in the development cohort: 174 anterior circulation ICAD (6% of 2870 MTs) and 174 controls. Multivariable analysis β coefficients led to a 20 point scale: absence of atrial fibrillation (5); vascular risk factor burden (1) for each of hypertension, diabetes, smoking, and hyperlipidemia; multifocal single artery stenoses on CT angiography (3); absence of territorial cortical infarct (3); presence of borderzone infarct (3); or ipsilateral carotid siphon calcification (2). The validation cohort comprised 56 ICAD patients (4.1% of 1359 MTs): 56 controls. Area under the receiver operating characteristic curve was 0.88 (0.84-0.91) and 0.82 (0.73-0.89) in the development and validation cohorts, respectively. Calibration slope and intercept showed a good fit for the development cohort although with overestimated risk for the validation cohort. After intercept adjustment, the overestimation was corrected (intercept 0, 95% CI -0.5 to -0.5; slope 0.8, 95% CI 0.5 to 1.1). In the full cohort (n=414), ≥11 points showed the best performance for distinguishing ICAD from non-ICAD, with 0.71 (95% CI 0.65 to 0.78) sensitivity and 0.82 (95% CI 0.77 to 0.87) specificity, and 3.92 (95% CI 2.92 to 5.28) positive and 0.35 (95% CI 0.28 to 0.44) negative likelihood ratio. Scores ≥12 showed 90% specificity and 63% sensitivity.

CONCLUSION

The proposed scoring system for preprocedural diagnosis of ICAD LVOs and DMVOs presented satisfactory discrimination and calibration based on clinical and non-invasive radiological data.

Machine learning models on a web application to predict short-term postoperative outcomes following anterior cervical discectomy and fusion

BACKGROUND

The frequency of anterior cervical discectomy and fusion (ACDF) has increased up to 400% since 2011, underscoring the need to preoperatively anticipate adverse postoperative outcomes given the procedure's expanding use. Our study aims to accomplish two goals: firstly, to develop a suite of explainable machine learning (ML) models capable of predicting adverse postoperative outcomes following ACDF surgery, and secondly, to embed these models in a user-friendly web application, demonstrating their potential utility.

METHODS

We utilized data from the National Surgical Quality Improvement Program database to identify patients who underwent ACDF surgery. The outcomes of interest were four short-term postoperative adverse outcomes: prolonged length of stay (LOS), non-home discharges, 30-day readmissions, and major complications. We utilized five ML algorithms - TabPFN, TabNET, XGBoost, LightGBM, and Random Forest - coupled with the Optuna optimization library for hyperparameter tuning. To bolster the interpretability of our models, we employed SHapley Additive exPlanations (SHAP) for evaluating predictor variables' relative importance and used partial dependence plots to illustrate the impact of individual variables on the predictions generated by our top-performing models. We visualized model performance using receiver operating characteristic (ROC) curves and precision-recall curves (PRC). Quantitative metrics calculated were the area under the ROC curve (AUROC), balanced accuracy, weighted area under the PRC (AUPRC), weighted precision, and weighted recall. Models with the highest AUROC values were selected for inclusion in a web application.

RESULTS

The analysis included 57,760 patients for prolonged LOS [11.1% with prolonged LOS], 57,780 for non-home discharges [3.3% non-home discharges], 57,790 for 30-day readmissions [2.9% readmitted], and 57,800 for major complications [1.4% with major complications]. The top-performing models, which were the ones built with the Random Forest algorithm, yielded mean AUROCs of 0.776, 0.846, 0.775, and 0.747 for predicting prolonged LOS, non-home discharges, readmissions, and complications, respectively.

CONCLUSIONS

Our study employs advanced ML methodologies to enhance the prediction of adverse postoperative outcomes following ACDF. We designed an accessible web application to integrate these models into clinical practice. Our findings affirm that ML tools serve as vital supplements in risk stratification, facilitating the prediction of diverse outcomes and enhancing patient counseling for ACDF.

Generalizability of polygenic prediction models: how is the R2 defined on test data?

BACKGROUND

Polygenic risk scores (PRS) quantify an individual's genetic predisposition for different traits and are expected to play an increasingly important role in personalized medicine. A crucial challenge in clinical practice is the generalizability and transferability of PRS models to populations with different ancestries. When assessing the generalizability of PRS models for continuous traits, the R 2 is a commonly used measure to evaluate prediction accuracy. While the R 2 is a well-defined goodness-of-fit measure for statistical linear models, there exist different definitions for its application on test data, which complicates interpretation and comparison of results.

METHODS

Based on large-scale genotype data from the UK Biobank, we compare three definitions of the R 2 on test data for evaluating the generalizability of PRS models to different populations. Polygenic models for several phenotypes, including height, BMI and lipoprotein A, are derived based on training data with European ancestry using state-of-the-art regression methods and are evaluated on various test populations with different ancestries.

RESULTS

Our analysis shows that the choice of the R 2  definition can lead to considerably different results on test data, making the comparison of R 2  values from the literature problematic. While the definition as the squared correlation between predicted and observed phenotypes solely addresses the discriminative performance and always yields values between 0 and 1, definitions of the R 2 based on the mean squared prediction error (MSPE) with reference to intercept-only models assess both discrimination and calibration. These MSPE-based definitions can yield negative values indicating miscalibrated predictions for out-of-target populations. We argue that the choice of the most appropriate definition depends on the aim of PRS analysis - whether it primarily serves for risk stratification or also for individual phenotype prediction. Moreover, both correlation-based and MSPE-based definitions of R 2 can provide valuable complementary information.

CONCLUSIONS

Awareness of the different definitions of the R 2 on test data is necessary to facilitate the reporting and interpretation of results on PRS generalizability. It is recommended to explicitly state which definition was used when reporting R 2 values on test data. Further research is warranted to develop and evaluate well-calibrated polygenic models for diverse populations.

Externally validated treatment algorithm acceptably predicts nonoperative treatment success in patients with anterior cruciate ligament rupture

PURPOSE

This study aims to develop and externally validate a treatment algorithm to predict nonoperative treatment success or failure in patients with anterior cruciate ligament (ACL) rupture.

METHODS

Data were used from two completed studies of adult patients with ACL ruptures: the Conservative versus Operative Methods for Patients with ACL Rupture Evaluation study (development cohort) and the KNee osteoArthritis anterior cruciate Ligament Lesion study (validation cohort). The primary outcome variable is nonoperative treatment success or failure. Potential predictor variables were collected, entered into the univariable logistic regression model and then incorporated into the multivariable logistic regression model for constructing the treatment algorithm. Finally, predictive performance and goodness-of-fit were assessed and externally validated by discrimination and calibration measures.

RESULTS

In the univariable logistic regression model, a stable knee measured with the pivot shift test and a posttrauma International Knee Documentation Committee (IKDC) score <50 were predictive of needing an ACL reconstruction. Age >30 years and a body mass index > 30 kg/m2 were predictive for not needing an ACL reconstruction. Age, pretrauma Tegner score, the outcome of the pivot shift test and the posttrauma IKDC score are entered into the treatment algorithm. The predictability of needing an ACL reconstruction after nonoperative treatment (discrimination) is acceptable in both the development and the validation cohort: area under the curve = resp. 0.69 (95% confidence interval [CI]: 0.58-0.81) and 0.68 (95% CI: 0.58-0.78).

CONCLUSION

This study shows that the treatment algorithm can acceptably predict whether an ACL injury patient will have a(n) (un)successful nonoperative treatment (discrimination). Calibration of the treatment algorithm suggests a systematical underestimation of the need for ACL reconstruction. Given the limitations regarding the sample size of this study, larger data sets must be constructed to improve the treatment algorithm further.

LEVEL OF EVIDENCE

Level II.

Machine Learning to Allocate Palliative Care Consultations During Cancer Treatment

PURPOSE

For patients with advanced cancer, early consultations with palliative care (PC) specialists reduce costs, improve quality of life, and prolong survival. However, capacity limitations prevent all patients from receiving PC shortly after diagnosis. We evaluated whether a prognostic machine learning system could promote early PC, given existing capacity.

METHODS

Using population-level administrative data in Ontario, Canada, we assembled a cohort of patients with incurable cancer who received palliative-intent systemic therapy between July 1, 2014, and December 30, 2019. We developed a machine learning system that predicted death within 1 year of each treatment using demographics, cancer characteristics, treatments, symptoms, laboratory values, and history of acute care admissions. We trained the system in patients who started treatment before July 1, 2017, and evaluated the potential impact of the system on PC in subsequent patients.

RESULTS

Among 560,210 treatments received by 54,628 patients, death occurred within 1 year of 45.2% of treatments. The machine learning system recommended the same number of PC consultations observed with usual care at the 60.0% 1-year risk of death, with a first-alarm positive predictive value of 69.7% and an outcome-level sensitivity of 74.9%. Compared with usual care, system-guided care could increase early PC by 8.5% overall (95% CI, 7.5 to 9.5; P < .001) and by 15.3% (95% CI, 13.9 to 16.6; P < .001) among patients who live 6 months beyond their first treatment, without requiring more PC consultations in total or substantially increasing PC among patients with a prognosis exceeding 2 years.

CONCLUSION

Prognostic machine learning systems could increase early PC despite existing resource constraints. These results demonstrate an urgent need to deploy and evaluate prognostic systems in real-time clinical practice to increase access to early PC.

Validation of the European Society of Cardiology pretest probability models for obstructive coronary artery disease in high-risk population

BACKGROUND

The pre-test probability (PTP) model for obstructive coronary artery disease (CAD) was updated in 2019 by the European Society of Cardiology (ESC). To our knowledge, this model was never externally validated in population with high incidence of CAD. The aim of this study is to validate the new PTP ESC model in our population which has a high CAD incidence and to compare it with previous PTP ESC model from 2013.

METHODS

We retrospectively analysed 1294 symptomatic patients with suspected CAD referred to our centre between 2015 and 2019. In all patients, the PTP score was calculated based on age, gender and symptoms according to the ESC model from 2013 (2013-ESC-PTP) and 2019 (2019-ESC-PTP). All patients underwent invasive coronary angiography (ICA).

RESULTS

Of the 1294 patients, obstructive CAD was diagnosed in 533 patients (41.2%). The 2019-ESC-PTP model categorised significantly more patients into the low probability group (PTP < 15%) than the 2013-ESC-PTP model (39.8% vs. 5.6%, P< 0.001). Obstructive CAD prevalence was underestimated using 2019-ESC-PTP at all PTP levels (calibration intercept 1.15, calibration slope 0.96). The 2013-ESC-PTP overestimated obstructive CAD prevalence (calibration intercept -0.24, calibration slope 0.73). The discrimination measured with an area under the curve was similar for both models, indicating moderate accuracy of the models.

CONCLUSIONS

In high-risk Serbian population, both the 2013 and 2019 ESC-PTP models had moderate accuracy in diagnosing CAD, with the 2019-ESC-PTP underestimating the prevalence of CAD, while the 2013-ESC-PTP overestimating it. Further studies are warranted to establish PTP models for high-risk countries.

External Validation of Updated Prediction Models for Neurological Outcomes at 90 Days in Patients With Out-of-Hospital Cardiac Arrest

BACKGROUND

Few prediction models for individuals with early-stage out-of-hospital cardiac arrest (OHCA) have undergone external validation. This study aimed to externally validate updated prediction models for OHCA outcomes using a large nationwide dataset.

METHODS AND RESULTS

We performed a secondary analysis of the JAAM-OHCA (Comprehensive Registry of In-Hospital Intensive Care for Out-of-Hospital Cardiac Arrest Survival and the Japanese Association for Acute Medicine Out-of-Hospital Cardiac Arrest) registry. Previously developed prediction models for patients with cardiac arrest who achieved the return of spontaneous circulation were updated. External validation was conducted using data from 56 institutions from the JAAM-OHCA registry. The primary outcome was a dichotomized 90-day cerebral performance category score. Two models were updated using the derivation set (n=3337). Model 1 included patient demographics, prehospital information, and the initial rhythm upon hospital admission; Model 2 included information obtained in the hospital immediately after the return of spontaneous circulation. In the validation set (n=4250), Models 1 and 2 exhibited a C-statistic of 0.945 (95% CI, 0.935-0.955) and 0.958 (95% CI, 0.951-0.960), respectively. Both models were well-calibrated to the observed outcomes. The decision curve analysis showed that Model 2 demonstrated higher net benefits at all risk thresholds than Model 1. A web-based calculator was developed to estimate the probability of poor outcomes (https://pcas-prediction.shinyapps.io/90d_lasso/).

CONCLUSIONS

The updated models offer valuable information to medical professionals in the prediction of long-term neurological outcomes for patients with OHCA, potentially playing a vital role in clinical decision-making processes.

Developing and validating prediction models for severe exacerbations and readmissions in patients hospitalised for COPD exacerbation (SERCO) in China: a prospective observational study

BACKGROUND

There is a lack of individualised prediction models for patients hospitalised with chronic obstructive pulmonary disease (COPD) for clinical practice. We developed and validated prediction models of severe exacerbations and readmissions in patients hospitalised for COPD exacerbation (SERCO).

METHODS

Data were obtained from the Acute Exacerbations of Chronic Obstructive Pulmonary Disease Inpatient Registry study (NCT02657525) in China. Cause-specific hazard models were used to estimate coefficients. C-statistic was used to evaluate the discrimination. Slope and intercept were used to evaluate the calibration and used for model adjustment. Models were validated internally by 10-fold cross-validation and externally using data from different regions. Risk-stratified scoring scales and nomograms were provided. The discrimination ability of the SERCO model was compared with the exacerbation history in the previous year.

RESULTS

Two sets with 2196 and 1869 patients from different geographical regions were used for model development and external validation. The 12-month severe exacerbations cumulative incidence rates were 11.55% (95% CI 10.06% to 13.16%) in development cohorts and 12.30% (95% CI 10.67% to 14.05%) in validation cohorts. The COPD-specific readmission incidence rates were 11.31% (95% CI 9.83% to 12.91%) and 12.26% (95% CI 10.63% to 14.02%), respectively. Demographic characteristics, medical history, comorbidities, drug usage, Global Initiative for Chronic Obstructive Lung Disease stage and interactions were included as predictors. C-indexes for severe exacerbations were 77.3 (95% CI 70.7 to 83.9), 76.5 (95% CI 72.6 to 80.4) and 74.7 (95% CI 71.2 to 78.2) at 1, 6 and 12 months. The corresponding values for readmissions were 77.1 (95% CI 70.1 to 84.0), 76.3 (95% CI 72.3 to 80.4) and 74.5 (95% CI 71.0 to 78.0). The SERCO model was consistently discriminative and accurate with C-indexes in the derivation and internal validation groups. In external validation, the C-indexes were relatively lower at 60-70 levels. The SERCO model discriminated outcomes better than prior severe exacerbation history. The slope and intercept after adjustment showed close agreement between predicted and observed risks. However, in external validation, the models may overestimate the risk in higher-risk groups. The model-driven risk groups showed significant disparities in prognosis.

CONCLUSION

The SERCO model provides individual predictions for severe exacerbation and COPD-specific readmission risk, which enables identifying high-risk patients and implementing personalised preventive intervention for patients with COPD.

Biomarkers to improve functional outcome prediction after ischemic stroke: Results from the SICFAIL, STRAWINSKI, and PREDICT studies

BACKGROUND AND AIMS

Acute ischemic stroke (AIS) outcome prognostication remains challenging despite available prognostic models. We investigated whether a biomarker panel improves the predictive performance of established prognostic scores.

METHODS

We investigated the improvement in discrimination, calibration, and overall performance by adding five biomarkers (procalcitonin, copeptin, cortisol, mid-regional pro-atrial natriuretic peptide (MR-proANP), and N-terminal pro-B-type natriuretic peptide (NT-proBNP)) to the Acute Stroke Registry and Analysis of Lausanne (ASTRAL) and age/NIHSS scores using data from two prospective cohort studies (SICFAIL, PREDICT) and one clinical trial (STRAWINSKI). Poor outcome was defined as mRS > 2 at 12 (SICFAIL, derivation dataset) or 3 months (PREDICT/STRAWINSKI, pooled external validation dataset).

RESULTS

Among 412 SICFAIL participants (median age 70 years, quartiles 59-78; 63% male; median NIHSS score 3, quartiles 1-5), 29% had a poor outcome. Area under the curve of the ASTRAL and age/NIHSS were 0.76 (95% CI 0.71-0.81) and 0.77 (95% CI 0.73-0.82), respectively. Copeptin (0.79, 95% CI 0.74-0.84), NT-proBNP (0.80, 95% CI 0.76-0.84), and MR-proANP (0.79, 95% CI 0.75-0.84) significantly improved ASTRAL score's discrimination, calibration, and overall performance. Copeptin improved age/NIHSS model's discrimination, copeptin, MR-proANP, and NT-proBNP improved its calibration and overall performance. In the validation dataset (450 patients, median age 73 years, quartiles 66-81; 54% men; median NIHSS score 8, quartiles 3-14), copeptin was independently associated with various definitions of poor outcome and also mortality. Copeptin did not increase model's discrimination but it did improve calibration and overall model performance.

DISCUSSION

Copeptin, NT-proBNP, and MR-proANP improved modest but consistently the predictive performance of established prognostic scores in patients with mild AIS. Copeptin was most consistently associated with poor outcome in patients with moderate to severe AIS, although its added prognostic value was less obvious.

Harnessing multi-source data for individualized care in Hodgkin Lymphoma

Hodgkin lymphoma is a rare, but highly curative form of cancer, primarily afflicting adolescents and young adults. Despite multiple seminal trials over the past twenty years, there is no single consensus-based treatment approach beyond use of multi-agency chemotherapy with curative intent. The use of radiation continues to be debated in early-stage disease, as part of combined modality treatment, as well as in salvage, as an important form of consolidation. While short-term disease outcomes have varied little across these different approaches across both early and advanced stage disease, the potential risk of severe, longer-term risk has varied considerably. Over the past decade novel therapeutics have been employed in the retrieval setting in preparation to and as consolidation after autologous stem cell transplant. More recently, these novel therapeutics have moved to the frontline setting, initially compared to standard-of-care treatment and later in a direct head-to-head comparison combined with multi-agent chemotherapy. In 2018, we established the HoLISTIC Consortium, bringing together disease and methods experts to develop clinical decision models based on individual patient data to guide providers, patients, and caregivers in decision-making. In this review, we detail the steps we followed to create the master database of individual patient data from patients treated over the past 20 years, using principles of data science. We then describe different methodological approaches we are taking to clinical decision making, beginning with clinical prediction tools at the time of diagnosis, to multi-state models, incorporating treatments and their response. Finally, we describe how simulation modeling can be used to estimate risks of late effects, based on cumulative exposure from frontline and salvage treatment. The resultant database and tools employed are dynamic with the expectation that they will be updated as better and more complete information becomes available.

Reliable prediction of difficult airway for tracheal intubation from patient preoperative photographs by machine learning methods

BACKGROUND

Estimating the risk of a difficult tracheal intubation should help clinicians in better anaesthesia planning, to maximize patient safety. Routine bedside screenings suffer from low sensitivity.

OBJECTIVE

To develop and evaluate machine learning (ML) and deep learning (DL) algorithms for the reliable prediction of intubation risk, using information about airway morphology.

METHODS

Observational, prospective cohort study enrolling n=623 patients who underwent tracheal intubation: 53/623 difficult cases (prevalence 8.51%). First, we used our previously validated deep convolutional neural network (DCNN) to extract 2D image coordinates for 27 + 13 relevant anatomical landmarks in two preoperative photos (frontal and lateral views). Here we propose a method to determine the 3D pose of the camera with respect to the patient and to obtain the 3D world coordinates of these landmarks. Then we compute a novel set of dM=59 morphological features (distances, areas, angles and ratios), engineered with our anaesthesiologists to characterize each individual's airway anatomy towards prediction. Subsequently, here we propose four ad hoc ML pipelines for difficult intubation prognosis, each with four stages: feature scaling, imputation, resampling for imbalanced learning, and binary classification (Logistic Regression, Support Vector Machines, Random Forests and eXtreme Gradient Boosting). These compound ML pipelines were fed with the dM=59 morphological features, alongside dD=7 demographic variables. Here we trained them with automatic hyperparameter tuning (Bayesian search) and probability calibration (Platt scaling). In addition, we developed an ad hoc multi-input DCNN to estimate the intubation risk directly from each pair of photographs, i.e. without any intermediate morphological description. Performance was evaluated using optimal Bayesian decision theory. It was compared against experts' judgement and against state-of-the-art methods (three clinical formulae, four ML, four DL models).

RESULTS

Our four ad hoc ML pipelines with engineered morphological features achieved similar discrimination capabilities: median AUCs between 0.746 and 0.766. They significantly outperformed both expert judgement and all state-of-the-art methods (highest AUC at 0.716). Conversely, our multi-input DCNN yielded low performance due to overfitting. This same behaviour occurred for the state-of-the-art DL algorithms. Overall, the best method was our XGB pipeline, with the fewest false negatives at the optimal Bayesian decision threshold.

CONCLUSIONS

We proposed and validated ML models to assist clinicians in anaesthesia planning, providing a reliable calibrated estimate of airway intubation risk, which outperformed expert assessments and state-of-the-art methods. Our novel set of engineered features succeeded in providing informative descriptions for prognosis.

Patients' and physicians' beliefs and attitudes towards integrating personalized risk estimates into patient education about left ventricular assist device therapy

BACKGROUND

Personalized risk (PR) estimates may enhance clinical decision making and risk communication by providing individualized estimates of patient outcomes. We explored stakeholder attitudes toward the utility, acceptability, usefulness and best-practices for integrating PR estimates into patient education and decision making about Left Ventricular Assist Device (LVAD).

METHODS AND RESULTS

As part of a 5-year multi-institutional AHRQ project, we conducted 40 interviews with stakeholders (physicians, nurse coordinators, patients, and caregivers), analyzed using Thematic Content Analysis. All stakeholder groups voiced positive views towards integrating PR in decision making. Patients, caregivers and coordinators emphasized that PR can help to better understand a patient's condition and risks, prepare mentally and logistically for likely outcomes, and meaningfully engage in decision making. Physicians felt it can improve their decision making by enhancing insight into outcomes, enhance tailored pre-emptive care, increase confidence in decisions, and reduce bias and subjectivity. All stakeholder groups also raised concerns about accuracy, representativeness and relevance of algorithms; predictive uncertainty; utility in relation to physician's expertise; potential negative reactions among patients; and overreliance.

CONCLUSION

Stakeholders are optimistic about integrating PR into clinical decision making, but acceptability depends on prospectively demonstrating accuracy, relevance and evidence that benefits of PR outweigh potential negative impacts on decision making quality.

Applying models of care for total hip and knee arthroplasty: External validation of a published predictive model to identify extended stay risk prior to lower-limb arthroplasty

OBJECTIVE

This study aimed to externally validate a reported model for identifying patients requiring extended stay following lower limb arthroplasty in a new setting.

DESIGN

External validation of a previously reported prognostic model, using retrospective data.

SETTING

Medium-sized hospital orthopaedic department, Australia.

PARTICIPANTS

Electronic medical records were accessed for data collection between Sep-2019 and Feb-2020 and retrospective data extracted from 200 randomly selected total hip or knee arthroplasty patients.

INTERVENTION

Participants received total hip or knee replacement between 2-Feb-16 and 4-Apr-19. This study was a non-interventional retrospective study.

MAIN MEASURES

Model validation was assessed with discrimination, calibration on both original and adjusted forms of the candidate model. Decision curve analysis was conducted on the outputs of the adjusted model to determine net benefit at a predetermined decision threshold (0.5).

RESULTS

The original model performed poorly, grossly overestimating length of stay with mean calibration of -3.6 (95% confidence interval -3.9 to -3.2) and calibration slope of 0.52. Performance improved following adjustment of the model intercept and model coefficients (mean calibration 0.48, 95% confidence interval 0.16 to 0.80 and slope of 1.0), but remained poorly calibrated at low and medium risk threshold and net benefit was modest (three additional patients per hundred identified as at-risk) at the a-priori risk threshold.

CONCLUSIONS

External validation demonstrated poor performance when applied to a new patient population and would provide limited benefit for our institution. Implementation of predictive models for arthroplasty should include practical assessment of discrimination, calibration and net benefit at a clinically acceptable threshold.

Effect of recipient age on prioritisation for liver transplantation in the UK: a population-based modelling study

BACKGROUND

Following the introduction of an algorithm aiming to maximise life-years gained from liver transplantation in the UK (the transplant benefit score [TBS]), donor livers were redirected from younger to older patients, mortality rate equalised across the age range and short-term waiting list mortality reduced. Understanding age-related prioritisation has been challenging, especially for younger patients and clinicians allocating non-TBS-directed livers. We aimed to assess age-related prioritisation within the TBS algorithm by modelling liver transplantation prioritisation based on data from a UK transplant unit and comparing these data with other regions.

METHODS

In this population-based modelling study, serum parameters and age at liver transplantation assessment of patients attending the Scottish Liver Transplant Unit, Edinburgh, UK, between December, 2002, and November, 2023, were combined with representative synthetic data to model TBS survival predictions, which were compared according to age group (25-49 years vs ≥60 years), chronic liver disease severity, and disease cause. Models for end-stage liver disease (UKELD [UK], MELD [Eurotransplant region], and MELD 3.0 [USA]) were used as validated comparators of liver disease severity.

FINDINGS

Of 2093 patients with chronic liver disease, 1808 (86%) had complete datasets and liver disease parameters consistent with eligibility for the liver transplant waiting list in the UK (UKELD ≥49). Disease severity as assessed by UKELD, MELD, and MELD 3.0 did not differ by age (median UKELD scores of 56 for patients aged ≥60 years vs 56 for patients aged 25-49 years; MELD scores of 16 vs 16; and MELD 3.0 scores of 18 vs 18). TBS increased with advancing age (R=0·45, p<0·0001). TBS predicted that transplantation in patients aged 60 years or older would provide a two-fold greater net benefit at 5 years than in patients aged 25-49 years (median TBS 1317 [IQR 1116-1436] in older patients vs 706 [411-1095] in younger patients; p<0·0001). Older patients were predicted to have shorter survival without transplantation than younger patients (263 days [IQR 144-473] in older patients vs 861 days [448-1164] in younger patients; p<0·0001) but similar survival after transplantation (1599 days [1563-1628] vs 1573 days [1525-1614]; p<0·0001). Older patients could reach a TBS for which a liver offer was likely below minimum criteria for transplantation (UKELD <49), whereas many younger patients were required to have high-urgent disease (UKELD >60). US and Eurotransplant programmes did not prioritise according to age.

INTERPRETATION

The UK liver allocation algorithm prioritises older patients for transplantation by predicting that advancing age increases the benefit from liver transplantation. Restricted follow-up and biases in waiting list data might limit the accuracy of these benefit predictions. Measures beyond overall waiting list mortality are required to fully capture the benefits of liver transplantation.

FUNDING

None.

A real-time prognostic model for venous thromboembolic events among hospitalized adults

Background

Hospital-acquired venous thromboembolism (HA-VTE) is a leading cause of morbidity and mortality among hospitalized adults. Guidelines recommend use of a risk-prediction model to estimate HA-VTE risk for individual patients. Extant models do not perform well for broad patient populations and are not conducive to automation in clinical practice.

Objectives

To develop an automated, real-time prognostic model for venous thromboembolism during hospitalization among all adult inpatients using readily available data from the electronic health record.

Methods

The derivation cohort included inpatient hospitalizations ("encounters") for patients ≥16 years old at Vanderbilt University Medical Center between 2018 and 2020 (n = 132,330). HA-VTE events were identified using International Classification of Diseases, 10th Revision, codes. The prognostic model was developed using least absolute shrinkage and selection operator regression. Temporal external validation was performed in a validation cohort of encounters between 2021 and 2022 (n = 62,546). Prediction performance was assessed by discrimination accuracy (C statistic) and calibration (integrated calibration index).

Results

There were 1187 HA-VTEs in the derivation cohort (9.0 per 1000 encounters) and 864 in the validation cohort (13.8 per 1000 encounters). The prognostic model included 25 variables, with placement of a central line among the most important predictors. Prediction performance of the model was excellent (C statistic, 0.891; 95% CI, 0.882-0.900; integrated calibration index, 0.001). The model performed similarly well across subgroups of patients defined by age, sex, race, and type of admission.

Conclusion

This fully automated prognostic model uses readily available data from the electronic health record, exhibits superior prediction performance compared with existing models, and generates granular risk stratification in the form of a predicted probability of HA-VTE for each patient.

Development and validation of risk prediction model for recurrent cardiovascular events among Chinese: the Personalized CARdiovascular DIsease risk Assessment for Chinese model

Aims

Cardiovascular disease (CVD) is a leading cause of mortality, especially in developing countries. This study aimed to develop and validate a CVD risk prediction model, Personalized CARdiovascular DIsease risk Assessment for Chinese (P-CARDIAC), for recurrent cardiovascular events using machine learning technique.

Methods and results

Three cohorts of Chinese patients with established CVD were included if they had used any of the public healthcare services provided by the Hong Kong Hospital Authority (HA) since 2004 and categorized by their geographical locations. The 10-year CVD outcome was a composite of diagnostic or procedure codes with specific International Classification of Diseases, Ninth Revision, Clinical Modification. Multivariate imputation with chained equations and XGBoost were applied for the model development. The comparison with Thrombolysis in Myocardial Infarction Risk Score for Secondary Prevention (TRS-2°P) and Secondary Manifestations of ARTerial disease (SMART2) used the validation cohorts with 1000 bootstrap replicates. A total of 48 799, 119 672 and 140 533 patients were included in the derivation and validation cohorts, respectively. A list of 125 risk variables were used to make predictions on CVD risk, of which 8 classes of CVD-related drugs were considered interactive covariates. Model performance in the derivation cohort showed satisfying discrimination and calibration with a C statistic of 0.69. Internal validation showed good discrimination and calibration performance with C statistic over 0.6. The P-CARDIAC also showed better performance than TRS-2°P and SMART2.

Conclusion

Compared with other risk scores, the P-CARDIAC enables to identify unique patterns of Chinese patients with established CVD. We anticipate that the P-CARDIAC can be applied in various settings to prevent recurrent CVD events, thus reducing the related healthcare burden.

Prediction of Morbidity and Mortality After Esophagectomy: A Systematic Review

BACKGROUND

Esophagectomy for esophageal cancer has a complication rate of up to 60%. Prediction models could be helpful to preoperatively estimate which patients are at increased risk of morbidity and mortality. The objective of this study was to determine the best prediction models for morbidity and mortality after esophagectomy and to identify commonalities among the models.

PATIENTS AND METHODS

A systematic review was performed in accordance to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement and was prospectively registered in PROSPERO ( https://www.crd.york.ac.uk/prospero/ , study ID CRD42022350846). Pubmed, Embase, and Clarivate Analytics/Web of Science Core Collection were searched for studies published between 2010 and August 2022. The Prediction model Risk of Bias Assessment Tool was used to assess the risk of bias. Extracted data were tabulated and a narrative synthesis was performed.

RESULTS

Of the 15,011 articles identified, 22 studies were included using data from tens of thousands of patients. This systematic review included 33 different models, of which 18 models were newly developed. Many studies showed a high risk of bias. The prognostic accuracy of models differed between 0.51 and 0.85. For most models, variables are readily available. Two models for mortality and one model for pulmonary complications have the potential to be developed further.

CONCLUSIONS

The availability of rigorous prediction models is limited. Several models are promising but need to be further developed. Some models provide information about risk factors for the development of complications. Performance status is a potential modifiable risk factor. None are ready for clinical implementation.

Predicting short- to medium-term care home admission risk in older adults: a systematic review of externally validated models

INTRODUCTION

Predicting risk of care home admission could identify older adults for early intervention to support independent living but require external validation in a different dataset before clinical use. We systematically reviewed external validations of care home admission risk prediction models in older adults.

METHODS

We searched Medline, Embase and Cochrane Library until 14 August 2023 for external validations of prediction models for care home admission risk in adults aged ≥65 years with up to 3 years of follow-up. We extracted and narratively synthesised data on study design, model characteristics, and model discrimination and calibration (accuracy of predictions). We assessed risk of bias and applicability using Prediction model Risk Of Bias Assessment Tool.

RESULTS

Five studies reporting validations of nine unique models were included. Model applicability was fair but risk of bias was mostly high due to not reporting model calibration. Morbidities were used as predictors in four models, most commonly neurological or psychiatric diseases. Physical function was also included in four models. For 1-year prediction, three of the six models had acceptable discrimination (area under the receiver operating characteristic curve (AUC)/c statistic 0.70-0.79) and the remaining three had poor discrimination (AUC < 0.70). No model accounted for competing mortality risk. The only study examining model calibration (but ignoring competing mortality) concluded that it was excellent.

CONCLUSIONS

The reporting of models was incomplete. Model discrimination was at best acceptable, and calibration was rarely examined (and ignored competing mortality risk when examined). There is a need to derive better models that account for competing mortality risk and report calibration as well as discrimination.

Predictive Models to Assess Risk of Persistent Opioid Use, Opioid Use Disorder, and Overdose

BACKGROUND

This systematic review summarizes the development, accuracy, quality, and clinical utility of predictive models to assess the risk of opioid use disorder (OUD), persistent opioid use, and opioid overdose.

METHODS

In accordance with Preferred Reporting Items for a Systematic Review and Meta-analysis guidelines, 8 electronic databases were searched for studies on predictive models and OUD, overdose, or persistent use in adults until June 25, 2023. Study selection and data extraction were completed independently by 2 reviewers. Risk of bias of included studies was assessed independently by 2 reviewers using the Prediction model Risk of Bias ASsessment Tool (PROBAST).

RESULTS

The literature search yielded 3130 reports; after removing 199 duplicates, excluding 2685 studies after abstract review, and excluding 204 studies after full-text review, the final sample consisted of 41 studies that developed more than 160 predictive models. Primary outcomes included opioid overdose (31.6% of studies), OUD (41.4%), and persistent opioid use (17%). The most common modeling approach was regression modeling, and the most common predictors included age, sex, mental health diagnosis history, and substance use disorder history. Most studies reported model performance via the c statistic, ranging from 0.507 to 0.959; gradient boosting tree models and neural network models performed well in the context of their own study. One study deployed a model in real time. Risk of bias was predominantly high; concerns regarding applicability were predominantly low.

CONCLUSIONS

Models to predict opioid-related risks are developed using diverse data sources and predictors, with a wide and heterogenous range of accuracy metrics. There is a need for further research to improve their accuracy and implementation.

Development and External Validation of Models to Predict Persistent Hypoxemic Respiratory Failure for Clinical Trial Enrichment

OBJECTIVES

Improving the efficiency of clinical trials in acute hypoxemic respiratory failure (HRF) depends on enrichment strategies that minimize enrollment of patients who quickly resolve with existing care and focus on patients at high risk for persistent HRF. We aimed to develop parsimonious models predicting risk of persistent HRF using routine data from ICU admission and select research immune biomarkers.

DESIGN

Prospective cohorts for derivation ( n = 630) and external validation ( n = 511).

SETTING

Medical and surgical ICUs at two U.S. medical centers.

PATIENTS

Adults with acute HRF defined as new invasive mechanical ventilation (IMV) and hypoxemia on the first calendar day after ICU admission.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We evaluated discrimination, calibration, and practical utility of models predicting persistent HRF risk (defined as ongoing IMV and hypoxemia on the third calendar day after admission): 1) a clinical model with least absolute shrinkage and selection operator (LASSO) selecting Pa o2 /F io2 , vasopressors, mean arterial pressure, bicarbonate, and acute respiratory distress syndrome as predictors; 2) a model adding interleukin-6 (IL-6) to clinical predictors; and 3) a comparator model with Pa o2 /F io2 alone, representing an existing strategy for enrichment. Forty-nine percent and 69% of patients had persistent HRF in derivation and validation sets, respectively. In validation, both LASSO (area under the receiver operating characteristic curve, 0.68; 95% CI, 0.64-0.73) and LASSO + IL-6 (0.71; 95% CI, 0.66-0.76) models had better discrimination than Pa o2 /F io2 (0.64; 95% CI, 0.59-0.69). Both models underestimated risk in lower risk deciles, but exhibited better calibration at relevant risk thresholds. Evaluating practical utility, both LASSO and LASSO + IL-6 models exhibited greater net benefit in decision curve analysis, and greater sample size savings in enrichment analysis, compared with Pa o2 /F io2 . The added utility of LASSO + IL-6 model over LASSO was modest.

CONCLUSIONS

Parsimonious, interpretable models that predict persistent HRF may improve enrichment of trials testing HRF-targeted therapies and warrant future validation.

The role of biomarkers in the early detection of pancreatic cancer

Pancreatic surveillance can detect early-stage pancreatic cancer and achieve long-term survival, but currently involves annual endoscopic ultrasound and MRI/MRCP, and is recommended only for individuals who meet familial/genetic risk criteria. To improve upon current approaches to pancreatic cancer early detection and to expand access, more accurate, inexpensive, and safe biomarkers are needed, but finding them has remained elusive. Newer approaches to early detection, such as using gene tests to personalize biomarker interpretation, and the increasing application of artificial intelligence approaches to integrate complex biomarker data, offer promise that clinically useful biomarkers for early pancreatic cancer detection are on the horizon.

Integrating molecular and clinical variables to predict myocardial recovery

Mechanical unloading and circulatory support with left ventricular assist devices (LVADs) mediate significant myocardial improvement in a subset of advanced heart failure (HF) patients. The clinical and biological phenomena associated with cardiac recovery are under intensive investigation. Left ventricular (LV) apical tissue, alongside clinical data, were collected from HF patients at the time of LVAD implantation (n=208). RNA was isolated and mRNA transcripts were identified through RNA sequencing and confirmed with RT-qPCR. To our knowledge this is the first study to combine transcriptomic and clinical data to derive predictors of myocardial recovery. We used a bioinformatic approach to integrate 59 clinical variables and 22,373 mRNA transcripts at the time of LVAD implantation for the prediction of post-LVAD myocardial recovery defined as LV ejection fraction (LVEF) ≥40% and LV end-diastolic diameter (LVEDD) ≤5.9cm, as well as functional and structural LV improvement independently by using LVEF and LVEDD as continuous variables, respectively. To substantiate the predicted variables, we used a multi-model approach with logistic and linear regressions. Combining RNA and clinical data resulted in a gradient boosted model with 80 features achieving an AUC of 0.731±0.15 for predicting myocardial recovery. Variables associated with myocardial recovery from a clinical standpoint included HF duration, pre-LVAD LVEF, LVEDD, and HF pharmacologic therapy, and LRRN4CL (ligand binding and programmed cell death) from a biological standpoint. Our findings could have diagnostic, prognostic, and therapeutic implications for advanced HF patients, and inform the care of the broader HF population.

Development and Validation of Two-Step Prediction Models for Postoperative Bedridden Status in Geriatric Intertrochanteric Hip Fractures

Patients with intertrochanteric hip fractures are at an elevated risk of becoming bedridden compared with those with intraarticular hip fractures. Accurate risk assessments can help clinicians select postoperative rehabilitation strategies to mitigate the risk of bedridden status. This study aimed to develop a two-step prediction model to predict bedridden status at 3 months postoperatively: one model (first step) for prediction at the time of admission to help dictate postoperative rehabilitation plans; and another (second step) for prediction at the time before discharge to determine appropriate discharge destinations and home rehabilitation programs. Three-hundred and eighty-four patients were retrospectively reviewed and divided into a development group (n = 291) and external validation group (n = 93). We developed a two-step prediction model to predict the three-month bedridden status of patients with intertrochanteric fractures from the development group. The first (preoperative) model incorporated four simple predictors: age, dementia, American Society of Anesthesiologists physical status classification (ASA), and pre-fracture ambulatory status. The second (predischarge) model used an additional predictor, ambulation status before discharge. Model performances were evaluated using the external validation group. The preoperative model performances were area under ROC curve (AUC) = 0.72 (95%CI 0.61-0.83) and calibration slope = 1.22 (0.40-2.23). The predischarge model performances were AUC = 0.83 (0.74-0.92) and calibration slope = 0.89 (0.51-1.35). A decision curve analysis (DCA) showed a positive net benefit across a threshold probability between 10% and 35%, with a higher positive net benefit for the predischarge model. Our prediction models demonstrated good discrimination, calibration, and net benefit gains. Using readily available predictors for prognostic prediction can assist clinicians in planning individualized postoperative rehabilitation programs, home-based rehabilitation programs, and determining appropriate discharge destinations, especially in environments with limited resources.

A Systematic Review of Predictor Composition, Outcomes, Risk of Bias, and Validation of COVID-19 Prognostic Scores

BACKGROUND

Numerous prognostic scores have been published to support risk stratification for patients with coronavirus disease 2019 (COVID-19).

METHODS

We performed a systematic review to identify the scores for confirmed or clinically assumed COVID-19 cases. An in-depth assessment and risk of bias (ROB) analysis (Prediction model Risk Of Bias ASsessment Tool [PROBAST]) was conducted for scores fulfilling predefined criteria ([I] area under the curve [AUC)] ≥ 0.75; [II] a separate validation cohort present; [III] training data from a multicenter setting [≥2 centers]; [IV] point-scale scoring system).

RESULTS

Out of 1522 studies extracted from MEDLINE/Web of Science (20/02/2023), we identified 242 scores for COVID-19 outcome prognosis (mortality 109, severity 116, hospitalization 14, long-term sequelae 3). Most scores were developed using retrospective (75.2%) or single-center (57.1%) cohorts. Predictor analysis revealed the primary use of laboratory data and sociodemographic information in mortality and severity scores. Forty-nine scores were included in the in-depth analysis. The results indicated heterogeneous quality and predictor selection, with only five scores featuring low ROB. Among those, based on the number and heterogeneity of validation studies, only the 4C Mortality Score can be recommended for clinical application so far.

CONCLUSIONS

The application and translation of most existing COVID scores appear unreliable. Guided development and predictor selection would have improved the generalizability of the scores and may enhance pandemic preparedness in the future.

Should we use nomograms for risk predictions in diffuse large B cell lymphoma patients? A systematic review

Models based on risk stratification are increasingly reported for Diffuse large B cell lymphoma (DLBCL). Due to a rising interest in nomograms for cancer patients, we aimed to review and critically appraise prognostic models based on nomograms in DLBCL patients. A literature search in PubMed/Embase identified 59 articles that proposed prognostic models for DLBCL by combining parameters of interest (e.g., clinical, laboratory, immunohistochemical, and genetic) between January 2000 and 2024. Of them, 40 studies proposed different gene expression signatures and incorporated them into nomogram-based prognostic models. Although most studies assessed discrimination and calibration when developing the model, many lacked external validation. Current nomogram-based models for DLBCL are mainly developed from publicly available databases, lack external validation, and have no applicability in clinical practice. However, they may be helpful in individual patient counseling, although careful considerations should be made regarding model development due to possible limitations when choosing nomograms for prognostication.