A new framework to enhance the interpretation of external validation studies of clinical prediction models

Predicting Danish EQ-5D-5L Utilities Based on United Kingdom EQ-5D-3L Utilities for Use in Health Economic Models

OBJECTIVES

Since 2021, the Danish Medicines Council recommends the use of the Danish EQ-5D-5L value set when estimating utilities. The aim of this research was to develop and validate an algorithm that can accurately predict mean Danish EQ-5D-5L utilities based on published mean UK EQ-5D-3L utilities.

METHODS

The study design incorporated a secondary analysis of patient-level UK EQ-5D-3L utility index scores from 11 oncology clinical trials. The EQ-5D-3L responses were mapped to EQ-5D-5L responses with the van Hout and Shaw preferred mapping algorithm. Model fitting and internal cross-validation were completed on a pooled dataset formed from eight trials including a total of 30,755 EQ-5D-3L responses. Three other trials were used for external validation (21,587 EQ-5D-3L observations).

RESULTS

From the model fitting phase, a simple linear model for mean utility scores exhibited good fit and was selected as the optimal prediction algorithm. External validation using the algorithm to predict mean Danish EQ-5D-5L utilities was excellent, with the largest absolute prediction error being 0.020 (observed UK EQ-5D-3L means: 0.628-0.835).

CONCLUSIONS

The prediction algorithm developed in this research can increase analysts' ability to apply utilities aligned with the Danish EQ-5D-5L value set and guideline recommendations, reducing decision uncertainty. Many health technology assessment (HTA) institutions are transitioning from the EQ-5D-3L to the EQ-5D-5L in the coming years; therefore, prediction algorithms are likely of interest to additional HTA institutions in the near future. This study can provide a blueprint for future studies.

Machine learning-based prediction models for falls in hospitalized patients: A systematic review and meta-analysis

OBJECTIVE

To systematically review published machine learning models aimed at predicting the risk of falls among hospitalized patients.

DESIGN

A systematic review and meta-analysis.

METHODS

According to the inclusion and exclusion criteria, we comprehensively searched the database PubMed, EMBASE, Web of Science and The Cochrane library from inception through November 14, 2023. Data extraction followed the CHARMS checklist, and bias risk and applicability were assessed using the PROBAST tool. A meta-analysis was performed utilizing Meta-disc software, with the area under the curve, sensitivity, and specificity serving as the effect measures. Heterogeneity was assessed through the Chi-square test and I2 test.

RESULTS

A systematic search yielded a total of 2007 studies, 14 of which were selected following screening, and 13 of these studies were subjected to quantitative analyses. The incidence rate of inpatient falls ranged from 0.14 % to 50.69 %, with corresponding area under the curve (AUC) values varying between 0.57 and 0.99. Age, multiple drugs, emerged as the most frequently employed predictive factors. While the overall quality of the studies was considered satisfactory, a high risk of bias was identified, primarily attributed to insufficient reporting in the participant and analysis domains. The combined AUC of the 13 predictive models was 0.82, with a sensitivity of 0.69 (95 % CI [0.68-0.7]) and a specificity of 0.70 (95 % CI [0.70-0.71]), indicating robust discriminative performance.

CONCLUSION

Although machine learning models provide an emerging and promising method for predicting hospital falls, they require broader validation to ensure practical applicability. This review highlights the potential drawbacks of current methods, including high risk of bias and low reproducibility, and provides various recommendations on how to address these challenges.

CLINICAL RELEVANCE

Falls are a frequent and significant issue for patients in hospitals, often resulting in severe physical harm and longer hospital stays. This research offers a fresh approach and tool for medical professionals by thoroughly examining how machine learning models can predict falls, aiming to enhance personalized and precise fall risk management.

Explanation and elaboration of MedinAI: guidelines for reporting artificial intelligence studies in medicines, pharmacotherapy, and pharmaceutical services

The increasing adoption of artificial intelligence (AI) in medicines, pharmacotherapy, and pharmaceutical services necessitates clear guidance on reporting standards. While the MedinAI Statement (Bottacin in Int J Clin Pharm, https://doi.org/10.1007/s11096-025-01905-3, 2025) provides core guidelines for reporting AI studies in these fields, detailed explanations and practical examples are crucial for optimal implementation. This companion document was developed to offer comprehensive guidance and real-world examples for each guideline item. The document elaborates on all 14 items and 78 sub-items across four domains: core, ethical considerations in medication and pharmacotherapy, medicines as products, and services related to medicines and pharmacotherapy. Through clear, actionable guidance and diverse examples, this document enhances MedinAI's utility, enabling researchers and stakeholders to improve the quality and transparency of AI research reporting across various contexts, study designs, and development stages.

Predicting prolonged work absence due to musculoskeletal disorders: development, validation, and clinical usefulness of prognostic prediction models

PURPOSE

Given the lack of robust prognostic models for early identification of individuals at risk of work disability, this study aimed to develop and externally validate three models for prolonged work absence among individuals on sick leave due to musculoskeletal disorders.

METHODS

We developed three multivariable logistic regression models using data from 934 individuals on sick leave for 4-12 weeks due to musculoskeletal disorders, recruited through the Norwegian Labour and Welfare Administration. The models predicted three outcomes: (1) > 90 consecutive sick days, (2) > 180 consecutive sick days, and (3) any new or increased work assessment allowance or disability pension within 12 months. Each model was externally validated in a separate cohort of participants (8-12 weeks of sick leave) from a different geographical region in Norway. We evaluated model performance using discrimination (c-statistic), calibration, and assessed clinical usefulness using decision curve analysis (net benefit). Bootstrapping was used to adjust for overoptimism.

RESULTS

All three models showed good predictive performance in the external validation sample, with c-statistics exceeding 0.76. The model predicting > 180 days performed best, demonstrating good calibration and discrimination (c-statistic 0.79 (95% CI 0.73-0.85), and providing net benefit across a range of decision thresholds from 0.10 to 0.80.

CONCLUSIONS

These models, particularly the one predicting > 180 days, may facilitate secondary prevention strategies and guide future clinical trials. Further validation and refinement are necessary to optimise the models and to test their performance in larger samples.

Multimodal convolutional neural networks for the prediction of acute kidney injury in the intensive care

Increased monitoring of health-related data for ICU patients holds great potential for the early prediction of medical outcomes. Research on whether the use of clinical notes and concepts from knowledge bases can improve the performance of prediction models is limited. We investigated the effects of combining clinical variables, clinical notes, and clinical concepts. We focus on the early prediction of Acute Kidney Injury (AKI) in the intensive care unit (ICU). AKI is a sudden reduction in kidney function measured by increased serum creatinine (SCr) or decreased urine output. AKI may occur in up to 30% of ICU stays. We developed three models based on convolutional neural networks using data from the Medical Information Mart for Intensive Care (MIMIC) database. The models used clinical variables, free-text notes, and concepts from the Elsevier H-Graph. Our models achieved good predictive performance (AUROC 0.73-0.90). These models were assessed both when using Scr and urine output as predictors and when omitting them. When Scr and urine output were used as predictors, models that included clinical notes and concepts together with clinical variables performed on par with models that only used clinical variables. When excluding SCr and urine output, predictive performance improved by combining multiple modalities. The models that used only clinical variables were externally validated on the eICU dataset and transported fairly to the new population (AUROC 0.68-0.77). Our in-depth comparison of modalities and text representations may further guide researchers and practitioners in applying multimodal models for predicting AKI and inspire them to investigate multimodality and contextualized embeddings for other tasks. Our models can support clinicians to promptly recognize and treat deteriorating AKI patients and may improve patient outcomes in the ICU.

Clinical prediction models for in vitro fertilization outcomes: a systematic review, meta-analysis, and external validation

STUDY QUESTION

What is the best-performing model currently predicting live birth outcomes for IVF or ICSI?

SUMMARY ANSWER

Among the identified prognostic models, McLernon's post-treatment model outperforms other models in both the meta-analysis and external validation of a Chinese cohort.

WHAT IS KNOWN ALREADY

With numerous similar models available across different time periods and using various predictors in IVF prognostic models, there is a need to summarize and evaluate them, due to a lack of validated evidence distinguishing high-quality from low-quality prediction tools. However, there is a notable dearth of research in the form of meta-analysis or external validation assessing the performance of models in predicting live births in this field.

STUDY DESIGN, SIZE, DURATION

The researchers conducted a comprehensive literature review in PubMed, EMBASE, and Web of Science, using keywords related to prognostic models and IVF/ICSI live birth outcomes. The search included studies published up to 3 April 2024, and was limited to English language studies.

PARTICIPANTS/MATERIALS, SETTING, METHODS

The review included studies that developed or validated prognostic models for IVF live birth outcomes while providing clear reports on model characteristics. Researchers extracted and analysed the data in accordance with the guidelines outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses and other model-related guidelines. For model effects in meta-analysis, the choice would be based on the heterogeneity assessed using the I2 statistic and the Cochrane Q test. Model performance was evaluated by assessing their area under the receiver operating characteristic curves (AUCs) and calibration plots in the studies.

MAIN RESULTS AND THE ROLE OF CHANCE

This review provides a comprehensive summary of data derived from 72 studies with an overall ROB of high or unclear. These studies contained a total of 132 predictors and 86 prognostic models, and then meta-analyses were performed for each of the five selected models. The total random effects of Templeton's, Nelson's, McLernon's pre-treatment and post-treatment model demonstrated AUCs of 0.65 (95% CI: 0.61-0.69), 0.63 (95% CI: 0.63-0.64), 0.67 (95% CI: 0.62-0.71), and 0.73 (95% CI: 0.71-0.75), respectively. The total fixed effects of the intelligent data analysis score (iDAScore) model estimated an AUC of 0.66 (95% CI: 0.63-0.68). The external validation of the initial four models in our cohort produced AUCs ranging from 0.53 to 0.58, and the calibration was confirmed through calibration plots.

LIMITATIONS, REASONS FOR CAUTION

While the focus on English-language studies and live birth outcomes may constrain the generalizability of the findings to diverse populations, this approach equips clinicians, who view live births as the ultimate objective, with more precise and actionable reference guidelines.

WIDER IMPLICATIONS OF THE FINDINGS

This study represents the first meta-analysis in the field of IVF prediction models, definitively confirming the superior performance of McLernon's post-treatment model. The conclusion is reinforced by independent validation from another perspective. Nevertheless, further investigation is warranted to develop new models and to externally validate existing high-performing models for prognostic accuracy in IVF outcomes.

STUDY FUNDING/COMPETING INTEREST(S)

This study was supported by the National Natural Science Foundation of China (Grant No. 82174517). The authors report no conflict of interest.

REGISTRATION NUMBER

2022 CRD42022312018.

Deep neural networks excel in COVID-19 disease severity prediction-a meta-regression analysis

COVID-19 is a disease in which early prognosis of severity is critical for desired patient outcomes and for the management of limited resources like intensive care unit beds and ventilation equipment. Many prognostic statistical tools have been developed for the prediction of disease severity, but it is still unclear which ones should be used in practice. We aim to guide clinicians in choosing the best available tools to make optimal decisions and assess their role in resource management and assess what can be learned from the COVID-19 scenario for development of prediction models in similar medical applications. Using the five major medical databases: MEDLINE (via PubMed), Embase, Cochrane Library (CENTRAL), Cochrane COVID-19 Study Register, and Scopus, we conducted a comprehensive systematic review of prediction tools between 2020 January and 2023 April for hospitalized COVID-19 patients. We identified both the relevant confounding factors of tool performance using the MetaForest algorithm and the best tools-comparing linear, machine learning, and deep learning methods-with mixed-effects meta-regression models. The risk of bias was evaluated using the PROBAST tool. Our systematic search identified eligible 27,312 studies, out of which 290 were eligible for data extraction, reporting on 430 independent evaluations of severity prediction tools with ~ 2.8 million patients. Neural Network-based tools have the highest performance with a pooled AUC of 0.893 (0.748-1.000), 0.752 (0.614-0.853) sensitivity, 0.914 (0.849-0.952) specificity, using clinical, laboratory, and imaging data. The relevant confounders of performance are the geographic region of patients, the rate of severe cases, and the use of C-Reactive Protein as input data. 88% of studies have a high risk of bias, mostly because of deficiencies in the data analysis. All investigated tools in use aid decision-making for COVID-19 severity prediction, but Machine Learning tools, specifically Neural Networks clearly outperform other methods, especially in cases when the basic characteristics of severe and non-severe patient groups are similar, and without the need for more data. When highly specific biomarkers are not available-such as in the case of COVID-19-practitioners should abandon general clinical severity scores and turn to disease specific Machine Learning tools.

Unmasking the chameleons: A benchmark for out-of-distribution detection in medical tabular data

BACKGROUND

Machine Learning (ML) models often struggle to generalize effectively to data that deviates from the training distribution. This raises significant concerns about the reliability of real-world healthcare systems encountering such inputs known as out-of-distribution (OOD) data. These concerns can be addressed by real-time detection of OOD inputs. While numerous OOD detection approaches have been suggested in other fields - especially in computer vision - it remains unclear whether similar methods effectively address challenges posed by medical tabular data.

OBJECTIVE

To answer this important question, we propose an extensive reproducible benchmark to compare different OOD detection methods in medical tabular data across a comprehensive suite of tests.

METHOD

To achieve this, we leverage 4 different and large public medical datasets, including eICU and MIMIC-IV, and consider various kinds of OOD cases within these datasets. For example, we examine OODs originating from a statistically different dataset than the training set according to the membership model introduced by Debray et al. [1], as well as OODs obtained by splitting a given dataset based on a value of a distinguishing variable. To identify OOD instances, we explore a range of 10 density-based methods that learn the marginal distribution of the data, alongside 17 post-hoc detectors that are applied on top of prediction models already trained on the data. The prediction models involve three distinct architectures, namely MLP, ResNet, and Transformer.

MAIN RESULTS

In our experiments, when the membership model achieved an AUC of 0.98, which indicated a clear distinction between OOD data and the training set, we observed that the OOD detection methods had achieved AUC values exceeding 0.95 in distinguishing OOD data. In contrast, in the experiments with subtler changes in data distribution such as selecting OOD data based on ethnicity and age characteristics, many OOD detection methods performed similarly to a random classifier with AUC values close to 0.5. This may suggest a correlation between separability, as indicated by the membership model, and OOD detection performance, as indicated by the AUC of the detection model. This warrants future research.

External validation of the "HOSPITAL simplified" scale as a predictor of 30-day readmission after hospitalisation in OSI Araba Medical Services

BACKGROUND

A readmission is defined as a new admission to hospital for at least one night in the 30 days following a discharge. Some are caused by events unrelated to the reason for the initial admission (unavoidable) and others caused by the same pathology (related). Different predictive models allow the identification of patients at higher risk of readmission. The "HOSPITAL simplified" score stands out for its performance and simplicity.

AIM

To validate the "HOSPITAL simplified" scale in a cohort of patients at the University Hospital of Alava.

METHOD

Review of patients discharged from OSI Araba Medical Services in June and November 2023. The scale was applied to each of them and re-admission was studied in the 30 days following discharge. The Brier Score, discrimination and calibration capacity were analysed, using the guidelines of the TRIPOD declaration.

RESULTS

In 1849 discharges there were 240 unscheduled readmissions: 73 for causes unrelated to the previous admission and 151 related. For predicting related readmissions the scale obtained a Brier score = 0.0703. The mean score for cases with related readmission was 4.7 versus 2.66 for cases without readmission (P < 0.001). The area under the ROC curve = 0.747. Related readmission occurred in 29.3% of patients at high risk, compared to 17.3% predicted by the scale.

CONCLUSIONS

The "HOSPITAL simplified" scale showed in OSI Araba a higher discrimination capacity than the original series although it underestimated the real risk. The scale offers good portability.

Prediction of cardiovascular risk: validation of a non-laboratory and a laboratory-based score in a Brazilian community-based cohort of the PURE study

Background

Risk scores are essential tools for implementing cardiovascular disease (CVD) prevention. Validating risk scores considering regional diversities and disparities is critical for reducing the burden of CVD on global morbidity and mortality. We aimed to validate two cardiovascular risk scores (laboratory and non-laboratory-based) to predict major adverse cardiovascular events in the Brazilian cohort of the PURE study.

Methods

We validated two risk scores derived from the INTERHEART study, the non-laboratory INTERHEART risk score (NL-IHRS) and the laboratory fasting cholesterol INTERHEART risk score (FC-IHRS) using data from 4623 (urban areas) and 1415 (rural areas) participants without CVD in the Brazilian cohort of the PURE study enrolled in 2004 and 2005 and followed up to September 2021. The endpoint was major cardiovascular events (MACE), defined as the composite of myocardial infarction, stroke, heart failure, or death from cardiovascular causes. We evaluated the model performance of IHRS through c-statistic and calibration methods.

Findings

After a mean follow-up of 8.8 years (range, 0.28-15.1 years), there were 312 cardiovascular events, corresponding to an incidence rate of 0.58% per year (0.56% per year in urban versus 0.64% per year in rural areas). For the NL-IHRS, the c-statistic was 0.69 (95% confidence interval, CI, 0.66-0.72) in the overall cohort, 0.68 (95% CI, 0.64-0.72) in the urban cohort, and 0.72 (95% CI, 0.66-0.78) in the rural cohort. C-statistic values for the recalibrated FC-IHRS were 0.71 (95% CI, 0.67-0.74), 0.71 (95% CI, 0.67-0.75), and 0.70 (95% CI, 0.64-0.76) in the overall, urban, and rural cohorts, respectively.

Interpretation

In this Brazilian community-based prospective cohort, both NL-IHRS and FC-IHRS-based models performed with reasonable discriminative accuracy on the risk estimation of long-term risk of major CVD. A non-laboratory-based CVD risk score may be instrumental in Brazilian communities with limited access to medical resources.

Funding

Population Health Research Institute, Novartis Biociências S.A.

Can artificial intelligence be the future solution to the enormous challenges and suffering caused by Schizophrenia?

This study evaluated the potential of artificial intelligence (AI) in the diagnosis, treatment, and prognostic assessment of schizophrenia (SZ) and explored collaborative directions for AI applications in future medical innovations. SZ is a severe mental disorder that causes significant suffering and imposes challenges on patients. With the rapid advancement of machine learning and deep learning technologies, AI has demonstrated notable advantages in the early diagnosis of high-risk populations. By integrating multidimensional biomarkers and linguistic behavior data of patients, AI can provide further objective and precise diagnostic criteria. Moreover, it aids in formulating personalized treatment plans, enhancing therapeutic outcomes, and offering new therapeutic strategies for patients with treatment-resistant SZ. Furthermore, AI excels in developing individualized prognostic plans, which enables the rapid identification of disease progression, accurate prediction of disease trajectory, and timely adjustment of treatment strategies, thereby improving prognosis and facilitating recovery. Despite the immense potential of AI in SZ management, its role as an auxiliary tool must be emphasized, with clinical judgment and compassionate care from healthcare professionals remaining crucial. Future research should focus on optimizing human-machine interactions to achieve efficient AI application in SZ management. The in-depth integration of AI technology into clinical practice will advance the field of SZ, ultimately improving the quality of life and treatment outcomes of patients.

The Barts Surgical Infection Risk (B-SIR) tool: external validation and comparison with existing tools to predict surgical site infection after cardiac surgery

OBJECTIVE

Further to previous development and internal validation of the Barts Surgical Infection Risk (B-SIR) tool, this study sought to explore the external validity of the B-SIR tool and compare it with the Australian Clinical Risk Index (ACRI), and the Brompton and Harefield Infection Score (BHIS).

STUDY DESIGN AND SETTING

This multi-centre retrospective analysis of prospectively collected local data included adult (age ≥18 years) patients undergoing cardiac surgery between January 2018 and December 2019. Pre-pandemic data were used as a reflection of standard practice. Area under the curve (AUC) was used to validate and compare the predictive power of the scores, and calibration was assessed using the Hosmer-Lemeshow test and calibration plots.

RESULTS

In total, 6022 patients from three centres were included in the complete case analysis. The mean age was 66 years, 75% were men and 3.19% developed a surgical site infection (SSI). The B-SIR tool had an area under the curve (AUC) of 0.686 [95% confidence interval (CI) 0.649-0.723], similar to the developmental study (AUC=0.682, 95% CI 0.652-0.713). This was significantly higher than the BHIS AUC of 0.610 (95% CI 0.045-0.109; P<0.001) and the ACRI AUC of 0.614 (95% CI 0.041-0.103; P<0.001). After recalibration using a correction factor, the B-SIR tool gave accurate risk predictions (Hosmer-Lemeshow test P=0.423). The multiple imputation result (AUC=0.676, 95% CI 0.639-0.712) was similar to development data, and higher than the ACRI and BHIS.

CONCLUSION

External validation indicated that the B-SIR tool predicted SSI after cardiac surgery better than the ACRI and BHIS. This suggests that the B-SIR tool could be useful for use in routine practice.

Performance of Radiomics-based machine learning and deep learning-based methods in the prediction of tumor grade in meningioma: a systematic review and meta-analysis

Currently, the World Health Organization (WHO) grade of meningiomas is determined based on the biopsy results. Therefore, accurate non-invasive preoperative grading could significantly improve treatment planning and patient outcomes. Considering recent advances in machine learning (ML) and deep learning (DL), this meta-analysis aimed to evaluate the performance of these models in predicting the WHO meningioma grade using imaging data. A systematic search was performed in PubMed/MEDLINE, Embase, and the Cochrane Library for studies published up to April 1, 2024, and reporting the performance metrics of the ML models in predicting of WHO meningioma grade using imaging studies. Pooled area under the receiver operating characteristics curve (AUROC), specificity, and sensitivity were estimated. Subgroup and meta-regression analyses were performed based on a number of potential influencing variables. A total of 32 studies with 15,365 patients were included in the present study. The overall pooled sensitivity, specificity, and AUROC of ML methods for prediction of tumor grade in meningioma were 85% (95% CI, 79-89%), 87% (95% CI, 81-91%), and 93% (95% CI, 90-95%), respectively. Both the type of validation and study cohort (training or test) were significantly associated with model performance. However, no significant association was found between the sample size or the type of ML method and model performance. The ML predictive models show a high overall performance in predicting the WHO meningioma grade using imaging data. Further studies on the performance of DL algorithms in larger datasets using external validation are needed.

Validity and Utility of a Risk Prediction Model for Wound Infection After Lower Third Molar Surgery

OBJECTIVES

To externally validate a clinical prediction model for surgical site infection (SSI) after lower third molar (L3M) surgery and evaluate its clinical usefulness.

METHODS

We conducted a retrospective cohort study of patients who underwent L3M surgery at Hokkaido University Hospital. The study was designed to evaluate the historical and methodological transportability. Clinical usefulness was evaluated using decision curve analysis on the data of the non-antibiotic-treated patients.

RESULTS

We obtained 2543 validation cohorts from April 2020 to March 2023, and 640 non-antibiotic cohorts from July 2010 to September 2023. The incidences of SSI after L3M surgery were 5.3% (135/2543) and 7.7% (49/640) in the validation and non-antibiotic cohorts, respectively. The discrimination ability of the prediction model was acceptable for the external validation cohort (c-statistic: 0.67; 95% CI: 0.62-0.71) and adequate for the non-antibiotic cohort (c-statistic: 0.72; 95% CI: 0.63-0.79). In both cohorts, the model showed excellent calibration between the observed and predicted probabilities. Decision curve analysis showed increased net benefit across a range of meaningful risk thresholds.

CONCLUSION

A simple risk prediction model for SSI after L3M surgery demonstrated clinical transportability and usefulness. This model may help surgeons/clinicians determine the appropriateness of prophylactic antibiotics administration for patients in L3M surgery.

Development, validation and recalibration of a prediction model for prediabetes: an EHR and NHANES-based study

BACKGROUND

A prediction model that estimates the risk of elevated glycated hemoglobin (HbA1c) was developed from electronic health record (EHR) data to identify adult patients at risk for prediabetes who may otherwise go undetected. We aimed to assess the internal performance of a new penalized regression model using the same EHR data and compare it to the previously developed stepdown approximation for predicting HbA1c ≥ 5.7%, the cut-off for prediabetes. Additionally, we sought to externally validate and recalibrate the approximation model using 2017-2020 pre-pandemic National Health and Nutrition Examination Survey (NHANES) data.

METHODS

We developed logistic regression models using EHR data through two approaches: the Least Absolute Shrinkage and Selection Operator (LASSO) and stepdown approximation. Internal validation was performed using the bootstrap method, with internal performance evaluated by the Brier score, C-statistic, calibration intercept and slope, and the integrated calibration index. We externally validated the approximation model by applying original model coefficients to NHANES, and we examined the approximation model's performance after recalibration in NHANES.

RESULTS

The EHR cohort included 22,635 patients, with 26% identified as having prediabetes. Both the LASSO and approximation models demonstrated similar discrimination in the EHR cohort, with optimism-corrected C-statistics of 0.760 and 0.763, respectively. The LASSO model included 23 predictor variables, while the approximation model contained 8. Among the 2,348 NHANES participants who met the inclusion criteria, 30.1% had prediabetes. External validation of the LASSO model was not possible due to the unavailability of some predictor variables. The approximation model discriminated well in the NHANES dataset, achieving a C-statistic of 0.787.

CONCLUSION

The approximation method demonstrated comparable performance to LASSO in the EHR development cohort, making it a viable option for healthcare organizations with limited resources to collect a comprehensive set of candidate predictor variables. NHANES data may be suitable for externally validating a clinical prediction model developed with EHR data to assess generalizability to a nationally representative sample, depending on the model's intended use and the alignment of predictor variable definitions with those used in the model's original development.

Development and External Validation of a Home-based Risk Prediction Model of Natural Onset of Menopause-Teuta

OBJECTIVE

To develop and externally validate a 10-year risk prediction model of natural onset of menopause using ready-to-use predictors.

DESIGN

Population-based prospective cohort study.

PARTICIPANTS

Community-dwelling, premenopausal women aged 28 years and older enrolled in the Swiss (CoLaus) and Dutch (PREVEND) study.

MAIN OUTCOME MEASURE

Incidence of self-reported natural menopause.

MODEL DEVELOPMENT

Based on existing literature, 11 predictors were tested in this study. The CoLaus cohort was used to develop the model by applying the backward-elimination approach and Bayesian Model Averaging. Internal validation was performed by bootstrapping. External validation was performed using data from the PREVEND cohort and recalibrating the baseline survival estimate. C-statistics, calibration slopes, and expected/observed probabilities were calculated as measures of model internal and/or external performances.

RESULTS

The final analysis included 750 and 1032 premenopausal women from the CoLaus and the PREVEND cohorts, respectively. Among them, 445 (59%) from CoLaus and 387 (38%) from PREVEND experienced menopause over a median follow-up of 10.7 and 9 years, respectively. The final model included age, alcohol consumption, smoking status, education level, and systolic blood pressure. Upon external calibration in the PREVEND cohort, the model exhibited good discrimination, with a C-statistic of 0.888 and an expected/observed probability of 0.82.

CONCLUSION

We present the first internally and externally validated prediction model of natural menopause onset using readily available predictors. Validation of our model to other populations is needed.

Clinical predictive models for recurrence and survival in treated laryngeal and hypopharyngeal cancer: a systematic review and meta-analysis

Background

The limitations of the traditional TNM system have spurred interest in multivariable models for personalized prognostication in laryngeal and hypopharyngeal cancers (LSCC/HPSCC). However, the performance of these models depends on the quality of data and modelling methodology, affecting their potential for clinical adoption. This systematic review and meta-analysis (SR-MA) evaluated clinical predictive models (CPMs) for recurrence and survival in treated LSCC/HPSCC. We assessed models' characteristics and methodologies, as well as performance, risk of bias (RoB), and applicability.

Methods

Literature searches were conducted in MEDLINE (OVID), Embase (OVID) and IEEE databases from January 2005 to November 2023. The search algorithm used comprehensive text word and index term combinations without language or publication type restrictions. Independent reviewers screened titles and abstracts using a predefined Population, Index, Comparator, Outcomes, Timing and Setting (PICOTS) framework. We included externally validated (EV) multivariable models, with at least one clinical predictor, that provided recurrence or survival predictions. The SR-MA followed PRISMA reporting guidelines, and PROBAST framework for RoB assessment. Model discrimination was assessed using C-index/AUC, and was presented for all models using forest plots. MA was only performed for models that were externally validated in two or more cohorts, using random-effects model. The main outcomes were model discrimination and calibration measures for survival (OS) and/or local recurrence (LR) prediction. All measures and assessments were preplanned prior to data collection.

Results

The SR-MA identified 11 models, reported in 16 studies. Seven models for OS showed good discrimination on development, with only one excelling (C-index >0.9), and three had weak or poor discrimination. Inclusion of a radiomics score as a model parameter achieved relatively better performance. Most models had poor generalisability, demonstrated by worse discrimination performance on EV, but they still outperformed the TNM system. Only two models met the criteria for MA, with pooled EV AUCs 0.73 (95% CI 0.71-0.76) and 0.67 (95% CI 0.6-0.74). RoB was high for all models, particularly in the analysis domain.

Conclusions

This review highlighted the shortcomings of currently available models, while emphasizing the need for rigorous independent evaluations. Despite the proliferation of models, most exhibited methodological limitations and bias. Currently, no models can confidently be recommended for routine clinical use.

Systematic review registration

https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021248762, identifier CRD42021248762.

Externally validated clinical prediction models for estimating treatment outcomes for patients with a mood, anxiety or psychotic disorder: systematic review and meta-analysis

BACKGROUND

Suboptimal treatment outcomes contribute to the high disease burden of mood, anxiety or psychotic disorders. Clinical prediction models could optimise treatment allocation, which may result in better outcomes. Whereas ample research on prediction models is performed, model performance in other clinical contexts (i.e. external validation) is rarely examined. This gap hampers generalisability and as such implementation in clinical practice.

AIMS

Systematically appraise studies on externally validated clinical prediction models for estimated treatment outcomes for mood, anxiety and psychotic disorders by (1) reviewing methodological quality and applicability of studies and (2) investigating how model properties relate to differences in model performance.

METHOD

The review and meta-analysis protocol was prospectively registered with PROSPERO (registration number CRD42022307987). A search was conducted on 8 November 2021 in the databases PubMED, PsycINFO and EMBASE. Random-effects meta-analysis and meta-regression were conducted to examine between-study heterogeneity in discriminative performance and its relevant influencing factors.

RESULTS

Twenty-eight studies were included. The majority of studies (n = 16) validated models for mood disorders. Clinical predictors (e.g. symptom severity) were most frequently included (n = 25). Low methodological and applicability concerns were found for two studies. The overall discrimination performance of the meta-analysis was fair with wide prediction intervals (0.72 [0.46; 0.89]). The between-study heterogeneity was not explained by number or type of predictors but by disorder diagnosis.

CONCLUSIONS

Few models seem ready for further implementation in clinical practice to aid treatment allocation. Besides the need for more external validation studies, we recommend close examination of the clinical setting before model implementation.

A beginner's guide to supervised analysis for mass cytometry data in cancer biology

Mass cytometry enables deep profiling of biological samples at single-cell resolution. This technology is more than relevant in cancer research due to high cellular heterogeneity and complexity. Downstream analysis of high-dimensional datasets increasingly relies on machine learning (ML) to extract clinically relevant information, including supervised algorithms for classification and regression purposes. In cancer research, they are used to develop predictive models that will guide clinical decision making. However, the development of supervised algorithms faces major challenges, such as sufficient validation, before being translated into the clinics. In this work, we provide a framework for the analysis of mass cytometry data with a specific focus on supervised algorithms and practical examples of their applications. We also raise awareness on key issues regarding good practices for researchers curious to implement supervised ML on their mass cytometry data. Finally, we discuss the challenges of supervised ML application to cancer research.

Adversarially trained RTMpose: A high-performance, non-contact method for detecting Genu valgum in adolescents

Genu valgum (GV), a prevalent postural deformity in adolescents, is traditionally diagnosed using methods that are complex, costly, and accompanied by radiation risks. To address these challenges, we evaluated 1519 Chinese adolescents, collecting GV annotations from three medical professionals to establish a robust dataset. Leveraging these annotations, we developed an end-to-end GV prediction model using RTMpose for body landmark extraction from images. However, a key challenge was the inaccuracy of landmarks, which adversely affects downstream tasks. To mitigate this, we harnessed the parallels between pose estimation biases and adversarial perturbations, implementing adversarial training to bolster model robustness against noisy landmark data. Our model achieved a significant improvement, with an accuracy of 75%, compared to the baseline's 64.25%. These results underscore the model's efficacy as a high-performance, non-contact GV detection method and demonstrate the effectiveness of adversarial training in enhancing landmark-related tasks, providing a safer, cost-effective alternative for adolescent GV diagnosis.

Clinical prediction models for the management of blunt chest trauma in the emergency department: a systematic review

BACKGROUND

The aim of this systematic review was to investigate how clinical prediction models compare in terms of their methodological development, validation, and predictive capabilities, for patients with blunt chest trauma presenting to the Emergency Department.

METHODS

A systematic review was conducted across databases from 1st Jan 2000 until 1st April 2024. Studies were categorised into three types of multivariable prediction research and data extracted regarding methodological issues and the predictive capabilities of each model. Risk of bias and applicability were assessed.

RESULTS

41 studies were included that discussed 22 different models. The most commonly observed study design was a single-centre, retrospective, chart review. The most widely externally validated clinical prediction models with moderate to good discrimination were the Thoracic Trauma Severity Score and the STUMBL Score.

DISCUSSION

This review demonstrates that the predictive ability of some of the existing clinical prediction models is acceptable, but high risk of bias and lack of subsequent external validation limits the extensive application of the models. The Thoracic Trauma Severity Score and STUMBL Score demonstrate better predictive accuracy in both development and external validation studies than the other models, but require recalibration and / or update and evaluation of their clinical and cost effectiveness.

REVIEW REGISTRATION

PROSPERO database ( https://www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=351638 ).

Recommendations for Performance Evaluation of Machine Learning in Pathology: A Concept Paper From the College of American Pathologists

CONTEXT.—

Machine learning applications in the pathology clinical domain are emerging rapidly. As decision support systems continue to mature, laboratories will increasingly need guidance to evaluate their performance in clinical practice. Currently there are no formal guidelines to assist pathology laboratories in verification and/or validation of such systems. These recommendations are being proposed for the evaluation of machine learning systems in the clinical practice of pathology.

OBJECTIVE.—

To propose recommendations for performance evaluation of in vitro diagnostic tests on patient samples that incorporate machine learning as part of the preanalytical, analytical, or postanalytical phases of the laboratory workflow. Topics described include considerations for machine learning model evaluation including risk assessment, predeployment requirements, data sourcing and curation, verification and validation, change control management, human-computer interaction, practitioner training, and competency evaluation.

DATA SOURCES.—

An expert panel performed a review of the literature, Clinical and Laboratory Standards Institute guidance, and laboratory and government regulatory frameworks.

CONCLUSIONS.—

Review of the literature and existing documents enabled the development of proposed recommendations. This white paper pertains to performance evaluation of machine learning systems intended to be implemented for clinical patient testing. Further studies with real-world clinical data are encouraged to support these proposed recommendations. Performance evaluation of machine learning models is critical to verification and/or validation of in vitro diagnostic tests using machine learning intended for clinical practice.

How to develop, validate, and update clinical prediction models using multinomial logistic regression

OBJECTIVES

Multicategory prediction models (MPMs) can be used in health care when the primary outcome of interest has more than two categories. The application of MPMs is scarce, possibly due to added methodological complexities compared to binary outcome models. We provide a guide of how to develop, validate, and update clinical prediction models based on multinomial logistic regression.

STUDY DESIGN AND SETTING

We present guidance and recommendations based on recent methodological literature, illustrated by a previously developed and validated MPM for treatment outcomes in rheumatoid arthritis. Prediction models using multinomial logistic regression can be developed for nominal outcomes, but also for ordinal outcomes. This article is intended to supplement existing general guidance on prediction model research.

RESULTS

This guide is split into three parts: 1) outcome definition and variable selection, 2) model development, and 3) model evaluation (including performance assessment, internal and external validation, and model recalibration). We outline how to evaluate and interpret the predictive performance of MPMs. R code is provided.

CONCLUSION

We recommend the application of MPMs in clinical settings where the prediction of a multicategory outcome is of interest. Future methodological research could focus on MPM-specific considerations for variable selection and sample size criteria for external validation.

Data set terminology of deep learning in medicine: a historical review and recommendation

Medicine and deep learning-based artificial intelligence (AI) engineering represent two distinct fields each with decades of published history. The current rapid convergence of deep learning and medicine has led to significant advancements, yet it has also introduced ambiguity regarding data set terms common to both fields, potentially leading to miscommunication and methodological discrepancies. This narrative review aims to give historical context for these terms, accentuate the importance of clarity when these terms are used in medical deep learning contexts, and offer solutions to mitigate misunderstandings by readers from either field. Through an examination of historical documents, including articles, writing guidelines, and textbooks, this review traces the divergent evolution of terms for data sets and their impact. Initially, the discordant interpretations of the word 'validation' in medical and AI contexts are explored. We then show that in the medical field as well, terms traditionally used in the deep learning domain are becoming more common, with the data for creating models referred to as the 'training set', the data for tuning of parameters referred to as the 'validation (or tuning) set', and the data for the evaluation of models as the 'test set'. Additionally, the test sets used for model evaluation are classified into internal (random splitting, cross-validation, and leave-one-out) sets and external (temporal and geographic) sets. This review then identifies often misunderstood terms and proposes pragmatic solutions to mitigate terminological confusion in the field of deep learning in medicine. We support the accurate and standardized description of these data sets and the explicit definition of data set splitting terminologies in each publication. These are crucial methods for demonstrating the robustness and generalizability of deep learning applications in medicine. This review aspires to enhance the precision of communication, thereby fostering more effective and transparent research methodologies in this interdisciplinary field.

Development and Multicenter, Multiprotocol Validation of Neural Network for Aberrant Right Subclavian Artery Detection

PURPOSE

This study aimed to develop and validate a convolutional neural network (CNN) that automatically detects an aberrant right subclavian artery (ARSA) on preoperative computed tomography (CT) for thyroid cancer evaluation.

MATERIALS AND METHODS

A total of 556 CT with ARSA and 312 CT with normal aortic arch from one institution were used as the training set for model development. A deep learning model for the classification of patch images for ARSA was developed using two-dimension CNN from EfficientNet. The diagnostic performance of our model was evaluated using external test sets (112 and 126 CT) from two institutions. The performance of the model was compared with that of radiologists for detecting ARSA using an independent dataset of 1683 consecutive neck CT.

RESULTS

The performance of the model was achieved using two external datasets with an area under the curve of 0.97 and 0.99, and accuracy of 97% and 99%, respectively. In the temporal validation set, which included a total of 20 patients with ARSA and 1663 patients without ARSA, radiologists overlooked 13 ARSA cases. In contrast, the CNN model successfully detected all the 20 patients with ARSA.

CONCLUSION

We developed a CNN-based deep learning model that detects ARSA using CT. Our model showed high performance in the multicenter validation.

Development of thresholds and a visualization tool for use of a blood test in routine clinical dementia practice

INTRODUCTION

We developed a multimarker blood test result interpretation tool for the clinical dementia practice, including phosphorylated (P-)tau181, amyloid-beta (Abeta)42/40, glial fibrillary acidic protein (GFAP), and neurofilament light (NfL).

METHODS

We measured the plasma biomarkers with Simoa (n = 1199), applied LASSO regression for biomarker selection and receiver operating characteristics (ROC) analyses to determine diagnostic accuracy. We validated our findings in two independent cohorts and constructed a visualization approach.

RESULTS

P-tau181, GFAP, and NfL were selected. This combination had area under the curve (AUC) = 83% to identify amyloid positivity in pre-dementia stages, AUC = 87%-89% to differentiate Alzheimer's or controls from frontotemporal dementia, AUC = 74%-76% to differentiate Alzheimer's or controls from dementia with Lewy bodies. Highly reproducible AUCs were obtained in independent cohorts. The resulting visualization tool includes UpSet plots to visualize the stand-alone biomarker results and density plots to visualize the biomarker results combined.

DISCUSSION

Our multimarker blood test interpretation tool is ready for testing in real-world clinical dementia settings.

HIGHLIGHTS

We developed a multimarker blood test interpretation tool for clinical dementia practice. Our interpretation tool includes plasma biomarkers P-tau, GFAP, and NfL. Our tool is particularly useful for Alzheimer's and frontotemporal dementia diagnosis.

External validation of a novel nomogram for diagnosis of Protein Energy Wasting in adult hemodialysis patients

Background

Protein Energy Wasting (PEW) has high incidence in adult hemodialysis patients and refers to a state of decreased protein and energy substance. It has been demonstrated that PEW highly affects the quality of survival and increases the risk of death. Nevertheless, its diagnostic criteria are complex in clinic. To simplify the diagnosis method of PEW in adult hemodialysis patients, we previously established a novel clinical prediction model that was well-validated internally using bootstrapping. In this multicenter cross-sectional study, we aimed to externally validate this nomogram in a new cohort of adult hemodialysis patients.

Methods

The novel prediction model was built by combining four independent variables with part of the International Society of Renal Nutrition and Metabolism (ISRNM) diagnostic criteria including albumin, total cholesterol, and body mass index (BMI). We evaluated the performance of the new model using discrimination (Concordance Index), calibration plots, and Clinical Impact Curve to assess its predictive utility.

Results

From September 1st, 2022 to August 31st, 2023, 1,158 patients were screened in five medical centers in Shanghai. 622 (53.7%) hemodialysis patients were included for analysis. The PEW predictive model was acceptable discrimination with the area under the curve of 0.777 (95% CI 0.741-0.814). Additionally, the model revealed well-fitted calibration curves. The McNemar test showed the novel model had similar diagnostic efficacy with the gold standard diagnostic method (p > 0.05).

Conclusion

Our results from this cross-sectional external validation study further demonstrate that the novel model is a valid tool to identify PEW in adult hemodialysis patients effectively.

Pitfalls of single-study external validation illustrated with a model predicting functional outcome after aneurysmal subarachnoid hemorrhage

BACKGROUND

Prediction models are often externally validated with data from a single study or cohort. However, the interpretation of performance estimates obtained with single-study external validation is not as straightforward as assumed. We aimed to illustrate this by conducting a large number of external validations of a prediction model for functional outcome in subarachnoid hemorrhage (SAH) patients.

METHODS

We used data from the Subarachnoid Hemorrhage International Trialists (SAHIT) data repository (n = 11,931, 14 studies) to refit the SAHIT model for predicting a dichotomous functional outcome (favorable versus unfavorable), with the (extended) Glasgow Outcome Scale or modified Rankin Scale score, at a minimum of three months after discharge. We performed leave-one-cluster-out cross-validation to mimic the process of multiple single-study external validations. Each study represented one cluster. In each of these validations, we assessed discrimination with Harrell's c-statistic and calibration with calibration plots, the intercepts, and the slopes. We used random effects meta-analysis to obtain the (reference) mean performance estimates and between-study heterogeneity (I2-statistic). The influence of case-mix variation on discriminative performance was assessed with the model-based c-statistic and we fitted a "membership model" to obtain a gross estimate of transportability.

RESULTS

Across 14 single-study external validations, model performance was highly variable. The mean c-statistic was 0.74 (95%CI 0.70-0.78, range 0.52-0.84, I2 = 0.92), the mean intercept was -0.06 (95%CI -0.37-0.24, range -1.40-0.75, I2 = 0.97), and the mean slope was 0.96 (95%CI 0.78-1.13, range 0.53-1.31, I2 = 0.90). The decrease in discriminative performance was attributable to case-mix variation, between-study heterogeneity, or a combination of both. Incidentally, we observed poor generalizability or transportability of the model.

CONCLUSIONS

We demonstrate two potential pitfalls in the interpretation of model performance with single-study external validation. With single-study external validation. (1) model performance is highly variable and depends on the choice of validation data and (2) no insight is provided into generalizability or transportability of the model that is needed to guide local implementation. As such, a single single-study external validation can easily be misinterpreted and lead to a false appreciation of the clinical prediction model. Cross-validation is better equipped to address these pitfalls.

Diagnostic prediction models for bacterial meningitis in children with a suspected central nervous system infection: a systematic review and prospective validation study

OBJECTIVES

Diagnostic prediction models exist to assess the probability of bacterial meningitis (BM) in paediatric patients with suspected meningitis. To evaluate the diagnostic accuracy of these models in a broad population of children suspected of a central nervous system (CNS) infection, we performed external validation.

METHODS

We performed a systematic literature review in Medline to identify articles on the development, refinement or validation of a prediction model for BM, and validated these models in a prospective cohort of children aged 0-18 years old suspected of a CNS infection.

PRIMARY AND SECONDARY OUTCOME MEASURES

We calculated sensitivity, specificity, predictive values, the area under the receiver operating characteristic curve (AUC) and evaluated calibration of the models for diagnosis of BM.

RESULTS

In total, 23 prediction models were validated in a cohort of 450 patients suspected of a CNS infection included between 2012 and 2015. In 75 patients (17%), the final diagnosis was a CNS infection including 30 with BM (7%). AUCs ranged from 0.69 to 0.94 (median 0.83, interquartile range [IQR] 0.79-0.87) overall, from 0.74 to 0.96 (median 0.89, IQR 0.82-0.92) in children aged ≥28 days and from 0.58 to 0.91 (median 0.79, IQR 0.75-0.82) in neonates.

CONCLUSIONS

Prediction models show good to excellent test characteristics for excluding BM in children and can be of help in the diagnostic workup of paediatric patients with a suspected CNS infection, but cannot replace a thorough history, physical examination and ancillary testing.

External validation of the computer aided risk scoring system in predicting in-hospital mortality following emergency medical admissions

BACKGROUND

Clinical prediction models have the potential to improve the quality of care and enhance patient safety outcomes. A Computer-aided Risk Scoring system (CARSS) was previously developed to predict in-hospital mortality following emergency admissions based on routinely collected blood tests and vitals. We aimed to externally validate the CARSS model.

METHODS

In this retrospective external validation study, we considered all adult (≥18 years) emergency medical admissions discharged between 11/11/2020 and 11/11/2022 from The Rotherham Foundation Trust (TRFT), UK. We assessed the predictive performance of the CARSS model based on its discriminative (c-statistic) and calibration characteristics (calibration slope and calibration plots).

RESULTS

Out of 32,774 admissions, 20,422 (62.3 %) admissions were included. The TRFT sample had similar demographic characteristics to the development sample but had higher mortality (6.1 % versus 5.7 %). The CARSS model demonstrated good discrimination (c-statistic 0.87 [95 % CI 0.86-0.88]) and good calibration to the TRFT dataset (slope = 1.03 [95 % CI 0.98-1.08] intercept = 0 [95 % CI -0.06-0.07]) after re-calibrating for differences in baseline mortality (intercept = 0.96 [95 % CI 0.90-1.03] before re-calibration).

CONCLUSION

In summary, the CARSS model is externally validated after correcting the baseline risk of death between development and validation datasets. External validation of the CARSS model showed that it under-predicted in-hospital mortality. Re-calibration of this model showed adequate performance in the TRFT dataset.

Prognostic value of total tumor volume in patients with colorectal liver metastases: A secondary analysis of the randomized CAIRO5 trial with external cohort validation

BACKGROUND

This study aimed to assess the prognostic value of total tumor volume (TTV) for early recurrence (within 6 months) and overall survival (OS) in patients with colorectal liver metastases (CRLM), treated with induction systemic therapy followed by complete local treatment.

METHODS

Patients with initially unresectable CRLM from the multicenter randomized phase 3 CAIRO5 trial (NCT02162563) who received induction systemic therapy followed by local treatment were included. Baseline TTV and change in TTV as response to systemic therapy were calculated using the CT scan before and the first after systemic treatment, and were assessed for their added prognostic value. The findings were validated in an external cohort of patients treated at a tertiary center.

RESULTS

In total, 215 CAIRO5 patients were included. Baseline TTV and absolute change in TTV were significantly associated with early recurrence (P = 0.005 and P = 0.040, respectively) and OS in multivariable analyses (P = 0.024 and P = 0.006, respectively), whereas RECIST1.1 was not prognostic for early recurrence (P = 0.88) and OS (P = 0.35). In the validation cohort (n = 85), baseline TTV and absolute change in TTV remained prognostic for early recurrence (P = 0.041 and P = 0.021, respectively) and OS in multivariable analyses (P < 0.0001 and P = 0.012, respectively), and showed added prognostic value over conventional clinicopathological variables (increase C-statistic, 0.06; 95 % CI, 0.02 to 0.14; P = 0.008).

CONCLUSION

Total tumor volume is strongly prognostic for early recurrence and OS in patients who underwent complete local treatment of initially unresectable CRLM, both in the CAIRO5 trial and the validation cohort. In contrast, RECIST1.1 did not show prognostic value for neither early recurrence nor OS.

International External Validation of Risk Prediction Model of 90-Day Mortality after Gastrectomy for Cancer Using Machine Learning

BACKGROUND

Radical gastrectomy remains the main treatment for gastric cancer, despite its high mortality. A clinical predictive model of 90-day mortality (90DM) risk after gastric cancer surgery based on the Spanish EURECCA registry database was developed using a matching learning algorithm. We performed an external validation of this model based on data from an international multicenter cohort of patients.

METHODS

A cohort of patients from the European GASTRODATA database was selected. Demographic, clinical, and treatment variables in the original and validation cohorts were compared. The performance of the model was evaluated using the area under the curve (AUC) for a random forest model.

RESULTS

The validation cohort included 2546 patients from 24 European hospitals. The advanced clinical T- and N-category, neoadjuvant therapy, open procedures, total gastrectomy rates, and mean volume of the centers were significantly higher in the validation cohort. The 90DM rate was also higher in the validation cohort (5.6%) vs. the original cohort (3.7%). The AUC in the validation model was 0.716.

CONCLUSION

The externally validated model for predicting the 90DM risk in gastric cancer patients undergoing gastrectomy with curative intent continues to be as useful as the original model in clinical practice.

Machine learning-assisted prediction of trabeculectomy outcomes among patients of juvenile glaucoma by using 5-year follow-up data

OBJECTIVE

To develop machine learning (ML) models, using pre and intraoperative surgical parameters, for predicting trabeculectomy outcomes in the eyes of patients with juvenile-onset primary open-angle glaucoma (JOAG) undergoing primary surgery.

SUBJECTS

The study included 207 JOAG patients from a single center who met the following criteria: diagnosed between 10 and 40 years of age, with an IOP of >22 mmHg in the eyes on two or more occasions, open angle on gonioscopy in both eyes, with glaucomatous optic neuropathy, and requiring a trabeculectomy for IOP control. Only the patients with a minimum 5-year follow-up after surgery were included in the study.

METHODS

A successful surgical outcome was defined as IOP ≤18 mmHg (criterion A) or 50% reduction in IOP from baseline (criterion B) 5 years after trabeculectomy. Feature selection techniques were used to select the most important contributory parameters, and tenfold cross-validation was used to evaluate model performance. The ML models were evaluated, compared, and prioritized based on their accuracy, sensitivity, specificity, Matthew correlation coefficient (MCC) index, and mean area under the receiver operating characteristic curve (AUROC). The prioritized models were further optimized by tuning the hyperparameters, and feature contributions were evaluated. In addition, an unbiased relationship analysis among the parameters was performed for clinical utility.

RESULTS

Age at diagnosis, preoperative baseline IOP, duration of preoperative medical treatment, Tenon's thickness, scleral fistulation technique, and intraoperative mitomycin C (MMC) use, were identified as the main contributing parameters for developing efficient models. The three models developed for a consensus-based outcome to predict trabeculectomy success showed an accuracy of >86%, sensitivity of >90%, and specificity of >74%, using tenfold cross-validation. The use of intraoperative MMC and a punch for scleral fistulation compared to the traditional excision with scissors were significantly associated with long-term success of trabeculectomy.

CONCLUSION

Optimizing surgical parameters by using these ML models might reduce surgical failures associated with trabeculectomy and provide more realistic expectations regarding surgical outcomes in young patients.

NIHR Global Health Research Unit on Global Surgery, STARSurg Collaborative. A prognostic model for use before elective surgery to estimate the risk of postoperative pulmonary complications (GSU-Pulmonary Score): a development and validation study in three international cohorts

BACKGROUND

Pulmonary complications are the most common cause of death after surgery. This study aimed to derive and externally validate a novel prognostic model that can be used before elective surgery to estimate the risk of postoperative pulmonary complications and to support resource allocation and prioritisation during pandemic recovery.

METHODS

Data from an international, prospective cohort study were used to develop a novel prognostic risk model for pulmonary complications after elective surgery in adult patients (aged ≥18 years) across all operation and disease types. The primary outcome measure was postoperative pulmonary complications at 30 days after surgery, which was a composite of pneumonia, acute respiratory distress syndrome, and unexpected mechanical ventilation. Model development with candidate predictor variables was done in the GlobalSurg-CovidSurg Week dataset (global; October, 2020). Two structured machine learning techniques were explored (XGBoost and the least absolute shrinkage and selection operator [LASSO]), and the model with the best performance (GSU-Pulmonary Score) underwent internal validation using bootstrap resampling. The discrimination and calibration of the score were externally validated in two further prospective cohorts: CovidSurg-Cancer (worldwide; February to August, 2020, during the COVID-19 pandemic) and RECON (UK and Australasia; January to October, 2019, before the COVID-19 pandemic). The model was deployed as an online web application. The GlobalSurg-CovidSurg Week and CovidSurg-Cancer studies were registered with ClinicalTrials.gov, NCT04509986 and NCT04384926.

FINDINGS

Prognostic models were developed from 13 candidate predictor variables in data from 86 231 patients (1158 hospitals in 114 countries). External validation included 30 492 patients from CovidSurg-Cancer (726 hospitals in 75 countries) and 6789 from RECON (150 hospitals in three countries). The overall rates of pulmonary complications were 2·0% in derivation data, and 3·9% (CovidSurg-Cancer) and 4·7% (RECON) in the validation datasets. Penalised regression using LASSO had similar discrimination to XGBoost (area under the receiver operating curve [AUROC] 0·786, 95% CI 0·774-0·798 vs 0·785, 0·772-0·797), was more explainable, and required fewer covariables. The final GSU-Pulmonary Score included ten predictor variables and showed good discrimination and calibration upon internal validation (AUROC 0·773, 95% CI 0·751-0·795; Brier score 0·020, calibration in the large [CITL] 0·034, slope 0·954). The model performance was acceptable on external validation in CovidSurg-Cancer (AUROC 0·746, 95% CI 0·733-0·760; Brier score 0·036, CITL 0·109, slope 1·056), but with some miscalibration in RECON data (AUROC 0·716, 95% CI 0·689-0·744; Brier score 0·045, CITL 1·040, slope 1·009).

INTERPRETATION

This novel prognostic risk score uses simple predictor variables available at the time of a decision for elective surgery that can accurately stratify patients' risk of postoperative pulmonary complications, including during SARS-CoV-2 outbreaks. It could inform surgical consent, resource allocation, and hospital-level prioritisation as elective surgery is upscaled to address global backlogs.

FUNDING

National Institute for Health Research.

A constrained maximum likelihood approach to developing well-calibrated models for predicting binary outcomes

The added value of candidate predictors for risk modeling is routinely evaluated by comparing the performance of models with or without including candidate predictors. Such comparison is most meaningful when the estimated risk by the two models are both unbiased in the target population. Very often data for candidate predictors are sourced from nonrepresentative convenience samples. Updating the base model using the study data without acknowledging the discrepancy between the underlying distribution of the study data and that in the target population can lead to biased risk estimates and therefore an unfair evaluation of candidate predictors. To address this issue assuming access to a well-calibrated base model, we propose a semiparametric method for model fitting that enforces good calibration. The central idea is to calibrate the fitted model against the base model by enforcing suitable constraints in maximizing the likelihood function. This approach enables unbiased assessment of model improvement offered by candidate predictors without requiring a representative sample from the target population, thus overcoming a significant practical challenge. We study theoretical properties for model parameter estimates, and demonstrate improvement in model calibration via extensive simulation studies. Finally, we apply the proposed method to data extracted from Penn Medicine Biobank to inform the added value of breast density for breast cancer risk assessment in the Caucasian woman population.

Framework for personalized prediction of treatment response in relapsing-remitting multiple sclerosis: a replication study in independent data

BACKGROUND

Individualizing and optimizing treatment of relapsing-remitting multiple sclerosis patients is a challenging problem, which would benefit from a clinically valid decision support. Stühler et al. presented black box models for this aim which were developed and internally evaluated in a German registry but lacked external validation.

METHODS

In patients from the French OFSEP registry, we independently built and validated models predicting being free of relapse and free of confirmed disability progression (CDP), following the methodological roadmap and predictors reported by Stühler. Hierarchical Bayesian models were fit to predict the outcomes under 6 disease-modifying treatments given the individual disease course up to the moment of treatment change. Data was temporally split on 2017, and models were developed in patients treated earlier (n = 5517). Calibration curves, discrimination, mean squared error (MSE) and relative percentage of root MSE (RMSE%) were assessed by external validation of models in more-recent patients (n = 3768). Non-Bayesian fixed-effects GLMs were also applied and their outcomes were compared to these of the Bayesian ones. For both, we modelled the number of on-therapy relapses with a negative binomial distribution, and CDP occurrence with a binomial distribution.

RESULTS

The performance of our temporally-validated relapse model (MSE: 0.326, C-Index: 0.639) is potentially superior to that of Stühler's (MSE: 0.784, C-index: 0.608). Calibration plots revealed miscalibration. Our CDP model (MSE: 0.072, C-Index: 0.777) was also better than its counterpart (MSE: 0.131, C-index: 0.554). Results from non-Bayesian fixed-effects GLM models were similar to the Bayesian ones.

CONCLUSIONS

The relapse and CDP models rebuilt and externally validated in independent data could compare and strengthen the credibility of the Stühler models. Their model-building strategy was replicable.

Risk calculators for the detection of prostate cancer: a systematic review

BACKGROUND

Prostate cancer (PCa) (early) detection poses significant challenges, including unnecessary testing and the risk of potential overdiagnosis. The European Association of Urology therefore suggests an individual risk-adapted approach, incorporating risk calculators (RCs) into the PCa detection pathway. In the context of 'The PRostate Cancer Awareness and Initiative for Screening in the European Union' (PRAISE-U) project ( https://uroweb.org/praise-u ), we aim to provide an overview of the currently available clinical RCs applicable in an early PCa detection algorithm.

METHODS

We performed a systematic review to identify RCs predicting detection of clinically significant PCa at biopsy. A search was performed in the databases Medline ALL, Embase, Web of Science Core Collection, Cochrane Central Register of Controlled Trials and Google Scholar for publications between January 2010 and July 2023. We retrieved relevant literature by using the terms "prostate cancer", "screening/diagnosis" and "predictive model". Inclusion criteria included systematic reviews, meta-analyses, and clinical trials. Exclusion criteria applied to studies involving pre-targeted high-risk populations, diagnosed PCa patients, or a sample sizes under 50 men.

RESULTS

We identified 6474 articles, of which 140 were included after screening abstracts and full texts. In total, we identified 96 unique RCs. Among these, 45 underwent external validation, with 28 validated in multiple cohorts. Of the externally validated RCs, 17 are based on clinical factors, 19 incorporate clinical factors along with MRI details, 4 were based on blood biomarkers alone or in combination with clinical factors, and 5 included urinary biomarkers. The median AUC of externally validated RCs ranged from 0.63 to 0.93.

CONCLUSIONS

This systematic review offers an extensive analysis of currently available RCs, their variable utilization, and performance within validation cohorts. RCs have consistently demonstrated their capacity to mitigate the limitations associated with early detection and have been integrated into modern practice and screening trials. Nevertheless, the lack of external validation data raises concerns about numerous RCs, and it is crucial to factor in this omission when evaluating whether a specific RC is applicable to one's target population.

Multicentre validation of a machine learning model for predicting respiratory failure after noncardiac surgery

BACKGROUND

Postoperative respiratory failure is a serious complication that could benefit from early accurate identification of high-risk patients. We developed and validated a machine learning model to predict postoperative respiratory failure, defined as prolonged (>48 h) mechanical ventilation or reintubation after surgery.

METHODS

Easily extractable electronic health record (EHR) variables that do not require subjective assessment by clinicians were used. From EHR data of 307,333 noncardiac surgical cases, the model, trained with a gradient boosting algorithm, utilised a derivation cohort of 99,025 cases from Seoul National University Hospital (2013-9). External validation was performed using three separate cohorts A-C from different hospitals comprising 208,308 cases. Model performance was assessed by area under the receiver operating characteristic (AUROC) curve and area under the precision-recall curve (AUPRC), a measure of sensitivity and precision at different thresholds.

RESULTS

The model included eight variables: serum albumin, age, duration of anaesthesia, serum glucose, prothrombin time, serum creatinine, white blood cell count, and body mass index. Internally, the model achieved an AUROC of 0.912 (95% confidence interval [CI], 0.908-0.915) and AUPRC of 0.113. In external validation cohorts A, B, and C, the model achieved AUROCs of 0.879 (95% CI, 0.876-0.882), 0.872 (95% CI, 0.870-0.874), and 0.931 (95% CI, 0.925-0.936), and AUPRCs of 0.029, 0.083, and 0.124, respectively.

CONCLUSIONS

Utilising just eight easily extractable variables, this machine learning model demonstrated excellent discrimination in both internal and external validation for predicting postoperative respiratory failure. The model enables personalised risk stratification and facilitates data-driven clinical decision-making.

External validation of the EFFECT mortality prediction scale in patients admitted for acute heart failure in Araba, Spain

AIM

To validate the EFFECT (Enhanced Feedback for Effective Cardiac Treatment) scales, which predict mortality at 1 month and 1 year after admission, in a defined cohort of patients admitted to the Araba University Hospital (HUA) with a diagnosis of acutely decompensated heart failure.

METHOD

External validation study of a predictive model, in a retrospective cohort of patients admitted between October 1, 2020 and September 30, 2021.

RESULTS

A total of 550 patients were included. The two scales demonstrated good overall discriminatory ability in our series, with an area under ROC (0.755 y 0.756) and values in Brier score (0.094 y 0.194) similar to the original series. Calibration was assessed using the Hosmer-Lemeshow test and calibration plots and was also adequate. All this despite the fact that significant differences were observed in many clinical characteristics between our series and the original one.

CONCLUSIONS

The EFFECT scales showed good predictive ability and transportability. The one-month prediction scale was also useful for predicting mortality at one year. For both time periods, mortality was similar in the groups established in the original as low and very low risk.

Development and validation of a prediction score to assess the risk of incurring in COPD-related exacerbations: a population-based study in primary care

BACKGROUND

Chronic obstructive pulmonary disease (COPD) is the fourth most important cause of death in high-income countries. Inappropriate use of COPD inhaled therapy, including the low adherence (only 10 %-40 % of patients reporting an adequate compliance) may shrink or even nullify the proven benefits of these medications. As such, an accurate prediction algorithm to assess at national level the risk of COPD exacerbation might be relevant for general practictioners (GPs) to improve patient's therapy.

METHODS

We formed a cohort of patients aged 45 years or older being diagnosed with COPD in the period between January 2013 to December 2021. Each patient was followed until occurrence of COPD exacerbation up to the end of 2021. Sixteen determinants were adopted to assemble the CopdEX(CEX)-Health Search(HS)core, which was therefore developed and validated through the related two sub-cohorts.

RESULTS

We idenfied 63763 patients aged 45 years or older being diagnosed with COPD (mean age: 67.8 (SD:11.7); 57.7 % males).When the risk of COPD exacerbation was estimated via CEX-HScore, its predicted value was equal to 14.22 % over a 6-month event horizon. Discrimination accuracy and explained variation were equal to 66 % (95 % CI: 65-67 %) and 10 % (95 % CI: 9-11 %), respectively. The calibration slope did not significantly differ from the unit (p = 0.514).

CONCLUSIONS

The CEX-HScore was featured by fair accuracy for prediction of COPD-related exacerbations over a 6-month follow-up. Such a tool might therefore support GPs to enhance COPD patients' care, and improve their outcomes by facilitating personalized approaches through a score-based decision support system.

Internal and External Validation of Machine Learning Models for Predicting Acute Kidney Injury Following Non-Cardiac Surgery Using Open Datasets

This study developed and validated a machine learning model to accurately predict acute kidney injury (AKI) after non-cardiac surgery, aiming to improve patient outcomes by assessing its clinical feasibility and generalizability. We conducted a retrospective cohort study using data from 76,032 adults who underwent non-cardiac surgery at a single tertiary medical center between March 2019 and February 2021, and used data from 5512 patients from the VitalDB open dataset for external model validation. The predictive variables for model training consisted of demographic, preoperative laboratory, and intraoperative data, including calculated statistical values such as the minimum, maximum, and mean intraoperative blood pressure. When predicting postoperative AKI, our gradient boosting machine model incorporating all the variables achieved the best results, with AUROC values of 0.868 and 0.757 for the internal and external validations using the VitalDB dataset, respectively. The model using intraoperative data performed best in internal validation, while the model with preoperative data excelled in external validation. In this study, we developed a predictive model for postoperative AKI in adult patients undergoing non-cardiac surgery using preoperative and intraoperative data, and external validation demonstrated the efficacy of open datasets for generalization in medical artificial modeling research.

Externally validated prediction models for pre-eclampsia: systematic review and meta-analysis

OBJECTIVE

This systematic review and meta-analysis aimed to evaluate the performance of existing externally validated prediction models for pre-eclampsia (PE) (specifically, any-onset, early-onset, late-onset and preterm PE).

METHODS

A systematic search was conducted in five databases (MEDLINE, EMBASE, Emcare, CINAHL and Maternity & Infant Care Database) and using Google Scholar/reference search to identify studies based on the Population, Index prediction model, Comparator, Outcome, Timing and Setting (PICOTS) approach until 20 May 2023. We extracted data using the CHARMS checklist and appraised the risk of bias using the PROBAST tool. A meta-analysis of discrimination and calibration performance was conducted when appropriate.

RESULTS

Twenty-three studies reported 52 externally validated prediction models for PE (one preterm, 20 any-onset, 17 early-onset and 14 late-onset PE models). No model had the same set of predictors. Fifteen any-onset PE models were validated externally once, two were validated twice and three were validated three times, while the Fetal Medicine Foundation (FMF) competing-risks model for preterm PE prediction was validated widely in 16 different settings. The most common predictors were maternal characteristics (prepregnancy body mass index, prior PE, family history of PE, chronic medical conditions and ethnicity) and biomarkers (uterine artery pulsatility index and pregnancy-associated plasma protein-A). The FMF model for preterm PE (triple test plus maternal factors) had the best performance, with a pooled area under the receiver-operating-characteristics curve (AUC) of 0.90 (95% prediction interval (PI), 0.76-0.96), and was well calibrated. The other models generally had poor-to-good discrimination performance (median AUC, 0.66 (range, 0.53-0.77)) and were overfitted on external validation. Apart from the FMF model, only two models that were validated multiple times for any-onset PE prediction, which were based on maternal characteristics only, produced reasonable pooled AUCs of 0.71 (95% PI, 0.66-0.76) and 0.73 (95% PI, 0.55-0.86).

CONCLUSIONS

Existing externally validated prediction models for any-, early- and late-onset PE have limited discrimination and calibration performance, and include inconsistent input variables. The triple-test FMF model had outstanding discrimination performance in predicting preterm PE in numerous settings, but the inclusion of specialized biomarkers may limit feasibility and implementation outside of high-resource settings. © 2023 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Tutorial: dos and don'ts in clinical prediction research for venous thromboembolism

Clinical prediction modeling has become an increasingly popular domain of venous thromboembolism research in recent years. Prediction models can help healthcare providers make decisions regarding starting or withholding therapeutic interventions, or referrals for further diagnostic workup, and can form a basis for risk stratification in clinical trials. The aim of the current guide is to assist in the practical application of complicated methodological requirements for well-performed prediction research by presenting key dos and don'ts while expanding the understanding of predictive research in general for (clinical) researchers who are not specifically trained in the topic; throughout we will use prognostic venous thromboembolism scores as an exemplar.

Predicting short- to medium-term care home admission risk in older adults: a systematic review of externally validated models

INTRODUCTION

Predicting risk of care home admission could identify older adults for early intervention to support independent living but require external validation in a different dataset before clinical use. We systematically reviewed external validations of care home admission risk prediction models in older adults.

METHODS

We searched Medline, Embase and Cochrane Library until 14 August 2023 for external validations of prediction models for care home admission risk in adults aged ≥65 years with up to 3 years of follow-up. We extracted and narratively synthesised data on study design, model characteristics, and model discrimination and calibration (accuracy of predictions). We assessed risk of bias and applicability using Prediction model Risk Of Bias Assessment Tool.

RESULTS

Five studies reporting validations of nine unique models were included. Model applicability was fair but risk of bias was mostly high due to not reporting model calibration. Morbidities were used as predictors in four models, most commonly neurological or psychiatric diseases. Physical function was also included in four models. For 1-year prediction, three of the six models had acceptable discrimination (area under the receiver operating characteristic curve (AUC)/c statistic 0.70-0.79) and the remaining three had poor discrimination (AUC < 0.70). No model accounted for competing mortality risk. The only study examining model calibration (but ignoring competing mortality) concluded that it was excellent.

CONCLUSIONS

The reporting of models was incomplete. Model discrimination was at best acceptable, and calibration was rarely examined (and ignored competing mortality risk when examined). There is a need to derive better models that account for competing mortality risk and report calibration as well as discrimination.

External validation of six COVID-19 prognostic models for predicting mortality risk in older populations in a hospital, primary care, and nursing home setting

OBJECTIVES

To systematically evaluate the performance of COVID-19 prognostic models and scores for mortality risk in older populations across three health-care settings: hospitals, primary care, and nursing homes.

STUDY DESIGN AND SETTING

This retrospective external validation study included 14,092 older individuals of ≥70 years of age with a clinical or polymerase chain reaction-confirmed COVID-19 diagnosis from March 2020 to December 2020. The six validation cohorts include three hospital-based (CliniCo, COVID-OLD, COVID-PREDICT), two primary care-based (Julius General Practitioners Network/Academisch network huisartsgeneeskunde/Network of Academic general Practitioners, PHARMO), and one nursing home cohort (YSIS) in the Netherlands. Based on a living systematic review of COVID-19 prediction models using Prediction model Risk Of Bias ASsessment Tool for quality and risk of bias assessment and considering predictor availability in validation cohorts, we selected six prognostic models predicting mortality risk in adults with COVID-19 infection (GAL-COVID-19 mortality, 4C Mortality Score, National Early Warning Score 2-extended model, Xie model, Wang clinical model, and CURB65 score). All six prognostic models were validated in the hospital cohorts and the GAL-COVID-19 mortality model was validated in all three healthcare settings. The primary outcome was in-hospital mortality for hospitals and 28-day mortality for primary care and nursing home settings. Model performance was evaluated in each validation cohort separately in terms of discrimination, calibration, and decision curves. An intercept update was performed in models indicating miscalibration followed by predictive performance re-evaluation.

MAIN OUTCOME MEASURE

In-hospital mortality for hospitals and 28-day mortality for primary care and nursing home setting.

RESULTS

All six prognostic models performed poorly and showed miscalibration in the older population cohorts. In the hospital settings, model performance ranged from calibration-in-the-large -1.45 to 7.46, calibration slopes 0.24-0.81, and C-statistic 0.55-0.71 with 4C Mortality Score performing as the most discriminative and well-calibrated model. Performance across health-care settings was similar for the GAL-COVID-19 model, with a calibration-in-the-large in the range of -2.35 to -0.15 indicating overestimation, calibration slopes of 0.24-0.81 indicating signs of overfitting, and C-statistic of 0.55-0.71.

CONCLUSION

Our results show that most prognostic models for predicting mortality risk performed poorly in the older population with COVID-19, in each health-care setting: hospital, primary care, and nursing home settings. Insights into factors influencing predictive model performance in the older population are needed for pandemic preparedness and reliable prognostication of health-related outcomes in this demographic.

External Validation of Postpartum Hemorrhage Prediction Models Using Electronic Health Record Data

OBJECTIVE

A recent study leveraging machine learning methods found that postpartum hemorrhage (PPH) can be predicted accurately at the time of labor admission in the U.S. Consortium for Safe Labor (CSL) dataset, with a C-statistic as high as 0.93. These CSL models were developed in older data (2002-2008) and used an estimated blood loss (EBL) of ≥1,000 mL to define PPH. We sought to externally validate these models using a more recent cohort of births where blood loss was measured using quantitative blood loss (QBL) methods.

STUDY DESIGN

Using data from 5,261 deliveries between February 1, 2019 and May 11, 2020 at a single tertiary hospital, we mapped our electronic health record (EHR) data to the 55 predictors described in previously published CSL models. PPH was defined as QBL ≥1,000 mL within 24 hours after delivery. Model discrimination and calibration of the four CSL models were measured using our cohort. In a secondary analysis, we fit new models in our study cohort using the same predictors and algorithms as the original CSL models.

RESULTS

The original study cohort had a substantially lower rate of PPH, 4.8% (7,279/228,438) versus 25% (1,321/5,261), possibly due to differences in measurement. The CSL models had lower discrimination in our study cohort, with a C-statistic as high as 0.57 (logistic regression). Models refit in our study cohort achieved better discrimination, with a C-statistic as high as 0.64 (random forest). Calibration improved in the refit models as compared with the original models.

CONCLUSION

The CSL models' accuracy was lower in a contemporary EHR where PPH is assessed using QBL. As institutions continue to adopt QBL methods, further data are needed to understand the differences between EBL and QBL to enable accurate prediction of PPH.

KEY POINTS

· Machine learning methods may help predict PPH.. · EBL models do not generalize when QBL is used.. · Blood loss estimation alters model accuracy..

External validation and update of the International Medical Prevention Registry on Venous Thromboembolism bleeding risk score for predicting bleeding in acutely ill hospitalized medical patients: a retrospective single-center cohort study in Japan

BACKGROUND

The International Medical Prevention Registry for Venous Thromboembolism (IMPROVE) Bleeding Risk Score is the recommended risk assessment model (RAM) for predicting bleeding risk in acutely ill medical inpatients in Western countries. However, few studies have assessed its predictive performance in local Asian settings.

METHODS

We retrospectively identified acutely ill adolescents and adults (aged ≥ 15 years) who were admitted to our general internal medicine department between July 5, 2016 and July 5, 2021, and extracted data from their electronic medical records. The outcome of interest was the cumulative incidence of major and nonmajor but clinically relevant bleeding 14 days after admission. For the two-risk-group model, we estimated sensitivity, specificity, and positive and negative predictive values (PPV and NPV, respectively). For the 11-risk-group model, we estimated C statistic, expected and observed event ratio (E/O), calibration-in-the-large (CITL), and calibration slope. In addition, we recalibrated the intercept using local data to update the RAM.

RESULTS

Among the 3,876 included patients, 998 (26%) were aged ≥ 85 years, while 656 (17%) were hospitalized in the intensive care unit. The median length of hospital stay was 14 days. Clinically relevant bleeding occurred in 58 patients (1.5%), 49 (1.3%) of whom experienced major bleeding. Sensitivity, specificity, NPV, and PPV were 26.1% (95% confidence interval [CI]: 15.8-40.0%), 84.8% (83.6-85.9%), 98.7% (98.2-99.0%), and 2.5% (1.5-4.3%) for any bleeding and 30.9% (95% CI: 18.8-46.3%), 84.9% (83.7-86.0%), 99.0% (98.5-99.3%), and 2.5% (1.5-4.3%) for major bleeding, respectively. The C statistic, E/O, CITL, and calibration slope were 0.64 (95% CI: 0.58-0.71), 1.69 (1.45-2.05), - 0.55 (- 0.81 to - 0.29), and 0.58 (0.29-0.87) for any bleeding and 0.67 (95% CI: 0.60-0.74), 0.76 (0.61-0.87), 0.29 (0.00-0.58), and 0.42 (0.19-0.64) for major bleeding, respectively. Updating the model substantially corrected the poor calibration observed.

CONCLUSIONS

In our Japanese cohort, the IMPROVE bleeding RAM retained the reported moderate discriminative performance. Model recalibration substantially improved the poor calibration obtained using the original RAM. Before its introduction into clinical practice, the updated RAM needs further validation studies and an optimized threshold.

A Vulnerability Index to Assess the Risk of SARS-CoV-2-Related Hospitalization/Death: Urgent Need for an Update after Diffusion of Anti-COVID Vaccines

Background: There are algorithms to predict the risk of SARS-CoV-2-related complications. Given the spread of anti-COVID vaccination, which sensibly modified the burden of risk of the infection, these tools need to be re-calibrated. Therefore, we updated our vulnerability index, namely, the Health Search (HS)-CoVulnerabiltyIndex (VI)d (HS-CoVId), to predict the risk of SARS-CoV-2-related hospitalization/death in the primary care setting. Methods: We formed a cohort of individuals aged ≥15 years and diagnosed with COVID-19 between 1 January and 31 December 2021 in the HSD. The date of COVID-19 diagnosis was the study index date. These patients were eligible if they had received an anti-COVID vaccine at least 15 days before the index date. Patients were followed up from the index date until one of the following events, whichever came first: COVID-19-related hospitalization/death (event date), end of registration with their GPs, and end of the study period (31 December 2022). To calculate the incidence rate of COVID-19-related hospitalization/death, a patient-specific score was derived through linear combination of the coefficients stemming from a multivariate Cox regression model. Its prediction performance was evaluated by obtaining explained variation, discrimination, and calibration measures. Results: We identified 2192 patients who had received an anti-COVID vaccine from 1 January to 31 December 2021. With this cohort, we re-calibrated the HS-CoVId by calculating optimism-corrected pseudo-R2, AUC, and calibration slope. The final model reported a good predictive performance by explaining 58% (95% CI: 48-71%) of variation in the occurrence of hospitalizations/deaths, the AUC was 83 (95% CI: 77-93%), and the calibration slope did not reject the equivalence hypothesis (p-value = 0.904). Conclusions: Two versions of HS-CoVId need to be differentially adopted to assess the risk of COVID-19-related complications among vaccinated and unvaccinated subjects. Therefore, this functionality should be operationalized in related patient- and population-based informatic tools intended for general practitioners.