Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.

Utility of automated memory measures in identifying cognitive impairment in adults with epilepsy

OBJECTIVE

Cognitive impairment is prevalent in epilepsy and often presents at the time of initial diagnosis. This study sought to validate brief, self-administered, iPad-based recognition memory tasks in a sample of patients with epilepsy and to examine their screening utility in identifying patients with cognitive impairment.

METHODS

The Words and Faces tests were administered to 145 adult patients with epilepsy along with a neuropsychological battery. Correlation analyses examined the convergent and divergent validity of the Words and Faces tests, and a series of logistic regression analyses examined discriminative ability in identifying patients with and without cognitive impairments on neuropsychological measures. Patient performance was compared to that of a healthy control group (n = 223), and the relationship between the Words and Faces test performance and disease-related variables (i.e., antiepileptic medication burden, seizure lateralization/localization) was examined.

RESULTS

The Words and Faces tests were positively correlated with traditional paper-and-pencil neuropsychological measures of episodic memory, with generally moderate to large effect sizes (r > .40), while correlations between the Words and Faces tests and non-memory measures were generally small in magnitude (r < .30). Patients with epilepsy had significantly lower scores on Words and Faces tests compared to healthy controls, and performance was associated with antiepileptic medication burden and seizure localization. The Words and Faces tests demonstrated good predictive accuracy in identifying any cognitive impairment (concordance (c) statistic = .77) and excellent predictive accuracy (c = .85) in identifying patients with impairments on traditional memory measures. The Words and Faces tests also demonstrated reasonable discrimination for impairments in non-memory domains including executive function, language, attention, processing speed, and visuospatial ability (c = .62 -.70). Importantly, the Words and Faces Immediate Index performed just as well as the Total Score (which included delayed memory performance), suggesting a short version of this measure is sufficient for identifying patients with cognitive impairment.

CONCLUSIONS

The Words and Faces tests are valid, computerized tools that can be used to screen for memory and other cognitive impairment in adults with epilepsy.

Modified neoadjuvant rectal score as a novel prognostic model for rectal cancer patients who underwent chemoradiotherapy

BACKGROUND

The neoadjuvant rectal score (NAR score) has recently been proposed as a better prognostic model than the conventional TNM classification for rectal cancer patients that have undergone neoadjuvant chemoradiotherapy. We recently developed an apoptosis-detection technique for assessing the viability of residual tumors in resected specimens after chemoradiotherapy. This study aimed to establish an improved prognostic classification by combining the NAR score and the assessment of the apoptosis of residual cancer cells.

METHODS

We retrospectively enrolled 319 rectal cancer patients who underwent chemoradiotherapy followed by radical surgery. The recurrence-free survival and overall survival of the four models were compared: TNM stage, NAR score, modified TNM stage by re-staging according to cancer cell viability, and modified NAR score also by re-staging.

RESULTS

Downstaging of the ypT stage was observed in 15.5% of cases, whereas only 4.5% showed downstaging of ypN stage. C-index was highest for the modified NAR score (0.715), followed by the modified TNM, TNM, and NAR score. Similarly, Akaike's information criterion was smallest in the modified NAR score (926.2), followed by modified TNM, TNM, and NAR score, suggesting that the modified NAR score was the best among these four models. The overall survival results were similar: C-index was the highest (0.767) and Akaike's information criterion was the smallest (383.9) for the modified NAR score among the four models tested.

CONCLUSION

We established a novel prognostic model, for rectal cancer patients that have undergone neoadjuvant chemoradiotherapy, using a combination of apoptosis-detecting immunohistochemistry and neoadjuvant rectal scores.

Development and internal validation of a multivariable model for the prediction of the probability of 1-year readmission to the emergency department for acute alcohol intoxication

To develop and internally validate a multivariable logistic regression model (LRM) for the prediction of the probability of 1-year readmission to the emergency department (ED) in patients with acute alcohol intoxication (AAI). We developed and internally validated the LRM on a previously analyzed retrospective cohort of 3304 patients with AAI admitted to the ED of the Sant'Orsola-Malpighi Hospital (Bologna, Italy). The benchmark LRM employed readmission to the same ED for AAI within 1 year as the binary outcome, age as a continuous predictor, and sex, alcohol use disorder, substance use disorder, at least one previous admission for trauma, mental or behavioral disease, and homelessness as the binary predictors. Optimism correction was performed using the bootstrap on 1000 samples without replacement. The benchmark LRM was gradually simplified to get the most parsimonious LRM with similar optimism-corrected overall fit, discrimination and calibration. The 1-year readmission rate was 15.7% (95% CI 14.4-16.9%). A reduced LRM based on sex, age, at least one previous admission for trauma, mental or behavioral disease, and homelessness, performed nearly as well as the benchmark LRM. The reduced LRM had the following optimism-corrected metrics: scaled Brier score 17.0%, C-statistic 0.799 (95% CI 0.778 to 0.821), calibration in the large 0.000 (95% CI - 0.099 to 0.099), calibration slope 0.985 (95% CI 0.893 to 1.088), and an acceptably accurate calibration plot. An LRM based on sex, age, at least one previous admission for trauma, mental or behavioral disease, and homelessness can be used to estimate the probability of 1-year readmission to ED for AAI. To begin proving its clinical utility, this LRM should be validated in external cohorts.

External validation and update of the International Medical Prevention Registry on Venous Thromboembolism bleeding risk score for predicting bleeding in acutely ill hospitalized medical patients: a retrospective single-center cohort study in Japan

BACKGROUND

The International Medical Prevention Registry for Venous Thromboembolism (IMPROVE) Bleeding Risk Score is the recommended risk assessment model (RAM) for predicting bleeding risk in acutely ill medical inpatients in Western countries. However, few studies have assessed its predictive performance in local Asian settings.

METHODS

We retrospectively identified acutely ill adolescents and adults (aged ≥ 15 years) who were admitted to our general internal medicine department between July 5, 2016 and July 5, 2021, and extracted data from their electronic medical records. The outcome of interest was the cumulative incidence of major and nonmajor but clinically relevant bleeding 14 days after admission. For the two-risk-group model, we estimated sensitivity, specificity, and positive and negative predictive values (PPV and NPV, respectively). For the 11-risk-group model, we estimated C statistic, expected and observed event ratio (E/O), calibration-in-the-large (CITL), and calibration slope. In addition, we recalibrated the intercept using local data to update the RAM.

RESULTS

Among the 3,876 included patients, 998 (26%) were aged ≥ 85 years, while 656 (17%) were hospitalized in the intensive care unit. The median length of hospital stay was 14 days. Clinically relevant bleeding occurred in 58 patients (1.5%), 49 (1.3%) of whom experienced major bleeding. Sensitivity, specificity, NPV, and PPV were 26.1% (95% confidence interval [CI]: 15.8-40.0%), 84.8% (83.6-85.9%), 98.7% (98.2-99.0%), and 2.5% (1.5-4.3%) for any bleeding and 30.9% (95% CI: 18.8-46.3%), 84.9% (83.7-86.0%), 99.0% (98.5-99.3%), and 2.5% (1.5-4.3%) for major bleeding, respectively. The C statistic, E/O, CITL, and calibration slope were 0.64 (95% CI: 0.58-0.71), 1.69 (1.45-2.05), - 0.55 (- 0.81 to - 0.29), and 0.58 (0.29-0.87) for any bleeding and 0.67 (95% CI: 0.60-0.74), 0.76 (0.61-0.87), 0.29 (0.00-0.58), and 0.42 (0.19-0.64) for major bleeding, respectively. Updating the model substantially corrected the poor calibration observed.

CONCLUSIONS

In our Japanese cohort, the IMPROVE bleeding RAM retained the reported moderate discriminative performance. Model recalibration substantially improved the poor calibration obtained using the original RAM. Before its introduction into clinical practice, the updated RAM needs further validation studies and an optimized threshold.

Evaluating the clinical utility of an easily applicable prediction model of suicide attempts, newly developed and validated with a general community sample of adults

BACKGROUND

A suicide attempt (SA) is a clinically serious action. Researchers have argued that reducing long-term SA risk may be possible, provided that at-risk individuals are identified and receive adequate treatment. Algorithms may accurately identify at-risk individuals. However, the clinical utility of algorithmically estimated long-term SA risk has never been the predominant focus of any study.

METHODS

The data of this report stem from CoLaus|PsyCoLaus, a prospective longitudinal study of general community adults from Lausanne, Switzerland. Participants (N = 4,097; Mage = 54 years, range: 36-86; 54% female) were assessed up to four times, starting in 2003, approximately every 4-5 years. Long-term individual SA risk was prospectively predicted, using logistic regression. This algorithm's clinical utility was assessed by net benefit (NB). Clinical utility expresses a tool's benefit after having taken this tool's potential harm into account. Net benefit is obtained, first, by weighing the false positives, e.g., 400 individuals, at the risk threshold, e.g., 1%, using its odds (odds of 1% yields 1/(100-1) = 1/99), then by subtracting the result (400*1/99 = 4.04) from the true positives, e.g., 5 individuals (5-4.04), and by dividing the result (0.96) by the sample size, e.g., 800 (0.96/800). All results are based on 100 internal cross-validations. The predictors used in this study were: lifetime SA, any lifetime mental disorder, sex, and age.

RESULTS

SA at any of the three follow-up study assessments was reported by 1.2%. For a range of seven a priori selected threshold probabilities, ranging between 0.5% and 2%, logistic regression showed highest overall NB in 97.4% of all 700 internal cross-validations (100 for each selected threshold probability).

CONCLUSION

Despite the strong class imbalance of the outcome (98.8% no, 1.2% yes) and only four predictors, clinical utility was observed. That is, using the logistic regression model for clinical decision making provided the most true positives, without an increase of false positives, compared to all competing decision strategies. Clinical utility is one among several important prerequisites of implementing an algorithm in routine practice, and may possibly guide a clinicians' treatment decision making to reduce long-term individual SA risk. The novel metric NB may become a standard performance measure, because the a priori invested clinical considerations enable clinicians to interpret the results directly.

External validation and application of the Diabetes Population Risk Tool (DPoRT) for prediction of type 2 diabetes onset in the US population

INTRODUCTION

Characterizing diabetes risk in the population is important for population health assessment and diabetes prevention planning. We aimed to externally validate an existing 10-year population risk model for type 2 diabetes in the USA and model the population benefit of diabetes prevention approaches using population survey data.

RESEARCH DESIGN AND METHODS

The Diabetes Population Risk Tool (DPoRT), originally derived and validated in Canada, was applied to an external validation cohort of 23 477 adults from the 2009 National Health Interview Survey (NHIS). We assessed predictive performance for discrimination (C-statistic) and calibration plots against observed incident diabetes cases identified from the NHIS 2009-2018 cycles. We applied DPoRT to the 2018 NHIS cohort (n=21 187) to generate 10-year risk prediction estimates and characterize the preventive benefit of three diabetes prevention scenarios: (1) community-wide strategy; (2) high-risk strategy and (3) combined approach.

RESULTS

DPoRT demonstrated good discrimination (C-statistic=0.778 (males); 0.787 (females)) and good calibration across the range of risk. We predicted a baseline risk of 10.2% and 21 076 000 new cases of diabetes in the USA from 2018 to 2028. The community-wide strategy and high-risk strategy estimated diabetes risk reductions of 0.2% and 0.3%, respectively. The combined approach estimated a 0.4% risk reduction and 843 000 diabetes cases averted in 10 years.

CONCLUSIONS

DPoRT has transportability for predicting population-level diabetes risk in the USA using routinely collected survey data. We demonstrate the model's applicability for population health assessment and diabetes prevention planning. Our modeling predicted that the combination of community-wide and targeted prevention approaches for those at highest risk are needed to reduce diabetes burden in the USA.

Temporal validation of a clinical prediction rule for distinguishing locomotive syndromes in community-dwelling older adults: A cross-sectional study from the DETECt-L study

Objectives

Clinical prediction rules are used to discriminate patients with locomotive syndrome and may enable early detection. This study aimed to validate the clinical predictive rules for locomotive syndrome in community-dwelling older adults.

Methods

We assessed the clinical prediction rules for locomotive syndrome in a cross-sectional setting. The age, sex, and body mass index of participants were recorded. Five physical function tests-grip strength, single-leg standing time, timed up-and-go test, and preferred and maximum walking speeds-were measured as predictive factors. Three previously developed clinical prediction models for determining the severity of locomotive syndrome were assessed using a decision tree analysis. To assess validity, the sensitivity, specificity, likelihood ratio, and post-test probability of the clinical prediction rules were calculated using receiver operating characteristic curve analysis for each model.

Results

Overall, 280 older adults were included (240 women; mean age, 74.8 ± 5.2 years), and 232 (82.9%), 68 (24.3%), and 28 (10.0%) participants had locomotive syndrome stages ≥ 1, ≥ 2, and = 3, respectively. The areas under the receiver operating characteristics curves were 0.701, 0.709, and 0.603, in models 1, 2, and 3, respectively. The accuracies of models 1 and 2 were moderate.

Conclusions

These findings indicate that the models are reliable for community-dwelling older adults.

Clinical Predictive Modeling of Heart Failure: Domain Description, Models' Characteristics and Literature Review

This study attempts to identify and briefly describe the current directions in applied and theoretical clinical prediction research. Context-rich chronic heart failure syndrome (CHFS) telemedicine provides the medical foundation for this effort. In the chronic stage of heart failure, there are sudden exacerbations of syndromes with subsequent hospitalizations, which are called acute decompensation of heart failure (ADHF). These decompensations are the subject of diagnostic and prognostic predictions. The primary purpose of ADHF predictions is to clarify the current and future health status of patients and subsequently optimize therapeutic responses. We proposed a simplified discrete-state disease model as an attempt at a typical summarization of a medical subject before starting predictive modeling. The study tries also to structure the essential common characteristics of quantitative models in order to understand the issue in an application context. The last part provides an overview of prediction works in the field of CHFS. These three parts provide the reader with a comprehensive view of quantitative clinical predictive modeling in heart failure telemedicine with an emphasis on several key general aspects. The target community is medical researchers seeking to align their clinical studies with prognostic or diagnostic predictive modeling, as well as other predictive researchers. The study was written by a non-medical expert.

geoFOR: A collaborative forensic taphonomy database for estimating the postmortem interval

Accurately assessing the postmortem interval (PMI), or the time since death, remains elusive within forensic science research and application. This paper introduces geoFOR, a web-based collaborative application that utilizes ArcGIS and machine learning to deliver improved PMI predictions. The geoFOR application provides a standardized, collaborative forensic taphonomy database that gives practitioners a readily available tool to enter case information that automates the collection of environmental data and delivers a PMI prediction using statistically robust methods. After case submission, the cross-validating machine learning PMI predictive model results in a R² value of 0.82. Contributors receive a predicted PMI with an 80% confidence interval. The geoFOR database currently contains 2529 entries from across the U.S. and includes cases from medicolegal investigations and longitudinal studies from human decomposition facilities. We present the overall findings of the data collected so far and compare results from medicolegal cases and longitudinal studies to highlight previously poorly understood limitations involved in the difficult task of PMI estimation. This novel approach for building a reference dataset of human decomposition is forensically and geographically representative of the realities in which human remains are discovered which allows for continual improvement of PMI estimations as more data is captured. It is our goal that the geoFOR data repository follow the principles of Open Science and be made available to forensic researchers to test, refine, and improve PMI models. Mass collaboration and data sharing can ultimately address enduring issues associated with accurately estimating the PMI within medicolegal death investigations.

Towards a framework for interoperability and reproducibility of predictive models

The development and deployment of machine learning (ML) models for biomedical research and healthcare currently lacks standard methodologies. Although tools for model replication are numerous, without a unifying blueprint it remains difficult to scientifically reproduce predictive ML models for any number of reasons (e.g., assumptions regarding data distributions and preprocessing, unclear test metrics, etc.) and ultimately, questions around generalizability and transportability are not readily answered. To facilitate scientific reproducibility, we built upon the Predictive Model Markup Language (PMML) to capture essential information. As a key component of the PREdictive Model Index and Exchange REpository (PREMIERE) platform, we present the Automated Metadata Pipeline (AMP) for conversion of a given predictive ML model into an extended PMML file that autocompletes an ML-based checklist, assessing model elements for interoperability and reproducibility. We demonstrate this pipeline on multiple test cases with three different ML algorithms and health-related datasets, providing a foundation for future predictive model reproducibility, sharing, and comparison.

Clinical prediction models for aneurysmal subarachnoid hemorrhage: a systematic review update

BACKGROUND

A systematic review of clinical prediction models for aneurysmal subarachnoid hemorrhage (aSAH) reported in 2011 noted that clinical prediction models for aSAH were developed using poor methods and were not externally validated. This study aimed to update the above review to guide the future development of predictive models in aSAH.

METHODS

We systematically searched Embase and MEDLINE databases (January 2010 to February 2022) for articles that reported the development of a clinical prediction model to predict functional outcomes in aSAH. Our reviews are based on the items included in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement (PRISMA) checklist, and on data abstracted from each study in accord with the Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) 2014 checklist. Bias and applicability were assessed using the Prediction model Risk Of Bias Assessment Tool (PROBAST).

RESULTS

We reviewed data on 30 466 patients contributing to 29 prediction models abstracted from 22 studies identified from an initial search of 7858 studies. Most models were developed using logistic regression (n=20) or machine learning (n=9) with prognostic variables selected through a range of methods. Age (n=13), World Federation of Neurological Surgeons (WFNS) grade (n=11), hypertension (n=6), aneurysm size (n=5), Fisher grade (n=12), Hunt and Hess score (n=5), and Glasgow Coma Scale (n=8) were the variables most frequently included in the reported models. External validation was performed in only four studies. All but one model had a high or unclear risk of bias due to poor performance or lack of validation.

CONCLUSION

Externally validated models for the prediction of functional outcome in aSAH patients have now become available. However, most of them still have a high risk of bias.

Assessment of the ABC2-SPH risk score to predict invasive mechanical ventilation in COVID-19 patients and comparison to other scores

Background

Predicting the need for invasive mechanical ventilation (IMV) is important for the allocation of human and technological resources, improvement of surveillance, and use of effective therapeutic measures. This study aimed (i) to assess whether the ABC2-SPH score is able to predict the receipt of IMV in COVID-19 patients; (ii) to compare its performance with other existing scores; (iii) to perform score recalibration, and to assess whether recalibration improved prediction.

Methods

Retrospective observational cohort, which included adult laboratory-confirmed COVID-19 patients admitted in 32 hospitals, from 14 Brazilian cities. This study was conducted in two stages: (i) for the assessment of the ABC2-SPH score and comparison with other available scores, patients hospitalized from July 31, 2020, to March 31, 2022, were included; (ii) for ABC2-SPH score recalibration and also comparison with other existing scores, patients admitted from January 1, 2021, to March 31, 2022, were enrolled. For both steps, the area under the receiving operator characteristic score (AUROC) was calculated for all scores, while a calibration plot was assessed only for the ABC2-SPH score. Comparisons between ABC2-SPH and the other scores followed the Delong Test recommendations. Logistic recalibration methods were used to improve results and adapt to the studied sample.

Results

Overall, 9,350 patients were included in the study, the median age was 58.5 (IQR 47.0-69.0) years old, and 45.4% were women. Of those, 33.5% were admitted to the ICU, 25.2% received IMV, and 17.8% died. The ABC2-SPH score showed a significantly greater discriminatory capacity, than the CURB-65, STSS, and SUM scores, with potentialized results when we consider only patients younger than 80 years old (AUROC 0.714 [95% CI 0.698-0.731]). Thus, after the ABC2-SPH score recalibration, we observed improvements in calibration (slope = 1.135, intercept = 0.242) and overall performance (Brier score = 0.127).

Conclusion

The ABC2-SPHr risk score demonstrated a good performance to predict the need for mechanical ventilation in COVID-19 hospitalized patients under 80 years of age.

Automatic correction of performance drift under acquisition shift in medical image classification

Image-based prediction models for disease detection are sensitive to changes in data acquisition such as the replacement of scanner hardware or updates to the image processing software. The resulting differences in image characteristics may lead to drifts in clinically relevant performance metrics which could cause harm in clinical decision making, even for models that generalise in terms of area under the receiver-operating characteristic curve. We propose Unsupervised Prediction Alignment, a generic automatic recalibration method that requires no ground truth annotations and only limited amounts of unlabelled example images from the shifted data distribution. We illustrate the effectiveness of the proposed method to detect and correct performance drift in mammography-based breast cancer screening and on publicly available histopathology data. We show that the proposed method can preserve the expected performance in terms of sensitivity/specificity under various realistic scenarios of image acquisition shift, thus offering an important safeguard for clinical deployment.

Cohort profile: the Turin prostate cancer prognostication (TPCP) cohort

Introduction

Prostate cancer (PCa) is the most frequent tumor among men in Europe and has both indolent and aggressive forms. There are several treatment options, the choice of which depends on multiple factors. To further improve current prognostication models, we established the Turin Prostate Cancer Prognostication (TPCP) cohort, an Italian retrospective biopsy cohort of patients with PCa and long-term follow-up. This work presents this new cohort with its main characteristics and the distributions of some of its core variables, along with its potential contributions to PCa research.

Methods

The TPCP cohort includes consecutive non-metastatic patients with first positive biopsy for PCa performed between 2008 and 2013 at the main hospital in Turin, Italy. The follow-up ended on December 31st 2021. The primary outcome is the occurrence of metastasis; death from PCa and overall mortality are the secondary outcomes. In addition to numerous clinical variables, the study's prognostic variables include histopathologic information assigned by a centralized uropathology review using a digital pathology software system specialized for the study of PCa, tumor DNA methylation in candidate genes, and features extracted from digitized slide images via Deep Neural Networks.

Results

The cohort includes 891 patients followed-up for a median time of 10 years. During this period, 97 patients had progression to metastatic disease and 301 died; of these, 56 died from PCa. In total, 65.3% of the cohort has a Gleason score less than or equal to 3 + 4, and 44.5% has a clinical stage cT1. Consistent with previous studies, age and clinical stage at diagnosis are important prognostic factors: the crude cumulative incidence of metastatic disease during the 14-years of follow-up increases from 9.1% among patients younger than 64 to 16.2% for patients in the age group of 75-84, and from 6.1% for cT1 stage to 27.9% in cT3 stage.

Discussion

This study stands to be an important resource for updating existing prognostic models for PCa on an Italian cohort. In addition, the integrated collection of multi-modal data will allow development and/or validation of new models including new histopathological, digital, and molecular markers, with the goal of better directing clinical decisions to manage patients with PCa.

Validation of Risk Stratification for Cardiac Events in Pregnant Women With Valvular Heart Disease

BACKGROUND

Most risk stratification tools for pregnant patients with heart disease were developed in high-income countries and in populations with predominantly congenital heart disease, and therefore, may not be generalizable to those with valvular heart disease (VHD).

OBJECTIVES

The purpose of this study was to validate and establish the clinical utility of 2 risk stratification tools-DEVI (VHD-specific tool) and CARPREG-II-for predicting adverse cardiac events in pregnant patients with VHD.

METHODS

We conducted a cohort study involving consecutive pregnancies complicated with VHD admitted to a tertiary center in a middle-income setting from January 2019 to April 2022. Individual risk for adverse composite cardiac events was calculated using DEVI and CARPREG-II models. Performance was assessed through discrimination and calibration characteristics. Clinical utility was evaluated with Decision Curve Analysis.

RESULTS

Of 577 eligible pregnancies, 69 (12.1%) experienced a component of the composite outcome. A majority (94.7%) had rheumatic etiology, with mitral regurgitation as the predominant lesion (48.2%). The area under the receiver-operating characteristic curve was 0.884 (95% CI: 0.844-0.923) for the DEVI and 0.808 (95% CI: 0.753-0.863) for the CARPREG-II models. Calibration plots suggested that DEVI score overestimates risk at higher probabilities, whereas CARPREG-II score overestimates risk at both extremes and underestimates risk at middle probabilities. Decision curve analysis demonstrated that both models were useful across predicted probability thresholds between 10% and 50%.

CONCLUSIONS

In pregnant patients with VHD, DEVI and CARPREG-II scores showed good discriminative ability and clinical utility across a range of probabilities. The DEVI score showed better agreement between predicted probabilities and observed events.

Prognostic models for chronic postsurgical pain-Current developments, trends, and challenges

PURPOSE OF REVIEW

Prognostic models for chronic postsurgical pain (CPSP) aim to predict the likelihood for development and severity of CPSP in individual patients undergoing surgical procedures. Such models might provide valuable information for healthcare providers, allowing them to identify patients at higher risk and implement targeted interventions to prevent or manage CPSP effectively. This review discusses the latest developments of prognostic models for CPSP, their challenges, limitations, and future directions.

RECENT FINDINGS

Numerous studies have been conducted aiming to develop prognostic models for CPSP using various perioperative factors. These include patient-related factors like demographic variables, preexisting pain conditions, psychosocial aspects, procedure-specific characteristics, perioperative analgesic strategies, postoperative complications and, as indicated most recently, biomarkers. Model generation, however, varies and performance and accuracy differ between prognostic models for several reasons and validation of models is rather scarce.

SUMMARY

Precise methodology of prognostic model development needs advancements in the field of CPSP. Development of more accurate, validated and refined models in large-scale cohorts is needed to improve reliability and applicability in clinical practice and validation studies are necessary to further refine and improve the performance of prognostic models for CPSP.

Violence risk assessment instruments in forensic psychiatric populations: a systematic review and meta-analysis

BACKGROUND

Although structured tools have been widely used to predict violence risk in specialist mental health settings, there is uncertainty about the extent and quality of evidence of their predictive performance. We aimed to systematically review the predictive performance of tools used to assess violence risk in forensic mental health, where they are routinely administered.

METHODS

In our systematic review and meta-analysis, we followed PRISMA guidelines and searched four databases (PsycINFO, Embase, Medline, and Global Health) from database inception to Nov 1, 2022, to identify studies examining the predictive performance of risk assessment tools in people discharged from forensic (secure) mental health hospitals. Systematic and narrative reviews were excluded from the review. Performance measures and descriptive statistics were extracted from published reports. A quality assessment was performed for each study using the Prediction Model Risk of Bias Assessment Tool. Meta-analysis was conducted on the performance of instruments that were independently externally validated with a sample size greater than 100. The study was registered with PROSPERO, CRD42022304716.

FINDINGS

We conducted a systematic review of 50 eligible publications, assessing the predictive performance of 36 tools, providing data for 10 460 participants (88% men, 12% women; median age [from 47 studies] was 35 years, IQR 33-38) from 12 different countries. Post-discharge interpersonal violence and crime was most often measured by new criminal offences or recidivism (47 [94%] of 50 studies); only three studies used informant or self-report data on physical aggression or violent behaviour. Overall, the predictive performance of risk assessment tools was mixed. Most studies reported one discrimination metric, the area under the receiver operating characteristic curve (AUC); other key performance measures such as calibration, sensitivity, and specificity were not presented. Most studies had a high risk of bias (49 [98%] of 50), partly due to poor analytical approaches. A meta-analysis was conducted for violent recidivism on 29 independent external validations from 19 studies with at least 100 patients. Pooled AUCs for predicting violent outcomes ranged from 0·72 (0·65-0·79; I2=0%) for H10, to 0·69 for the Historical Clinical Risk Management-20 version 2 (95% CI 0·65-0·72; I2=0%) and Violence Risk Appraisal Guide (0·63-0·75; I2=0%), to 0·64 for the Static-99 (0·53-0·73; I2=45%).

INTERPRETATION

Current violence risk assessment tools in forensic mental health have mixed evidence of predictive performance. Forensic mental health services should review their use of current risk assessment tools and consider implementing those with higher-quality evidence in support.

FUNDING

Wellcome Trust.

Understanding the contribution of lifestyle in breast cancer risk prediction: a systematic review of models applicable to Europe

BACKGROUND

Breast cancer (BC) is a significant health concern among European women, with the highest prevalence rates among all cancers. Existing BC prediction models account for major risks such as hereditary, hormonal and reproductive factors, but research suggests that adherence to a healthy lifestyle can reduce the risk of developing BC to some extent. Understanding the influence and predictive role of lifestyle variables in current risk prediction models could help identify actionable, modifiable, targets among high-risk population groups.

PURPOSE

To systematically review population-based BC risk prediction models applicable to European populations and identify lifestyle predictors and their corresponding parameter values for a better understanding of their relative contribution to the prediction of incident BC.

METHODS

A systematic review was conducted in PubMed, Embase and Web of Science from January 2000 to August 2021. Risk prediction models were included if (i) developed and/or validated in adult cancer-free women in Europe, (ii) based on easily ascertained information, and (iii) reported models' final predictors. To investigate further the comparability of lifestyle predictors across models, estimates were standardised into risk ratios and visualised using forest plots.

RESULTS

From a total of 49 studies, 33 models were developed and 22 different existing models, mostly from Gail (22 studies) and Tyrer-Cuzick and co-workers (12 studies) were validated or modified for European populations. Family history of BC was the most frequently included predictor (31 models), while body mass index (BMI) and alcohol consumption (26 and 21 models, respectively) were the lifestyle predictors most often included, followed by smoking and physical activity (7 and 6 models respectively). Overall, for lifestyle predictors, their modest predictive contribution was greater for riskier lifestyle levels, though highly variable model estimates across different models.

CONCLUSIONS

Given the increasing BC incidence rates in Europe, risk models utilising readily available risk factors could greatly aid in widening the population coverage of screening efforts, while the addition of lifestyle factors could help improving model performance and serve as intervention targets of prevention programmes.

Development and External Validation of Partial Proportional Odds Risk Prediction Models for Cancer Stage at Diagnosis among Males and Females in Canada

Risk prediction models for cancer stage at diagnosis may identify individuals at higher risk of late-stage cancer diagnoses. Partial proportional odds risk prediction models for cancer stage at diagnosis for males and females were developed using data from Alberta's Tomorrow Project (ATP). Prediction models were validated on the British Columbia Generations Project (BCGP) cohort using discrimination and calibration measures. Among ATP males, older age at diagnosis was associated with an earlier stage at diagnosis, while full- or part-time employment, prostate-specific antigen testing, and former/current smoking were associated with a later stage at diagnosis. Among ATP females, mammogram and sigmoidoscopy or colonoscopy were associated with an earlier stage at diagnosis, while older age at diagnosis, number of pregnancies, and hysterectomy were associated with a later stage at diagnosis. On external validation, discrimination results were poor for both males and females while calibration results indicated that the models did not over- or under-fit to derivation data or over- or under-predict risk. Multiple factors associated with cancer stage at diagnosis were identified among ATP participants. While the prediction model calibration was acceptable, discrimination was poor when applied to BCGP data. Updating our models with additional predictors may help improve predictive performance.

Predicting outcomes at the individual patient level: what is the best method?

OBJECTIVE

When developing prediction models, researchers commonly employ a single model which uses all the available data (end-to-end approach). Alternatively, a similarity-based approach has been previously proposed, in which patients with similar clinical characteristics are first grouped into clusters, then prediction models are developed within each cluster. The potential advantage of the similarity-based approach is that it may better address heterogeneity in patient characteristics. However, it remains unclear whether it improves the overall predictive performance. We illustrate the similarity-based approach using data from people with depression and empirically compare its performance with the end-to-end approach.

METHODS

We used primary care data collected in general practices in the UK. Using 31 predefined baseline variables, we aimed to predict the severity of depressive symptoms, measured by Patient Health Questionnaire-9, 60 days after initiation of antidepressant treatment. Following the similarity-based approach, we used k-means to cluster patients based on their baseline characteristics. We derived the optimal number of clusters using the Silhouette coefficient. We used ridge regression to build prediction models in both approaches. To compare the models' performance, we calculated the mean absolute error (MAE) and the coefficient of determination (R2) using bootstrapping.

RESULTS

We analysed data from 16 384 patients. The end-to-end approach resulted in an MAE of 4.64 and R2 of 0.20. The best-performing similarity-based model was for four clusters, with MAE of 4.65 and R2 of 0.19.

CONCLUSIONS

The end-to-end and the similarity-based model yielded comparable performance. Due to its simplicity, the end-to-end approach can be favoured when using demographic and clinical data to build prediction models on pharmacological treatments for depression.

Prediction of ciprofloxacin resistance in hospitalized patients using machine learning

BACKGROUND

Ciprofloxacin is a widely used antibiotic that has lost efficiency due to extensive resistance. We developed machine learning (ML) models that predict the probability of ciprofloxacin resistance in hospitalized patients.

METHODS

Data were collected from electronic records of hospitalized patients with positive bacterial cultures, during 2016-2019. Susceptibility results to ciprofloxacin (n = 10,053 cultures) were obtained for Escherichia coli, Klebsiella pneumoniae, Morganella morganii, Pseudomonas aeruginosa, Proteus mirabilis and Staphylococcus aureus. An ensemble model, combining several base models, was developed to predict ciprofloxacin resistant cultures, either with (gnostic) or without (agnostic) information on the infecting bacterial species.

RESULTS

The ensemble models' predictions are well-calibrated, and yield ROC-AUCs (area under the receiver operating characteristic curve) of 0.737 (95%CI 0.715-0.758) and 0.837 (95%CI 0.821-0.854) on independent test-sets for the agnostic and gnostic datasets, respectively. Shapley additive explanations analysis identifies that influential variables are related to resistance of previous infections, where patients arrived from (hospital, nursing home, etc.), and recent resistance frequencies in the hospital. A decision curve analysis reveals that implementing our models can be beneficial in a wide range of cost-benefits considerations of ciprofloxacin administration.

CONCLUSIONS

This study develops ML models to predict ciprofloxacin resistance in hospitalized patients. The models achieve high predictive ability, are well calibrated, have substantial net-benefit across a wide range of conditions, and rely on predictors consistent with the literature. This is a further step on the way to inclusion of ML decision support systems into clinical practice.

There is no such thing as a validated prediction model

BACKGROUND

Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context?

MAIN BODY

We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models.

CONCLUSION

Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.