External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges

Development and external validation of a head and neck cancer risk prediction model

BACKGROUND

Head and neck cancer (HNC) incidence is on the rise, often diagnosed at late stage and associated with poor prognoses. Risk prediction tools have a potential role in prevention and early detection.

METHODS

The IARC-ARCAGE European case-control study was used as the model development dataset. A clinical HNC risk prediction model using behavioral and demographic predictors was developed via multivariable logistic regression analyses. The model was then externally validated in the UK Biobank cohort. Model performance was tested using discrimination and calibration metrics.

RESULTS

1926 HNC cases and 2043 controls were used for the development of the model. The development dataset model including sociodemographic, smoking, and alcohol variables had moderate discrimination, with an area under curve (AUC) value of 0.75 (95% CI, 0.74-0.77); the calibration slope (0.75) and tests were suggestive of good calibration. 384 616 UK Biobank participants (with 1177 HNC cases) were available for external validation of the model. Upon external validation, the model had an AUC of 0.62 (95% CI, 0.61-0.64).

CONCLUSION

We developed and externally validated a HNC risk prediction model using the ARCAGE and UK Biobank studies, respectively. This model had moderate performance in the development population and acceptable performance in the validation dataset. Demographics and risk behaviors are strong predictors of HNC, and this model may be a helpful tool in primary dental care settings to promote prevention and determine recall intervals for dental examination. Future addition of HPV serology or genetic factors could further enhance individual risk prediction.

Identifying patients with undiagnosed small intestinal neuroendocrine tumours in primary care using statistical and machine learning: model development and validation study

BACKGROUND

Neuroendocrine tumours (NETs) are increasing in incidence, often diagnosed at advanced stages, and individuals may experience years of diagnostic delay, particularly when arising from the small intestine (SI). Clinical prediction models could present novel opportunities for case finding in primary care.

METHODS

An open cohort of adults (18+ years) contributing data to the Optimum Patient Care Research Database between 1st Jan 2000 and 30th March 2023 was identified. This database collects de-identified data from general practices in the UK. Model development approaches comprised logistic regression, penalised regression, and XGBoost. Performance (discrimination and calibration) was assessed using internal-external cross-validation. Decision analysis curves compared clinical utility.

RESULTS

Of 11.7 million individuals, 382 had recorded SI NET diagnoses (0.003%). The XGBoost model had the highest AUC (0.869, 95% confidence interval [CI]: 0.841-0.898) but was mildly miscalibrated (slope 1.165, 95% CI: 1.088-1.243; calibration-in-the-large 0.010, 95% CI: -0.164 to 0.185). Clinical utility was similar across all models.

DISCUSSION

Multivariable prediction models may have clinical utility in identifying individuals with undiagnosed SI NETs using information in their primary care records. Further evaluation including external validation and health economics modelling may identify cost-effective strategies for case finding for this uncommon tumour.

Systematic Review of Prognosis Models in Predicting Tooth Loss in Periodontitis

This study reviews and appraises the methodological and reporting quality of prediction models for tooth loss in periodontitis patients, including the use of regression and machine learning models. Studies involving prediction modeling for tooth loss in periodontitis patients were screened. A search was performed in MEDLINE via PubMed, Embase, and CENTRAL up to 12 February 2022, with citation chasing. Studies exploring model development or external validation studies for models assessing tooth loss in periodontitis patients for clinical use at any time point, with all prediction horizons in English, were considered. Studies were excluded if models were not developed for use in periodontitis patients, were not developed or validated on any data set, predicted outcomes other than tooth loss, or were prognostic factor studies. The CHARMS checklist was used for data extraction, TRIPOD to assess reporting quality, and PROBAST to assess the risk of bias. In total, 4,661 records were screened, and 45 studies were included. Only 26 studies reported any kind of performance measure. The median C-statistic reported was 0.671 (range, 0.57-0.97). All studies were at a high risk of bias due to inappropriate handling of missing data (96%), inappropriate evaluation of model performance (92%), and lack of accounting for model overfitting in evaluating model performance (68%). Many models predicting tooth loss in periodontitis are available, but studies evaluating these models are at a high risk of bias. Model performance measures are likely to be overly optimistic and might not be replicated in clinical use. While this review is unable to recommend any model for clinical practice, it has collated the existing models and their model performance at external validation and their associated sample sizes, which would be helpful to identify promising models for future external validation studies.

Multicentre validation of a machine learning model for predicting respiratory failure after noncardiac surgery

BACKGROUND

Postoperative respiratory failure is a serious complication that could benefit from early accurate identification of high-risk patients. We developed and validated a machine learning model to predict postoperative respiratory failure, defined as prolonged (>48 h) mechanical ventilation or reintubation after surgery.

METHODS

Easily extractable electronic health record (EHR) variables that do not require subjective assessment by clinicians were used. From EHR data of 307,333 noncardiac surgical cases, the model, trained with a gradient boosting algorithm, utilised a derivation cohort of 99,025 cases from Seoul National University Hospital (2013-9). External validation was performed using three separate cohorts A-C from different hospitals comprising 208,308 cases. Model performance was assessed by area under the receiver operating characteristic (AUROC) curve and area under the precision-recall curve (AUPRC), a measure of sensitivity and precision at different thresholds.

RESULTS

The model included eight variables: serum albumin, age, duration of anaesthesia, serum glucose, prothrombin time, serum creatinine, white blood cell count, and body mass index. Internally, the model achieved an AUROC of 0.912 (95% confidence interval [CI], 0.908-0.915) and AUPRC of 0.113. In external validation cohorts A, B, and C, the model achieved AUROCs of 0.879 (95% CI, 0.876-0.882), 0.872 (95% CI, 0.870-0.874), and 0.931 (95% CI, 0.925-0.936), and AUPRCs of 0.029, 0.083, and 0.124, respectively.

CONCLUSIONS

Utilising just eight easily extractable variables, this machine learning model demonstrated excellent discrimination in both internal and external validation for predicting postoperative respiratory failure. The model enables personalised risk stratification and facilitates data-driven clinical decision-making.

Generalization of a Deep Learning Model for Continuous Glucose Monitoring-Based Hypoglycemia Prediction: Algorithm Development and Validation Study

Background

Predicting hypoglycemia while maintaining a low false alarm rate is a challenge for the wide adoption of continuous glucose monitoring (CGM) devices in diabetes management. One small study suggested that a deep learning model based on the long short-term memory (LSTM) network had better performance in hypoglycemia prediction than traditional machine learning algorithms in European patients with type 1 diabetes. However, given that many well-recognized deep learning models perform poorly outside the training setting, it remains unclear whether the LSTM model could be generalized to different populations or patients with other diabetes subtypes.

Objective

The aim of this study was to validate LSTM hypoglycemia prediction models in more diverse populations and across a wide spectrum of patients with different subtypes of diabetes.

Methods

We assembled two large data sets of patients with type 1 and type 2 diabetes. The primary data set including CGM data from 192 Chinese patients with diabetes was used to develop the LSTM, support vector machine (SVM), and random forest (RF) models for hypoglycemia prediction with a prediction horizon of 30 minutes. Hypoglycemia was categorized into mild (glucose=54-70 mg/dL) and severe (glucose<54 mg/dL) levels. The validation data set of 427 patients of European-American ancestry in the United States was used to validate the models and examine their generalizations. The predictive performance of the models was evaluated according to the sensitivity, specificity, and area under the receiver operating characteristic curve (AUC).

Results

For the difficult-to-predict mild hypoglycemia events, the LSTM model consistently achieved AUC values greater than 97% in the primary data set, with a less than 3% AUC reduction in the validation data set, indicating that the model was robust and generalizable across populations. AUC values above 93% were also achieved when the LSTM model was applied to both type 1 and type 2 diabetes in the validation data set, further strengthening the generalizability of the model. Under different satisfactory levels of sensitivity for mild and severe hypoglycemia prediction, the LSTM model achieved higher specificity than the SVM and RF models, thereby reducing false alarms.

Conclusions

Our results demonstrate that the LSTM model is robust for hypoglycemia prediction and is generalizable across populations or diabetes subtypes. Given its additional advantage of false-alarm reduction, the LSTM model is a strong candidate to be widely implemented in future CGM devices for hypoglycemia prediction.

Predictive modelling of transport decisions and resources optimisation in pre-hospital setting using machine learning techniques

BACKGROUND

The global evolution of pre-hospital care systems faces dynamic challenges, particularly in multinational settings. Machine learning (ML) techniques enable the exploration of deeply embedded data patterns for improved patient care and resource optimisation. This study's objective was to accurately predict cases that necessitated transportation versus those that did not, using ML techniques, thereby facilitating efficient resource allocation.

METHODS

ML algorithms were utilised to predict patient transport decisions in a Middle Eastern national pre-hospital emergency medical care provider. A comprehensive dataset comprising 93,712 emergency calls from the 999-call centre was analysed using R programming language. Demographic and clinical variables were incorporated to enhance predictive accuracy. Random Forest (RF), Support Vector Machine (SVM), Extreme Gradient Boosting (XGBoost), and Adaptive Boosting (AdaBoost) algorithms were trained and validated.

RESULTS

All the trained algorithm models, particularly XGBoost (Accuracy = 83.1%), correctly predicted patients' transportation decisions. Further, they indicated statistically significant patterns that could be leveraged for targeted resource deployment. Moreover, the specificity rates were high; 97.96% in RF and 95.39% in XGBoost, minimising the incidence of incorrectly identified "Transported" cases (False Positive).

CONCLUSION

The study identified the transformative potential of ML algorithms in enhancing the quality of pre-hospital care in Qatar. The high predictive accuracy of the employed models suggested actionable avenues for day and time-specific resource planning and patient triaging, thereby having potential to contribute to pre-hospital quality, safety, and value improvement. These findings pave the way for more nuanced, data-driven quality improvement interventions with significant implications for future operational strategies.

Predictive value of C-reactive protein for postoperative liver-specific surgical site infections

BACKGROUND

C-reactive protein is a useful biological tool to predict infectious complications, but its predictive value in detecting organ-specific surgical site infection after liver resection has never been studied. We aimed to evaluate the predictive value of c-reactive protein and determine the cut-off values to detect postoperative liver resection-surgical site infection.

METHODS

A multicentric analysis of consecutive patients with liver resection between 2018 and 2021 was performed. The predictive value of postoperative day 1, postoperative day 3, and postoperative day 5 C-reactive protein levels was evaluated using the area under the receiver operating characteristic curve. Cut-off values were determined using the Youden index in a 500-fold bootstrap resampling of 500 patients treated at 3 centers, who comprised the development cohort and were tested in an external independent validation cohort of 166 patients at a fourth center.

RESULTS

Among the 500 patients who underwent liver resection of the development cohort, liver resection-surgical site infection occurred in 66 patients (13.2%), and the median time to diagnosis was 6.0 days (interquartile range, 4.0-9.0) days. Median C-reactive protein levels were significantly higher on postoperative day 1, postoperative day 3, and postoperative day 5 in the liver resection-surgical site infection group compared with the non-surgical site infection group (50.5 vs 34.5 ng/mL, 148.0 vs 72.5 ng/mL, and 128.4 vs 35.2 ng/mL, respectively; P < .001). Postoperative day 3 and postoperative day 5 C-reactive protein-level area under the curve values were 0.76 (95% confidence interval, 0.64-0.88, P < .001) and 0.82 (95% confidence interval, 0.72-0.92, P < .001), respectively. Postoperative day 3 and postoperative day 5 optimal cut-off values of 100 mg/L and 87.0 mg/L could be used to rule out liver resection-surgical site infection, with a negative predictive value of 87.0% (interquartile range, 70.2-93.8) and 76.0% (interquartile range, 65.0-88.0), respectively, in the validation cohort.

CONCLUSION

Postoperative day 3 and postoperative day 5 C-reactive protein levels may be valuable predictive tools for liver resection-surgical site infection and aid in hospital discharge decision-making in the absence of other liver-related complications.

The development and internal validation of a multivariable model predicting 6-month mortality for people with opioid use disorder presenting to community drug services in England: a protocol

BACKGROUND

People with opioid use disorder have substantially higher standardised mortality rates compared to the general population; however, lack of clear individual prognostic information presents challenges to prioritise or target interventions within drug treatment services. Previous prognostic models have been developed to estimate the risk of developing opioid use disorder and opioid-related overdose in people routinely prescribed opioids but, to our knowledge, none have been developed to estimate mortality risk in people accessing drug services with opioid use disorder. Initial presentation to drug services is a pragmatic time to evaluate mortality risk given the contemporaneous routine collection of prognostic indicators and as a decision point for appropriate service prioritisation and targeted intervention delivery. This study aims to develop and internally validate a model to estimate 6-month mortality risk for people with opioid use disorder from prognostic indicators recorded at initial assessment in drug services in England.

METHODS

An English national dataset containing records from individuals presenting to drug services between 1 April 2013 and 1 April 2023 (n > 800,000) (the National Drug Treatment Monitoring System (NDTMS)) linked to their lifetime hospitalisation and death records (Hospital Episode Statistics-Office of National Statistics (HES-ONS)). Twelve candidate prognostic indicator variables were identified based on literature review of demographic and clinical features associated with increased mortality for people in treatment for opioid use disorder. Variables will be extracted at initial presentation to drug services with mortality measured at 6 months. Two multivariable Cox regression models will be developed one for 6-month all-cause mortality and one for 6-month drug-related mortality using backward elimination with a fractional polynomial approach for continuous variables. Internal validation will be undertaken using bootstrapping methods. Discrimination of both models will be reported using Harrel's c and d-statistics. Calibration curves and slopes will be presented comparing expected and observed event rates.

DISCUSSION

The models developed and internally validated in this study aim to improve clinical assessment of mortality risk for people with opioid use disorder presenting to drug services in England. External validation in different populations will be required to develop the model into a tool to assist future clinical decision-making.

Metabolic profiling of smoking, associations with type 2 diabetes and interaction with genetic susceptibility

BACKGROUND

Smokers are at increased risk of type 2 diabetes (T2D), but the underlying mechanisms are unclear. We investigated if the smoking-T2D association is mediated by alterations in the metabolome and assessed potential interaction with genetic susceptibility to diabetes or insulin resistance.

METHODS

In UK Biobank (n = 93,722), cross-sectional analyses identified 208 metabolites associated with smoking, of which 131 were confirmed in Mendelian Randomization analyses, including glycoprotein acetyls, fatty acids, and lipids. Elastic net regression was applied to create a smoking-related metabolic signature. We estimated hazard ratios (HR) of incident T2D in relation to baseline smoking/metabolic signature and calculated the proportion of the smoking-T2D association mediated by the signature. Additive interaction between the signature and genetic risk scores for T2D (GRS-T2D) and insulin resistance (GRS-IR) on incidence of T2D was assessed as relative excess risk due to interaction (RERI).

FINDINGS

The HR of T2D was 1·73 (95% confidence interval (CI) 1·54 - 1·94) for current versus never smoking, and 38·3% of the excess risk was mediated by the metabolic signature. The metabolic signature and its mediation role were replicated in TwinGene. The metabolic signature was associated with T2D (HR: 1·61, CI 1·46 - 1·77 for values above vs. below median), with evidence of interaction with GRS-T2D (RERI: 0·81, CI: 0·23 - 1·38) and GRS-IR (RERI 0·47, CI: 0·02 - 0·92).

INTERPRETATION

The increased risk of T2D in smokers may be mediated through effects on the metabolome, and the influence of such metabolic alterations on diabetes risk may be amplified in individuals with genetic susceptibility to T2D or insulin resistance.

External validation and update of the International Medical Prevention Registry on Venous Thromboembolism bleeding risk score for predicting bleeding in acutely ill hospitalized medical patients: a retrospective single-center cohort study in Japan

BACKGROUND

The International Medical Prevention Registry for Venous Thromboembolism (IMPROVE) Bleeding Risk Score is the recommended risk assessment model (RAM) for predicting bleeding risk in acutely ill medical inpatients in Western countries. However, few studies have assessed its predictive performance in local Asian settings.

METHODS

We retrospectively identified acutely ill adolescents and adults (aged ≥ 15 years) who were admitted to our general internal medicine department between July 5, 2016 and July 5, 2021, and extracted data from their electronic medical records. The outcome of interest was the cumulative incidence of major and nonmajor but clinically relevant bleeding 14 days after admission. For the two-risk-group model, we estimated sensitivity, specificity, and positive and negative predictive values (PPV and NPV, respectively). For the 11-risk-group model, we estimated C statistic, expected and observed event ratio (E/O), calibration-in-the-large (CITL), and calibration slope. In addition, we recalibrated the intercept using local data to update the RAM.

RESULTS

Among the 3,876 included patients, 998 (26%) were aged ≥ 85 years, while 656 (17%) were hospitalized in the intensive care unit. The median length of hospital stay was 14 days. Clinically relevant bleeding occurred in 58 patients (1.5%), 49 (1.3%) of whom experienced major bleeding. Sensitivity, specificity, NPV, and PPV were 26.1% (95% confidence interval [CI]: 15.8-40.0%), 84.8% (83.6-85.9%), 98.7% (98.2-99.0%), and 2.5% (1.5-4.3%) for any bleeding and 30.9% (95% CI: 18.8-46.3%), 84.9% (83.7-86.0%), 99.0% (98.5-99.3%), and 2.5% (1.5-4.3%) for major bleeding, respectively. The C statistic, E/O, CITL, and calibration slope were 0.64 (95% CI: 0.58-0.71), 1.69 (1.45-2.05), - 0.55 (- 0.81 to - 0.29), and 0.58 (0.29-0.87) for any bleeding and 0.67 (95% CI: 0.60-0.74), 0.76 (0.61-0.87), 0.29 (0.00-0.58), and 0.42 (0.19-0.64) for major bleeding, respectively. Updating the model substantially corrected the poor calibration observed.

CONCLUSIONS

In our Japanese cohort, the IMPROVE bleeding RAM retained the reported moderate discriminative performance. Model recalibration substantially improved the poor calibration obtained using the original RAM. Before its introduction into clinical practice, the updated RAM needs further validation studies and an optimized threshold.

Inter hospital external validation of interpretable machine learning based triage score for the emergency department using common data model

Emergency departments (ED) are complex, triage is a main task in the ED to prioritize patient with limited medical resources who need them most. Machine learning (ML) based ED triage tool, Score for Emergency Risk Prediction (SERP), was previously developed using an interpretable ML framework with single center. We aimed to develop SERP with 3 Korean multicenter cohorts based on common data model (CDM) without data sharing and compare performance with inter-hospital validation design. This retrospective cohort study included all adult emergency visit patients of 3 hospitals in Korea from 2016 to 2017. We adopted CDM for the standardized multicenter research. The outcome of interest was 2-day mortality after the patients' ED visit. We developed each hospital SERP using interpretable ML framework and validated inter-hospital wisely. We accessed the performance of each hospital's score based on some metrics considering data imbalance strategy. The study population for each hospital included 87,670, 83,363 and 54,423 ED visits from 2016 to 2017. The 2-day mortality rate were 0.51%, 0.56% and 0.65%. Validation results showed accurate for inter hospital validation which has at least AUROC of 0.899 (0.858-0.940). We developed multicenter based Interpretable ML model using CDM for 2-day mortality prediction and executed Inter-hospital external validation which showed enough high accuracy.

Comparing code-free and bespoke deep learning approaches in ophthalmology

AIM

Code-free deep learning (CFDL) allows clinicians without coding expertise to build high-quality artificial intelligence (AI) models without writing code. In this review, we comprehensively review the advantages that CFDL offers over bespoke expert-designed deep learning (DL). As exemplars, we use the following tasks: (1) diabetic retinopathy screening, (2) retinal multi-disease classification, (3) surgical video classification, (4) oculomics and (5) resource management.

METHODS

We performed a search for studies reporting CFDL applications in ophthalmology in MEDLINE (through PubMed) from inception to June 25, 2023, using the keywords 'autoML' AND 'ophthalmology'. After identifying 5 CFDL studies looking at our target tasks, we performed a subsequent search to find corresponding bespoke DL studies focused on the same tasks. Only English-written articles with full text available were included. Reviews, editorials, protocols and case reports or case series were excluded. We identified ten relevant studies for this review.

RESULTS

Overall, studies were optimistic towards CFDL's advantages over bespoke DL in the five ophthalmological tasks. However, much of such discussions were identified to be mono-dimensional and had wide applicability gaps. High-quality assessment of better CFDL applicability over bespoke DL warrants a context-specific, weighted assessment of clinician intent, patient acceptance and cost-effectiveness. We conclude that CFDL and bespoke DL are unique in their own assets and are irreplaceable with each other. Their benefits are differentially valued on a case-to-case basis. Future studies are warranted to perform a multidimensional analysis of both techniques and to improve limitations of suboptimal dataset quality, poor applicability implications and non-regulated study designs.

CONCLUSION

For clinicians without DL expertise and easy access to AI experts, CFDL allows the prototyping of novel clinical AI systems. CFDL models concert with bespoke models, depending on the task at hand. A multidimensional, weighted evaluation of the factors involved in the implementation of those models for a designated task is warranted.

Why do probabilistic clinical models fail to transport between sites

The rising popularity of artificial intelligence in healthcare is highlighting the problem that a computational model achieving super-human clinical performance at its training sites may perform substantially worse at new sites. In this perspective, we argue that we should typically expect this failure to transport, and we present common sources for it, divided into those under the control of the experimenter and those inherent to the clinical data-generating process. Of the inherent sources we look a little deeper into site-specific clinical practices that can affect the data distribution, and propose a potential solution intended to isolate the imprint of those practices on the data from the patterns of disease cause and effect that are the usual target of probabilistic clinical models.

Predictors of Mortality in Veterans with Amyotrophic Lateral Sclerosis: Respiratory Status and Speech Disorder at Presentation

BACKGROUND There is a lack of accurate models to predict amyotrophic lateral sclerosis (ALS) disease course and outcomes. As a result, risk assessment and counseling, the timing of interventions, and their stratification in clinical trials are difficult. This study aimed to evaluate the association between symptoms at presentation and mortality. MATERIAL AND METHODS A single veterans hospital reviewed the electronic records of 105 veterans with ALS who were periodically followed in our ALS clinic between 2010 and 2021. A survival decision tree (≤3 or >3 years) was generated based on the statistical median survival of our data. The variables known to influence survival when alive were compared to patients who died. RESULTS The (mean±SD) age at onset was 62±11 years, M/F ratio 101: 4, and 90% were non-Hispanic whites. The initial score for the Amyotrophic Lateral Sclerosis Functional Rating Scale-Revised (ALSFRS-R) was 31±8.3. Dysarthria and shortness of breath (SOB) were present on initial presentation in 52 (49.5%) and 32 (30.5%) patients, respectively. Deaths occurred in 80 (76.2%) patients during the study period. The main cause of death was respiratory disease (failure and pneumonia, n=43 53.75%). Patients survived for >3 years on initial presentation with normal respiration and speech, compared to ≤3 years of survival in patients with dysarthria and SOB, irrespective of age. CONCLUSIONS This study suggests that for veterans with ALS, the main predictors of shorter survival were respiratory status and speech disorder on initial presentation to the clinic.

Multiple imputation of systematically missing data on gait speed in the Swedish National Study on Aging and Care

BACKGROUND

There is insufficient investigation of multiple imputation for systematically missing discrete variables in individual participant data meta-analysis (IPDMA) with a small number of included studies. Therefore, this study aims to evaluate the performance of three multiple imputation strategies - fully conditional specification (FCS), multivariate normal (MVN), conditional quantile imputation (CQI) - on systematically missing data on gait speed in the Swedish National Study on Aging and Care (SNAC).

METHODS

In total, 1 000 IPDMA were simulated with four prospective cohort studies based on the characteristics of the SNAC. The three multiple imputation strategies were analysed with a two-stage common-effect multivariable logistic model targeting the effect of three levels of gait speed (100% missing in one study) on 5-years mortality with common odds ratios set to OR1 = 0.55 (0.8-1.2 vs ≤0.8 m/s), and OR2 = 0.29 (>1.2 vs ≤0.8 m/s).

RESULTS

The average combined estimate for the mortality odds ratio OR1 (relative bias %) were 0.58 (8.2%), 0.58 (7.5%), and 0.55 (0.7%) for the FCS, MVN, and CQI, respectively. The average combined estimate for the mortality odds ratio OR2 (relative bias %) were 0.30 (2.5%), 0.33 (10.0%), and 0.29 (0.9%) for the FCS, MVN, and CQI respectively.

CONCLUSIONS

In our simulations of an IPDMA based on the SNAC where gait speed data was systematically missing in one study, all three imputation methods performed relatively well. The smallest bias was found for the CQI approach.

Machine learning classification of autism spectrum disorder based on reciprocity in naturalistic social interactions

Autism spectrum disorder is characterized by impaired social communication and interaction. As a neurodevelopmental disorder typically diagnosed during childhood, diagnosis in adulthood is preceded by a resource-heavy clinical assessment period. The ongoing developments in digital phenotyping give rise to novel opportunities within the screening and diagnostic process. Our aim was to quantify multiple non-verbal social interaction characteristics in autism and build diagnostic classification models independent of clinical ratings. We analyzed videos of naturalistic social interactions in a sample including 28 autistic and 60 non-autistic adults paired in dyads and engaging in two conversational tasks. We used existing open-source computer vision algorithms for objective annotation to extract information based on the synchrony of movement and facial expression. These were subsequently used as features in a support vector machine learning model to predict whether an individual was part of an autistic or non-autistic interaction dyad. The two prediction models based on reciprocal adaptation in facial movements, as well as individual amounts of head and body motion and facial expressiveness showed the highest precision (balanced accuracies: 79.5% and 68.8%, respectively), followed by models based on reciprocal coordination of head (balanced accuracy: 62.1%) and body (balanced accuracy: 56.7%) motion, as well as intrapersonal coordination processes (balanced accuracy: 44.2%). Combinations of these models did not increase overall predictive performance. Our work highlights the distinctive nature of non-verbal behavior in autism and its utility for digital phenotyping-based classification. Future research needs to both explore the performance of different prediction algorithms to reveal underlying mechanisms and interactions, as well as investigate the prospective generalizability and robustness of these algorithms in routine clinical care.

How Real-World Data Can Facilitate the Development of Precision Medicine Treatment in Psychiatry

Precision medicine has the ambition to improve treatment response and clinical outcomes through patient stratification and holds great potential for the treatment of mental disorders. However, several important factors are needed to transform current practice into a precision psychiatry framework. Most important are 1) the generation of accessible large real-world training and test data including genomic data integrated from multiple sources, 2) the development and validation of advanced analytical tools for stratification and prediction, and 3) the development of clinically useful management platforms for patient monitoring that can be integrated into health care systems in real-life settings. This narrative review summarizes strategies for obtaining the key elements-well-powered samples from large biobanks integrated with electronic health records and health registry data using novel artificial intelligence algorithms-to predict outcomes in severe mental disorders and translate these models into clinical management and treatment approaches. Key elements are massive mental health data and novel artificial intelligence algorithms. For the clinical translation of these strategies, we discuss a precision medicine platform for improved management of mental disorders. We use cases to illustrate how precision medicine interventions could be brought into psychiatry to improve the clinical outcomes of mental disorders.

Development and External Validation of Models to Predict Need for Nursing Home Level of Care in Community-Dwelling Older Adults With Dementia

Importance

Most older adults living with dementia ultimately need nursing home level of care (NHLOC).

Objective

To develop models to predict need for NHLOC among older adults with probable dementia using self-report and proxy reports to aid patients and family with planning and care management.

Design, Setting, and Participants

This prognostic study included data from 1998 to 2016 from the Health and Retirement Study (development cohort) and from 2011 to 2019 from the National Health and Aging Trends Study (validation cohort). Participants were community-dwelling adults 65 years and older with probable dementia. Data analysis was conducted between January 2022 and October 2023.

Exposures

Candidate predictors included demographics, behavioral/health factors, functional measures, and chronic conditions.

Main Outcomes and Measures

The primary outcome was need for NHLOC defined as (1) 3 or more activities of daily living (ADL) dependencies, (2) 2 or more ADL dependencies and presence of wandering/need for supervision, or (3) needing help with eating. A Weibull survival model incorporating interval censoring and competing risk of death was used. Imputation-stable variable selection was used to develop 2 models: one using proxy responses and another using self-responses. Model performance was assessed by discrimination (integrated area under the receiver operating characteristic curve [iAUC]) and calibration (calibration plots).

Results

Of 3327 participants with probable dementia in the Health and Retirement Study, the mean (SD) age was 82.4 (7.4) years and 2301 (survey-weighted 70%) were female. At the end of follow-up, 2107 participants (63.3%) were classified as needing NHLOC. Predictors for both final models included age, baseline ADL and instrumental ADL dependencies, and driving status. The proxy model added body mass index and falls history. The self-respondent model added female sex, incontinence, and date recall. Optimism-corrected iAUC after bootstrap internal validation was 0.72 (95% CI, 0.70-0.75) in the proxy model and 0.64 (95% CI, 0.62-0.66) in the self-respondent model. On external validation in the National Health and Aging Trends Study (n = 1712), iAUC in the proxy and self-respondent models was 0.66 (95% CI, 0.61-0.70) and 0.64 (95% CI, 0.62-0.67), respectively. There was excellent calibration across the range of predicted risk.

Conclusions and Relevance

This prognostic study showed that relatively simple models using self-report or proxy responses can predict need for NHLOC in community-dwelling older adults with probable dementia with moderate discrimination and excellent calibration. These estimates may help guide discussions with patients and families in future care planning.

Transportability of bacterial infection prediction models for critically ill patients

OBJECTIVE

Bacterial infections (BIs) are common, costly, and potentially life-threatening in critically ill patients. Patients with suspected BIs may require empiric multidrug antibiotic regimens and therefore potentially be exposed to prolonged and unnecessary antibiotics. We previously developed a BI risk model to augment practices and help shorten the duration of unnecessary antibiotics to improve patient outcomes. Here, we have performed a transportability assessment of this BI risk model in 2 tertiary intensive care unit (ICU) settings and a community ICU setting. We additionally explored how simple multisite learning techniques impacted model transportability.

METHODS

Patients suspected of having a community-acquired BI were identified in 3 datasets: Medical Information Mart for Intensive Care III (MIMIC), Northwestern Medicine Tertiary (NM-T) ICUs, and NM "community-based" ICUs. ICU encounters from MIMIC and NM-T datasets were split into 70/30 train and test sets. Models developed on training data were evaluated against the NM-T and MIMIC test sets, as well as NM community validation data.

RESULTS

During internal validations, models achieved AUROCs of 0.78 (MIMIC) and 0.81 (NM-T) and were well calibrated. In the external community ICU validation, the NM-T model had robust transportability (AUROC 0.81) while the MIMIC model transported less favorably (AUROC 0.74), likely due to case-mix differences. Multisite learning provided no significant discrimination benefit in internal validation studies but offered more stability during transport across all evaluation datasets.

DISCUSSION

These results suggest that our BI risk models maintain predictive utility when transported to external cohorts.

CONCLUSION

Our findings highlight the importance of performing external model validation on myriad clinically relevant populations prior to implementation.

Clinical prediction models and the multiverse of madness

BACKGROUND

Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice.

MAIN BODY

We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it-were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual's predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual's prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice.

CONCLUSIONS

Instability is concerning as an individual's predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.

Attention-based neural networks for clinical prediction modelling on electronic health records

BACKGROUND

Deep learning models have had a lot of success in various fields. However, on structured data they have struggled. Here we apply four state-of-the-art supervised deep learning models using the attention mechanism and compare against logistic regression and XGBoost using discrimination, calibration and clinical utility.

METHODS

We develop the models using a general practitioners database. We implement a recurrent neural network, a transformer with and without reverse distillation and a graph neural network. We measure discrimination using the area under the receiver operating characteristic curve (AUC) and the area under the precision recall curve (AUPRC). We assess smooth calibration using restricted cubic splines and clinical utility with decision curve analysis.

RESULTS

Our results show that deep learning approaches can improve discrimination up to 2.5% points AUC and 7.4% points AUPRC. However, on average the baselines are competitive. Most models are similarly calibrated as the baselines except for the graph neural network. The transformer using reverse distillation shows the best performance in clinical utility on two out of three prediction problems over most of the prediction thresholds.

CONCLUSION

In this study, we evaluated various approaches in supervised learning using neural networks and attention. Here we do a rigorous comparison, not only looking at discrimination but also calibration and clinical utility. There is value in using deep learning models on electronic health record data since it can improve discrimination and clinical utility while providing good calibration. However, good baseline methods are still competitive.

Development and validation of a clinical prediction rule for acute appendicitis in children in primary care

BACKGROUND

Recognising acute appendicitis in children presenting with acute abdominal pain in primary care is challenging. General practitioners (GPs) may benefit from a clinical prediction rule.

OBJECTIVES

To develop and validate a clinical prediction rule for acute appendicitis in children presenting with acute abdominal pain in primary care.

METHODS

In a historical cohort study data was retrieved from GP electronic health records included in the Integrated Primary Care Information database. We assigned children aged 4-18 years presenting with acute abdominal pain (≤ 7 days) to development (2010-2012) and validation (2013-2016) cohorts, using acute appendicitis within six weeks as the outcome. Multiple logistic regression was used to develop a prediction model based on predictors with > 50% data availability derived from existing rules for secondary care. We performed internal and external temporal validation and derived a point score to stratify risk of appendicitis into three groups, i.e. low-risk, medium-risk and high-risk.

RESULTS

The development and validation cohorts included 2,041 and 3,650 children, of whom 95 (4.6%) and 195 (5.3%) had acute appendicitis. The model included male sex, pain duration (<24, 24-48, > 48 h), nausea/vomiting, elevated temperature (≥ 37.3 °C), abnormal bowel sounds, right lower quadrant tenderness, and peritoneal irritation. Internal and temporal validation showed good discrimination (C-statistics: 0.93 and 0.90, respectively) and excellent calibration. In the three groups, the risks of acute appendicitis were 0.5%, 7.5%, and 41%.

CONCLUSION

Combined with further testing in the medium-risk group, the prediction rule could improve clinical decision making and outcomes.

Development and Validation of the CANHEART Population-Based Laboratory Prediction Models for Atherosclerotic Cardiovascular Disease

BACKGROUND

Prediction of atherosclerotic cardiovascular disease (ASCVD) in primary prevention assessments exclusively with laboratory results may facilitate automated risk reporting and improve uptake of preventive therapies.

OBJECTIVE

To develop and validate sex-specific prediction models for ASCVD using age and routine laboratory tests and compare their performance with that of the pooled cohort equations (PCEs).

DESIGN

Derivation and validation of the CANHEART (Cardiovascular Health in Ambulatory Care Research Team) Lab Models.

SETTING

Population-based cohort study in Ontario, Canada.

PARTICIPANTS

A derivation and internal validation cohort of adults aged 40 to 75 years without cardiovascular disease from April 2009 to December 2015; an external validation cohort of primary care patients from January 2010 to December 2014.

MEASUREMENTS

Age and laboratory predictors measured in the outpatient setting included serum total cholesterol, high-density lipoprotein cholesterol, triglycerides, hemoglobin, mean corpuscular volume, platelets, leukocytes, estimated glomerular filtration rate, and glucose. The ASCVD outcomes were defined as myocardial infarction, stroke, and death from ischemic heart or cerebrovascular disease within 5 years.

RESULTS

Sex-specific models were developed and internally validated in 2 160 497 women and 1 833 147 men. They were well calibrated, with relative differences less than 1% between mean predicted and observed risk for both sexes. The c-statistic was 0.77 in women and 0.71 in men. External validation in 31 697 primary care patients showed a relative difference less than 14% and an absolute difference less than 0.3 percentage points in mean predicted and observed risks for both sexes. The c-statistics for the laboratory models were 0.72 for both sexes and were not statistically significantly different from those for the PCEs in women (change in c-statistic, -0.01 [95% CI, -0.03 to 0.01]) or men (change in c-statistic, -0.01 [CI, -0.04 to 0.02]).

LIMITATION

Medication use was not available at the population level.

CONCLUSION

The CANHEART Lab Models predict ASCVD with similar accuracy to more complex models, such as the PCEs.

PRIMARY FUNDING SOURCE

Canadian Institutes of Health Research.

Optimising prediction of mortality, stroke, and major bleeding for patients with atrial fibrillation: validation of the GARFIELD-AF tool in UK primary care electronic records

BACKGROUND

The GARFIELD-AF tool is a novel risk tool that simultaneously assesses the risk of all-cause mortality, stroke or systemic embolism, and major bleeding in patients with atrial fibrillation (AF).

AIM

To validate the GARFIELD-AF tool using UK primary care electronic records.

DESIGN AND SETTING

A retrospective cohort study using the Clinical Practice Research Datalink (CPRD) linked with Hospital Episode Statistics data and Office for National Statistics mortality data.

METHOD

Discrimination was evaluated using the area under the curve (AUC) and calibration was evaluated using calibration-in-the-large regression and calibration plots.

RESULTS

A total of 486 818 patients aged ≥18 years with incident diagnosis of non-valvular AF between 2 January 1998 and 31 July 2020 were included; 50.6% (n = 246 425/486 818) received anticoagulation at diagnosis The GARFIELD- AF models outperformed the CHA2DS2VASc and HAS-BLED scores in discrimination ability of death, stroke, and major bleeding at all the time points. The AUC for events at 1 year for the 2017 models were: death 0.747 (95% confidence interval [CI] = 0.744 to 0.751) versus 0.635 (95% CI = 0.631 to 0.639) for CHA2DS2VASc; stroke 0.666 (95% CI = 0.663 to 0.669) versus 0.625 (95% CI = 0.622 to 0.628) for CHA2DS2VASc; and major bleeding 0.602 (95% CI = 0.598 to 0.606) versus 0.558 (95% CI = 0.554 to 0.562) for HAS- BLED. Calibration between predicted and Kaplan- Meier observed events was inadequate with the GARFIELD-AF models.

CONCLUSION

The GARFIELD-AF models were superior to the CHA2DS2VASc score for discriminating stroke and death and superior to the HAS-BLED score for discriminating major bleeding. The models consistently underpredicted the level of risk, suggesting that a recalibration is needed to optimise its use in the UK population.

The Trade Secret Taboo: Open Science Methods are Required to Improve Prediction Models in Sports Medicine and Performance

Clinical prediction models in sports medicine that utilize regression or machine learning techniques have become more widely published, used, and disseminated. However, these models are typically characterized by poor methodology and incomplete reporting, and an inadequate evaluation of performance, leading to unreliable predictions and weak clinical utility within their intended sport population. Before implementation in practice, models require a thorough evaluation. Strong replicable methods and transparency reporting allow practitioners and researchers to make independent judgments as to the model's validity, performance, clinical usefulness, and confidence it will do no harm. However, this is not reflected in the sports medicine literature. As shown in a recent systematic review of models for predicting sports injury models, most were typically characterized by poor methodology, incomplete reporting, and inadequate performance evaluation. Because of constraints imposed by data from individual teams, the development of accurate, reliable, and useful models is highly reliant on external validation. However, a barrier to collaboration is a desire to maintain a competitive advantage; a team's proprietary information is often perceived as high value, and so these 'trade secrets' are frequently guarded. These 'trade secrets' also apply to commercially available models, as developers are unwilling to share proprietary (and potentially profitable) development and validation information. In this Current Opinion, we: (1) argue that open science is essential for improving sport prediction models and (2) critically examine sport prediction models for open science practices.

Genetic Studies Investigating Susceptibility to Psoriatic Arthritis: A Narrative Review

PURPOSE

Approximately 30% of patients with psoriasis will develop psoriatic arthritis (PsA), leading to a decreased quality of life for the patient caused by increasing disability and additional health complications. The identification of risk factors for the development of PsA would facilitate the development of risk prediction models in which patients with psoriasis at high risk of developing PsA could be targeted in a stratified medicine approach, enabling early intervention and treatment. PsA is known to have a genetic contribution to susceptibility, and the identification of genetic risk factors that differentiate PsA from cutaneous-only psoriasis is a key area of research. This narrative review summarizes the discovery of genetic risk factors and, with the aid of a primer on risk prediction models, discusses their potential role for the classification of PsA risk and diagnosis.

METHODS

All relevant research articles were identified through searches of the PubMed database for literature published up until December 2022. Search terms included psoriatic arthritis, genetic susceptibility, genetic association, genome-wide association study, GWAS, prediction, and polygenic risk score.

FINDINGS

The current literature reveals considerable overlap between the genetic susceptibility loci for PsA and psoriasis. Several PsA-specific genetic risk factors have been reported, and most notably these implicate the HLA-B and IL23R genes. Efforts to include genetic risk factors in prediction models for the development of PsA have reported good discrimination.

IMPLICATIONS

Key messages emerging from this narrative are as follows: the limited number of PsA-specific susceptibility loci reported to date suggest larger studies are required, facilitated by international collaboration, to achieve the power to detect further genetic factors; the early promising results for genetic-based risk prediction require further validation in independent datasets; and risk prediction models combining clinical and genetic risk factors have yet to be explored.

Fixing the Leaky Pipe: How to Improve the Uptake of Patient-Reported Outcomes-Based Prognostic and Predictive Models in Cancer Clinical Practice

PURPOSE

This discussion paper outlines challenges and proposes solutions for successfully implementing prediction models that incorporate patient-reported outcomes (PROs) in cancer practice.

METHODS

We organized a full-day multidisciplinary meeting of people with expertise in cancer care delivery, PRO collection, PRO use in prediction modeling, computing, implementation, and decision science. The discussions presented here focused on identifying challenges to the development, implementation and use of prediction models incorporating PROs, and suggesting possible solutions.

RESULTS

Specific challenges and solutions were identified across three broad areas. (1) Understanding decision making and implementation: necessitating multidisciplinary collaboration in the early stages and throughout; early stakeholder engagement to define the decision problem and ensure acceptability of PROs in prediction; understanding patient/clinician interpretation of PRO predictions and uncertainty to optimize prediction impact; striving for model integration into existing electronic health records; and early regulatory alignment. (2) Recognizing the limitations to PRO collection and their impact on prediction: incorporating validated, clinically important PROs to maximize model generalizability and clinical engagement; and minimizing missing PRO data (resulting from both structural digital exclusion and time-varying factors) to avoid exacerbating existing inequalities. (3) Statistical and modeling challenges: incorporating statistical methods to address missing data; ensuring predictive modeling recognizes complex causal relationships; and considering temporal and geographic recalibration so that model predictions reflect the relevant population.

CONCLUSION

Developing and implementing PRO-based prediction models in cancer care requires extensive multidisciplinary working from the earliest stages, recognition of implementation challenges because of PRO collection and model presentation, and robust statistical methods to manage missing data, causality, and calibration. Prediction models incorporating PROs should be viewed as complex interventions, with their development and impact assessment carried out to reflect this.

The Subgroup Imperative: Chest Radiograph Classifier Generalization Gaps in Patient, Setting, and Pathology Subgroups

Purpose

To externally test four chest radiograph classifiers on a large, diverse, real-world dataset with robust subgroup analysis.

Materials and Methods

In this retrospective study, adult posteroanterior chest radiographs (January 2016-December 2020) and associated radiology reports from Trillium Health Partners in Ontario, Canada, were extracted and de-identified. An open-source natural language processing tool was locally validated and used to generate ground truth labels for the 197 540-image dataset based on the associated radiology report. Four classifiers generated predictions on each chest radiograph. Performance was evaluated using accuracy, positive predictive value, negative predictive value, sensitivity, specificity, F1 score, and Matthews correlation coefficient for the overall dataset and for patient, setting, and pathology subgroups.

Results

Classifiers demonstrated 68%-77% accuracy, 64%-75% sensitivity, and 82%-94% specificity on the external testing dataset. Algorithms showed decreased sensitivity for solitary findings (43%-65%), patients younger than 40 years (27%-39%), and patients in the emergency department (38%-60%) and decreased specificity on normal chest radiographs with support devices (59%-85%). Differences in sex and ancestry represented movements along an algorithm's receiver operating characteristic curve.

Conclusion

Performance of deep learning chest radiograph classifiers was subject to patient, setting, and pathology factors, demonstrating that subgroup analysis is necessary to inform implementation and monitor ongoing performance to ensure optimal quality, safety, and equity.Keywords: Conventional Radiography, Thorax, Ethics, Supervised Learning, Convolutional Neural Network (CNN), Machine Learning Algorithms Supplemental material is available for this article. © RSNA, 2023See also the commentary by Huisman and Hannink in this issue.

Development and validation of a novel risk prediction algorithm to estimate 10-year risk of oesophageal cancer in primary care: prospective cohort study and evaluation of performance against two other risk prediction models

Background

Methods to identify patients at increased risk of oesophageal cancer are needed to better identify those for targeted screening. We aimed to derive and validate novel risk prediction algorithms (CanPredict) to estimate the 10-year risk of oesophageal cancer and evaluate performance against two other risk prediction models.

Methods

Prospective open cohort study using routinely collected data from 1804 QResearch® general practices. We used 1354 practices (12.9 M patients) to develop the algorithm. We validated the algorithm in 450 separate practices from QResearch (4.12 M patients) and 355 Clinical Practice Research Datalink (CPRD) practices (2.53 M patients). The primary outcome was an incident diagnosis of oesophageal cancer found in GP, mortality, hospital, or cancer registry data. Patients were aged 25-84 years and free of oesophageal cancer at baseline. Cox proportional hazards models were used with prediction selection to derive risk equations. Risk factors included age, ethnicity, Townsend deprivation score, body mass index (BMI), smoking, alcohol, family history, relevant co-morbidities and medications. Measures of calibration, discrimination, sensitivity, and specificity were calculated in the validation cohorts.

Finding

There were 16,384 incident cases of oesophageal cancer in the derivation cohort (0.13% of 12.9 M). The predictors in the final algorithms were: age, BMI, Townsend deprivation score, smoking, alcohol, ethnicity, Barrett's oesophagus, hiatus hernia, H. pylori infection, use of proton pump inhibitors, anaemia, lung and blood cancer (with breast cancer in women). In the QResearch validation cohort in women the explained variation (R2) was 57.1%; Royston's D statistic 2.36 (95% CI 2.26-2.46); C statistic 0.859 (95% CI 0.849-0.868) and calibration was good. Results were similar in men. For the 20% at highest predicted risk, the sensitivity was 76%, specificity was 80.1% and the observed risk at 10 years was 0.76%. The results from the CPRD validation were similar.

Interpretation

We have developed and validated a novel prediction algorithm to quantify the absolute risk of oesophageal cancer. The CanPredict algorithms could be used to identify high risk patients for targeted screening.

Funding

Innovate UK and CRUK (grant 105857).

Propensity-based standardization to enhance the validation and interpretation of prediction model discrimination for a target population

External validation of the discriminative ability of prediction models is of key importance. However, the interpretation of such evaluations is challenging, as the ability to discriminate depends on both the sample characteristics (ie, case-mix) and the generalizability of predictor coefficients, but most discrimination indices do not provide any insight into their respective contributions. To disentangle differences in discriminative ability across external validation samples due to a lack of model generalizability from differences in sample characteristics, we propose propensity-weighted measures of discrimination. These weighted metrics, which are derived from propensity scores for sample membership, are standardized for case-mix differences between the model development and validation samples, allowing for a fair comparison of discriminative ability in terms of model characteristics in a target population of interest. We illustrate our methods with the validation of eight prediction models for deep vein thrombosis in 12 external validation data sets and assess our methods in a simulation study. In the illustrative example, propensity score standardization reduced between-study heterogeneity of discrimination, indicating that between-study variability was partially attributable to case-mix. The simulation study showed that only flexible propensity-score methods (allowing for non-linear effects) produced unbiased estimates of model discrimination in the target population, and only when the positivity assumption was met. Propensity score-based standardization may facilitate the interpretation of (heterogeneity in) discriminative ability of a prediction model as observed across multiple studies, and may guide model updating strategies for a particular target population. Careful propensity score modeling with attention for non-linear relations is recommended.

A pooled analysis of the risk prediction models for mortality in acute exacerbation of chronic obstructive pulmonary disease

OBJECTIVE

The prognosis for acute exacerbation of chronic obstructive pulmonary disease (AECOPD) is not optimistic, and severe AECOPD leads to an increased risk of mortality. Prediction models help distinguish between high- and low-risk groups. At present, many prediction models have been established and validated, which need to be systematically reviewed to screen out more suitable models that can be used in the clinic and provide evidence for future research.

METHODS

We searched PubMed, EMBASE, Cochrane Library and Web of Science databases for studies on risk models for AECOPD mortality from their inception to 10 April 2022. The risk of bias was assessed using the prediction model risk of bias assessment tool (PROBAST). Stata software (version 16) was used to synthesize the C-statistics for each model.

RESULTS

A total of 37 studies were included. The development of risk prediction models for mortality in patients with AECOPD was described in 26 articles, in which the most common predictors were age (n = 17), dyspnea grade (n = 11), altered mental status (n = 8), pneumonia (n = 6) and blood urea nitrogen (BUN, n = 6). The remaining 11 articles only externally validated existing models. All 37 studies were evaluated at a high risk of bias using PROBAST. We performed a meta-analysis of five models included in 15 studies. DECAF (dyspnoea, eosinopenia, consolidation, acidemia and atrial fibrillation) performed well in predicting in-hospital death [C-statistic = 0.91, 95% confidence interval (CI): 0.83, 0.98] and 90-day death [C-statistic = 0.76, 95% CI: 0.69, 0.82] and CURB-65 (confusion, urea, respiratory rate, blood pressure and age) performed well in predicting 30-day death [C-statistic = 0.74, 95% CI: 0.70, 0.77].

CONCLUSIONS

This study provides information on the characteristics, performance and risk of bias of a risk model for AECOPD mortality. This pooled analysis of the present study suggests that the DECAF performs well in predicting in-hospital and 90-day deaths. Yet, external validation in different populations is still needed to prove this performance.

The Skeletal Oncology Research Group Machine Learning Algorithm (SORG-MLA) for predicting prolonged postoperative opioid prescription after total knee arthroplasty: an international validation study using 3,495 patients from a Taiwanese cohort

BACKGROUND

Preoperative prediction of prolonged postoperative opioid use (PPOU) after total knee arthroplasty (TKA) could identify high-risk patients for increased surveillance. The Skeletal Oncology Research Group machine learning algorithm (SORG-MLA) has been tested internally while lacking external support to assess its generalizability. The aims of this study were to externally validate this algorithm in an Asian cohort and to identify other potential independent factors for PPOU.

METHODS

In a tertiary center in Taiwan, 3,495 patients receiving TKA from 2010-2018 were included. Baseline characteristics were compared between the external validation cohort and the original developmental cohorts. Discrimination (area under receiver operating characteristic curve [AUROC] and precision-recall curve [AUPRC]), calibration, overall performance (Brier score), and decision curve analysis (DCA) were applied to assess the model performance. A multivariable logistic regression was used to evaluate other potential prognostic factors.

RESULTS

There were notable differences in baseline characteristics between the validation and the development cohort. Despite these variations, the SORG-MLA ( https://sorg-apps.shinyapps.io/tjaopioid/ ) remained its good discriminatory ability (AUROC, 0.75; AUPRC, 0.34) and good overall performance (Brier score, 0.029; null model Brier score, 0.032). The algorithm could bring clinical benefit in DCA while somewhat overestimating the probability of prolonged opioid use. Preoperative acetaminophen use was an independent factor to predict PPOU (odds ratio, 2.05).

CONCLUSIONS

The SORG-MLA retained its discriminatory ability and good overall performance despite the different pharmaceutical regulations. The algorithm could be used to identify high-risk patients and tailor personalized prevention policy.

Development and External Validation of the International Early Warning Score for Improved Age- and Sex-Adjusted In-Hospital Mortality Prediction in the Emergency Department

OBJECTIVES

Early Warning Scores (EWSs) have a great potential to assist clinical decision-making in the emergency department (ED). However, many EWS contain methodological weaknesses in development and validation and have poor predictive performance in older patients. The aim of this study was to develop and externally validate an International Early Warning Score (IEWS) based on a recalibrated National Early warning Score (NEWS) model including age and sex and evaluate its performance independently at arrival to the ED in three age categories (18-65, 66-80, > 80 yr).

DESIGN

International multicenter cohort study.

SETTING

Data was used from three Dutch EDs. External validation was performed in two EDs in Denmark.

PATIENTS

All consecutive ED patients greater than or equal to 18 years in the Netherlands Emergency department Evaluation Database (NEED) with at least two registered vital signs were included, resulting in 95,553 patients. For external validation, 14,809 patients were included from a Danish Multicenter Cohort (DMC).

MEASUREMENTS AND MAIN RESULTS

Model performance to predict in-hospital mortality was evaluated by discrimination, calibration curves and summary statistics, reclassification, and clinical usefulness by decision curve analysis. In-hospital mortality rate was 2.4% ( n = 2,314) in the NEED and 2.5% ( n = 365) in the DMC. Overall, the IEWS performed significantly better than NEWS with an area under the receiving operating characteristic of 0.89 (95% CIs, 0.89-0.90) versus 0.82 (0.82-0.83) in the NEED and 0.87 (0.85-0.88) versus 0.82 (0.80-0.84) at external validation. Calibration for NEWS predictions underestimated risk in older patients and overestimated risk in the youngest, while calibration improved for IEWS with a substantial reclassification of patients from low to high risk and a standardized net benefit of 5-15% in the relevant risk range for all age categories.

CONCLUSIONS

The IEWS substantially improves in-hospital mortality prediction for all ED patients greater than or equal to18 years.

Determination of optimum intensity and duration of exercise based on the immune system response using a machine-learning model

One of the important concerns in the field of exercise immunology is determining the appropriate intensity and duration of exercise to prevent suppression of the immune system. Adopting a reliable approach to predict the number of white blood cells (WBCs) during exercise can help to identify the appropriate intensity and duration. Therefore, this study was designed to predict leukocyte levels during exercise with the application of a machine-learning model. We used a random forest (RF) model to predict the number of lymphocytes (LYMPH), neutrophils (NEU), monocytes (MON), eosinophils, basophils, and WBC. Intensity and duration of exercise, WBCs values before exercise training, body mass index (BMI), and maximal aerobic capacity (VO₂ max) were used as inputs and WBCs values after exercise training were assessed as outputs of the RF model. In this study, the data was collected from 200 eligible people and K-fold cross-validation was used to train and test the model. Finally, model efficiency was assessed using standard statistics (root mean square error (RMSE), mean absolute error (MAE), relative absolute error (RAE), root relative square error (RRSE), coefficient of determination (R²), and Nash-Sutcliffe efficiency coefficient (NSE)). Our findings revealed that the RF model performed well for predicting the number of WBC with RMSE = 0.94, MAE = 0.76, RAE = 48.54, RRSE = 48.17, NSE = 0.76, and R² = 0.77. Furthermore, the results showed that intensity and duration of exercise are more effective parameters than BMI and VO₂ max to predict the number of LYMPH, NEU, MON, and WBC during exercise. Totally, this study developed a novel approach based on the RF model using the relevant and accessible variables to predict WBCs during exercise. The proposed method can be applied as a promising and cost-effective tool for determining the correct intensity and duration of exercise in healthy people according to the body's immune system response.

Development and internal-external validation of statistical and machine learning models for breast cancer prognostication: cohort study

OBJECTIVE

To develop a clinically useful model that estimates the 10 year risk of breast cancer related mortality in women (self-reported female sex) with breast cancer of any stage, comparing results from regression and machine learning approaches.

DESIGN

Population based cohort study.

SETTING

QResearch primary care database in England, with individual level linkage to the national cancer registry, Hospital Episodes Statistics, and national mortality registers.

PARTICIPANTS

141 765 women aged 20 years and older with a diagnosis of invasive breast cancer between 1 January 2000 and 31 December 2020.

MAIN OUTCOME MEASURES

Four model building strategies comprising two regression (Cox proportional hazards and competing risks regression) and two machine learning (XGBoost and an artificial neural network) approaches. Internal-external cross validation was used for model evaluation. Random effects meta-analysis that pooled estimates of discrimination and calibration metrics, calibration plots, and decision curve analysis were used to assess model performance, transportability, and clinical utility.

RESULTS

During a median 4.16 years (interquartile range 1.76-8.26) of follow-up, 21 688 breast cancer related deaths and 11 454 deaths from other causes occurred. Restricting to 10 years maximum follow-up from breast cancer diagnosis, 20 367 breast cancer related deaths occurred during a total of 688 564.81 person years. The crude breast cancer mortality rate was 295.79 per 10 000 person years (95% confidence interval 291.75 to 299.88). Predictors varied for each regression model, but both Cox and competing risks models included age at diagnosis, body mass index, smoking status, route to diagnosis, hormone receptor status, cancer stage, and grade of breast cancer. The Cox model's random effects meta-analysis pooled estimate for Harrell's C index was the highest of any model at 0.858 (95% confidence interval 0.853 to 0.864, and 95% prediction interval 0.843 to 0.873). It appeared acceptably calibrated on calibration plots. The competing risks regression model had good discrimination: pooled Harrell's C index 0.849 (0.839 to 0.859, and 0.821 to 0.876, and evidence of systematic miscalibration on summary metrics was lacking. The machine learning models had acceptable discrimination overall (Harrell's C index: XGBoost 0.821 (0.813 to 0.828, and 0.805 to 0.837); neural network 0.847 (0.835 to 0.858, and 0.816 to 0.878)), but had more complex patterns of miscalibration and more variable regional and stage specific performance. Decision curve analysis suggested that the Cox and competing risks regression models tested may have higher clinical utility than the two machine learning approaches.

CONCLUSION

In women with breast cancer of any stage, using the predictors available in this dataset, regression based methods had better and more consistent performance compared with machine learning approaches and may be worthy of further evaluation for potential clinical use, such as for stratified follow-up.

Six Externally Validated Prognostic Models Have Potential Clinical Value to Predict Patient Health Outcomes in the Rehabilitation of Musculoskeletal Conditions: A Systematic Review

OBJECTIVE

The purpose of this systematic review was to identify and appraise externally validated prognostic models to predict a patient's health outcomes relevant to physical rehabilitation of musculoskeletal (MSK) conditions.

METHODS

We systematically reviewed 8 databases and reported our findings according to Preferred Reporting Items for Systematic Reviews and Meta-Analysis 2020. An information specialist designed a search strategy to identify externally validated prognostic models for MSK conditions. Paired reviewers independently screened the title, abstract, and full text and conducted data extraction. We extracted characteristics of included studies (eg, country and study design), prognostic models (eg, performance measures and type of model) and predicted clinical outcomes (eg, pain and disability). We assessed the risk of bias and concerns of applicability using the prediction model risk of bias assessment tool. We proposed and used a 5-step method to determine which prognostic models were clinically valuable.

RESULTS

We found 4896 citations, read 300 full-text articles, and included 46 papers (37 distinct models). Prognostic models were externally validated for the spine, upper limb, lower limb conditions, and MSK trauma, injuries, and pain. All studies presented a high risk of bias. Half of the models showed low concerns for applicability. Reporting of calibration and discrimination performance measures was often lacking. We found 6 externally validated models with adequate measures, which could be deemed clinically valuable [ie, (1) STart Back Screening Tool, (2) Wallis Occupational Rehabilitation RisK model, (3) Da Silva model, (4) PICKUP model, (5) Schellingerhout rule, and (6) Keene model]. Despite having a high risk of bias, which is mostly explained by the very conservative properties of the PROBAST tool, the 6 models remain clinically relevant.

CONCLUSION

We found 6 externally validated prognostic models developed to predict patients' health outcomes that were clinically relevant to the physical rehabilitation of MSK conditions.

IMPACT

Our results provide clinicians with externally validated prognostic models to help them better predict patients' clinical outcomes and facilitate personalized treatment plans. Incorporating clinically valuable prognostic models could inherently improve the value of care provided by physical therapists.

Leveraging machine learning to identify acute myeloid leukemia patients and their chemotherapy regimens in an administrative database

BACKGROUND

Administrative datasets are useful for identifying rare disease cohorts such as pediatric acute myeloid leukemia (AML). Previously, cohorts were assembled using labor-intensive, manual reviews of patients' longitudinal chemotherapy data.

METHODS

We utilized a two-step machine learning (ML) method to (i) identify pediatric patients with newly diagnosed AML, and (ii) among the identified AML patients, their chemotherapy courses, in an administrative/billing database. Using 2558 patients previously manually reviewed, multiple ML algorithms were derived from 75% of the study sample, and the selected model was tested in the remaining hold-out sample. The selected model was also applied to assemble a new pediatric AML cohort and further assessed in an external validation, using a standalone cohort established by manual chart abstraction.

RESULTS

For patient identification, the selected Support Vector Machine model yielded a sensitivity of 0.97 and a positive predictive value (PPV) of 0.97 in the hold-out test sample. For course-specific chemotherapy regimen and start date identification, the selected Random Forest model yielded overall PPV greater than or equal to 0.88 and sensitivity greater than or equal to 0.86 across all courses in the test sample. When applied to new cohort assembly, ML identified 3016 AML patients with 10,588 treatment courses. In the external validation subset, PPV was greater than or equal to 0.75 and sensitivity was greater than or equal to 0.82 for patient identification, and PPV was greater than or equal to 0.93 and sensitivity was greater than or equal to 0.94 for regimen identifications.

CONCLUSION

A carefully designed ML model can accurately identify pediatric AML patients and their chemotherapy courses from administrative databases. This approach may be generalizable to other diseases and databases.

Distal Pancreatectomy Fistula Risk Score (D-FRS): Development and International Validation

OBJECTIVE

To develop 2 distinct preoperative and intraoperative risk scores to predict postoperative pancreatic fistula (POPF) after distal pancreatectomy (DP) to improve preventive and mitigation strategies, respectively.

BACKGROUND

POPF remains the most common complication after DP. Despite several known risk factors, an adequate risk model has not been developed yet.

METHODS

Two prediction risk scores were designed using data of patients undergoing DP in 2 Italian centers (2014-2016) utilizing multivariable logistic regression. The preoperative score (calculated before surgery) aims to facilitate preventive strategies and the intraoperative score (calculated at the end of surgery) aims to facilitate mitigation strategies. Internal validation was achieved using bootstrapping. These data were pooled with data from 5 centers from the United States and the Netherlands (2007-2016) to assess discrimination and calibration in an internal-external validation procedure.

RESULTS

Overall, 1336 patients after DP were included, of whom 291 (22%) developed POPF. The preoperative distal fistula risk score (preoperative D-FRS) included 2 variables: pancreatic neck thickness [odds ratio: 1.14; 95% confidence interval (CI): 1.11-1.17 per mm increase] and pancreatic duct diameter (OR: 1.46; 95% CI: 1.32-1.65 per mm increase). The model performed well with an area under the receiver operating characteristic curve of 0.83 (95% CI: 0.78-0.88) and 0.73 (95% CI: 0.70-0.76) upon internal-external validation. Three risk groups were identified: low risk (<10%), intermediate risk (10%-25%), and high risk (>25%) for POPF with 238 (18%), 684 (51%), and 414 (31%) patients, respectively. The intraoperative risk score (intraoperative D-FRS) added body mass index, pancreatic texture, and operative time as variables with an area under the receiver operating characteristic curve of 0.80 (95% CI: 0.74-0.85).

CONCLUSIONS

The preoperative and the intraoperative D-FRS are the first validated risk scores for POPF after DP and are readily available at: http://www.pancreascalculator.com . The 3 distinct risk groups allow for personalized treatment and benchmarking.

Predicting patient-reported outcomes following lumbar spine surgery: development and external validation of multivariable prediction models

BACKGROUND

This study aimed to develop and externally validate prediction models of spinal surgery outcomes based on a retrospective review of a prospective clinical database, uniquely comparing multivariate regression and random forest (machine learning) approaches, and identifying the most important predictors.

METHODS

Outcomes were change in back and leg pain intensity and Core Outcome Measures Index (COMI) from baseline to the last available postoperative follow-up (3-24 months), defined as minimal clinically important change (MCID) and continuous change score. Eligible patients underwent lumbar spine surgery for degenerative pathology between 2011 and 2021. Data were split by surgery date into development (N = 2691) and validation (N = 1616) sets for temporal external validation. Multivariate logistic and linear regression, and random forest classification and regression models, were fit to the development data and validated on the external data.

RESULTS

All models demonstrated good calibration in the validation data. Discrimination ability (area under the curve) for MCID ranged from 0.63 (COMI) to 0.72 (back pain) in regression, and from 0.62 (COMI) to 0.68 (back pain) in random forests. The explained variation in continuous change scores spanned 16%-28% in linear, and 15%-25% in random forests regression. The most important predictors included age, baseline scores on the respective outcome measures, type of degenerative pathology, previous spinal surgeries, smoking status, morbidity, and duration of hospital stay.

CONCLUSIONS

The developed models appear robust and generalisable across different outcomes and modelling approaches but produced only borderline acceptable discrimination ability, suggesting the need to assess further prognostic factors. External validation showed no advantage of the random forest approach.

Prediction nomogram development and validation for postoperative radiotherapy in the management of oral squamous cell carcinoma

BACKGROUND

Predictive nomograms are useful tools to guide clinicians in estimating disease course. Oral squamous cell carcinoma (OSCC) patients would benefit from an interactive prediction calculator that defines their levels of survival-risk specific to their tumors to guide the use of postoperative radiotherapy (PORT).

METHODS

Patients with OSCC surgically treated with curative intent at four Head and Neck Cancer Centres were recruited retrospectively for development and validation of nomograms. Predictor variables include PORT, age, T and N classification, surgical margins, perineural invasion, and lymphovascular invasion. Outcomes were disease-free, disease-specific, and overall survivals over 5 years.

RESULTS

1296 patients with OSCC were in training cohort for nomogram analysis. Algorithms were developed to show relative benefit of PORT in survivals for higher-risk patients. External validation on 1212 patients found the nomogram to be robust with favorable discrimination and calibration.

CONCLUSION

The proposed calculator can assist clinicians and patients in the decision-making process for PORT.

Temporal validation of a multivariable surgical mortality prediction model (NZRisk): a New Zealand national cohort study

OBJECTIVES

Clinical risk calculators (CRCs), such as NZRisk, are used daily by clinicians to guide clinical decisions and explain individual risk to patients. The utility and robustness of these tools depends on the methods used to create the underlying mathematical model, as well as the stability of that model in relation to changing clinical practice and patient populations over time. The later should be checked by temporal validation using external data. Few if any of the clinical prediction models in current clinical use have published temporal validation. Here, we use a large external dataset to temporally validate NZRisk; a perioperative risk prediction model used in the New Zealand population.

METHODS

A sample of 1 976 362 adult non-cardiac surgical procedures collected over 15 years from the New Zealand Ministry of Health National Minimum Dataset, was used to temporally validate NZRisk. We divided the dataset into 15 single year cohorts and compared 13 of these to our NZRisk model (2 years used for the model building were excluded). We compared the area under the curve (AUC) value, calibration slope and intercept for each single year cohort, to the same values produced by the data used to create NZRisk, by fitting a random effects meta-regression with each year cohort acting as a separate study point. In addition, we used two-sided t-tests to compare each measure across the cohorts.

RESULTS

The AUC values for the 30-day NZRisk model applied to our single year cohorts ranged from 0.918 to 0.940 (NZRisk AUC was 0.921). There were eight statistically different AUC values for the following years 2007-2009, 2016 and 2018-2021. The intercept values ranged from -0.004 to 0.007 and 7 years had statistically significant different intercepts during leave-one-out t-tests; 2007-2010, 2012, 2018 and 2021. The slope values ranged from 0.72 to 1.12 and 7 years had statistically significant different slopes during leave-one-out t-tests; 2010, 2011, 2017, 2018 and 2019-2021. The random effects meta-regression upheld our results related to AUC (0.54 (95% CI 0.40 to 0.99), I² 67.57 (95% CI 40.67 to 88.50), Cochran's Q<0.001) and slope (τ 0.14 (95% CI 0.01 to 0.23), I² 98.61 (95% CI 97.31 to 99.50), Cochran's Q<0.001) between year difference.

CONCLUSION

The NZRisk model shows differences in AUC and slope but not intercept values over time. The biggest differences were in the calibration slope. The models maintained excellent discrimination over time as shown by the AUC values. These findings suggest we update our model in the next 5 years. To our knowledge, this is the first temporal validation of a CRC in current use.

External validation of the American prediction model for incident type 2 diabetes in the Iranian population

Background

The primary aim of the present study was to validate the REasons for Geographic and Racial Differences in Stroke (REGARDS) model for incident Type 2 diabetes (T2DM) in Iran.

Methods

Present study was a prospective cohort study on 1835 population aged ≥ 45 years from Tehran lipids and glucose study (TLGS).The predictors of REGARDS model based on Bayesian hierarchical techniques included age, sex, race, body mass index, systolic and diastolic blood pressures, triglycerides, high-density lipoprotein cholesterol, and fasting plasma glucose. For external validation, the area under the curve (AUC), sensitivity, specificity, Youden’s index, and positive and negative predictive values (PPV and NPV) were assessed.

Results

During the 10-year follow-up 15.3% experienced T2DM. The model showed acceptable discrimination (AUC (95%CI): 0.79 (0.76–0.82)), and good calibration. Based on the highest Youden’s index the suggested cut-point for the REGARDS probability would be ≥ 13% which yielded a sensitivity of 77.2%, specificity 66.8%, NPV 94.2%, and PPV 29.6%.

Conclusions

Our findings do support that the REGARDS model is a valid tool for incident T2DM in the Iranian population. Moreover, the probability value higher than the 13% cut-off point is stated to be significant for identifying those with incident T2DM.

Predicting the Absolute Risk of Ischemic Stroke in Asian Patients with Atrial Fibrillation: Comparing the COOL-AF Risk Score with CARS/mCARS Models for Absolute Risk and the CHA2DS2-VASc Score

BACKGROUND

The aims of this study were (1) to validate the CARS and mCARS methods in an Asian population with atrial fibrillation (AF) and (2) to compare the CARS and mCARS models for absolute risk using the COOL-AF method and CHA₂DS₂VASc scores for the prediction of ischemic stroke or systemic embolism (SSE).

METHODS

We analyzed the results from a prospective nationwide multicenter AF registry. Follow-up data were collected for 3 years. The main outcomes were SSE. Predictive models of the 3-year SSE of the COOL-AF model, the CHA₂DS₂VASc score, the CARS for the no-OAC group, and the mCARS for the OAC group were developed and evaluated by C-statistics, and calibration plots were created for the whole group, as well as for oral anticoagulant (OAC) users and no-OAC patients.

RESULTS

We studied 3405 patients (mean age: 67.8 years; 58.2% male, 75.4% OAC). The incidence rates of SSE were 1.51 (1.26-1.78), 1.93 (1.39-2.60), and 1.37 (1.10-1.68) for all patients, no-OAC patients, and OAC patients, respectively. For the whole population, the COOL-AF score had a C-statistic of 0.697 (0.682-0.713), which was superior to the CHA₂DS₂-VASc [0.655 (0.639-0.671)]. For the no-OAC group, the CARS predicted SSE with a C-statistic of 0.685 (0.652-0.716), which was similar to the CHA₂DS₂-VASc [0.684 (0.651-0.7150] and COOL-AF models [0.692 (0.659-0.723)]. For the OAC group, the mCARS had a C-statistic of 0.687 (0.669-0.705) that was similar to the COOL-AF [0.704 (0.686-0.721)] and better than the CHA₂DS₂-VASc score [0.655 (0.637-0.674)].

CONCLUSIONS

The calculation of the individual absolute risks using the CARS and mCARS models can predict SSE in an Asian population. Small differences were evident between the COOL-AF and CHA₂DS₂-VASc scores.

Clinical prediction model for prognosis in kidney transplant recipients (KIDMO): study protocol

BACKGROUND

Many potential prognostic factors for predicting kidney transplantation outcomes have been identified. However, in Switzerland, no widely accepted prognostic model or risk score for transplantation outcomes is being routinely used in clinical practice yet. We aim to develop three prediction models for the prognosis of graft survival, quality of life, and graft function following transplantation in Switzerland.

METHODS

The clinical kidney prediction models (KIDMO) are developed with data from a national multi-center cohort study (Swiss Transplant Cohort Study; STCS) and the Swiss Organ Allocation System (SOAS). The primary outcome is the kidney graft survival (with death of recipient as competing risk); the secondary outcomes are the quality of life (patient-reported health status) at 12 months and estimated glomerular filtration rate (eGFR) slope. Organ donor, transplantation, and recipient-related clinical information will be used as predictors at the time of organ allocation. We will use a Fine & Gray subdistribution model and linear mixed-effects models for the primary and the two secondary outcomes, respectively. Model optimism, calibration, discrimination, and heterogeneity between transplant centres will be assessed using bootstrapping, internal-external cross-validation, and methods from meta-analysis.

DISCUSSION

Thorough evaluation of the existing risk scores for the kidney graft survival or patient-reported outcomes has been lacking in the Swiss transplant setting. In order to be useful in clinical practice, a prognostic score needs to be valid, reliable, clinically relevant, and preferably integrated into the decision-making process to improve long-term patient outcomes and support informed decisions for clinicians and their patients. The state-of-the-art methodology by taking into account competing risks and variable selection using expert knowledge is applied to data from a nationwide prospective multi-center cohort study. Ideally, healthcare providers together with patients can predetermine the risk they are willing to accept from a deceased-donor kidney, with graft survival, quality of life, and graft function estimates available for their consideration.

STUDY REGISTRATION

Open Science Framework ID: z6mvj.