Prognostic Models in Medicine

Prognostic models of in-hospital mortality of intensive care patients using neural representation of unstructured text: A systematic review and critical appraisal

OBJECTIVE

To review and critically appraise published and preprint reports of prognostic models of in-hospital mortality of patients in the intensive-care unit (ICU) based on neural representations (embeddings) of clinical notes.

METHODS

PubMed and arXiv were searched up to August 1, 2022. At least two reviewers independently selected the studies that developed a prognostic model of in-hospital mortality of intensive-care patients using free-text represented as embeddings and extracted data using the CHARMS checklist. Risk of bias was assessed using PROBAST. Reporting on the model was assessed with the TRIPOD guideline. To assess the machine learning components that were used in the models, we present a new descriptive framework based on different techniques to represent text and provide predictions from text. The study protocol was registered in the PROSPERO database (CRD42022354602).

RESULTS

Eighteen studies out of 2,825 were included. All studies used the publicly-available MIMIC dataset. Context-independent word embeddings are widely used. Model discrimination was provided by all studies (AUROC 0.75-0.96), but measures of calibration were scarce. Seven studies used both structural clinical variables and notes. Model discrimination improved when adding clinical notes to variables. None of the models was externally validated and often a simple train/test split was used for internal validation. Our critical appraisal demonstrated a high risk of bias in all studies and concerns regarding their applicability in clinical practice.

CONCLUSION

All studies used a neural architecture for prediction and were based on one publicly available dataset. Clinical notes were reported to improve predictive performance when used in addition to only clinical variables. Most studies had methodological, reporting, and applicability issues. We recommend reporting both model discrimination and calibration, using additional data sources, and using more robust evaluation strategies, including prospective and external validation. Finally, sharing data and code is encouraged to improve study reproducibility.

Development and validation of prognostic models for anal cancer outcomes using distributed learning: protocol for the international multi-centre atomCAT2 study

BACKGROUND

Anal cancer is a rare cancer with rising incidence. Despite the relatively good outcomes conferred by state-of-the-art chemoradiotherapy, further improving disease control and reducing toxicity has proven challenging. Developing and validating prognostic models using routinely collected data may provide new insights for treatment development and selection. However, due to the rarity of the cancer, it can be difficult to obtain sufficient data, especially from single centres, to develop and validate robust models. Moreover, multi-centre model development is hampered by ethical barriers and data protection regulations that often limit accessibility to patient data. Distributed (or federated) learning allows models to be developed using data from multiple centres without any individual-level patient data leaving the originating centre, therefore preserving patient data privacy. This work builds on the proof-of-concept three-centre atomCAT1 study and describes the protocol for the multi-centre atomCAT2 study, which aims to develop and validate robust prognostic models for three clinically important outcomes in anal cancer following chemoradiotherapy.

METHODS

This is a retrospective multi-centre cohort study, investigating overall survival, locoregional control and freedom from distant metastasis after primary chemoradiotherapy for anal squamous cell carcinoma. Patient data will be extracted and organised at each participating radiotherapy centre (n = 18). Candidate prognostic factors have been identified through literature review and expert opinion. Summary statistics will be calculated and exchanged between centres prior to modelling. The primary analysis will involve developing and validating Cox proportional hazards models across centres for each outcome through distributed learning. Outcomes at specific timepoints of interest and factor effect estimates will be reported, allowing for outcome prediction for future patients.

DISCUSSION

The atomCAT2 study will analyse one of the largest available cross-institutional cohorts of patients with anal cancer treated with chemoradiotherapy. The analysis aims to provide information on current international clinical practice outcomes and may aid the personalisation and design of future anal cancer clinical trials through contributing to a better understanding of patient risk stratification.

The GERtality Score: The Development of a Simple Tool to Help Predict in-Hospital Mortality in Geriatric Trauma Patients

Feasible and predictive scoring systems for severely injured geriatric patients are lacking. Therefore, the aim of this study was to develop a scoring system for the prediction of in-hospital mortality in severely injured geriatric trauma patients. The TraumaRegister DGU^® (TR-DGU) was utilized. European geriatric patients (≥65 years) admitted between 2008 and 2017 were included. Relevant patient variables were implemented in the GERtality score. By conducting a receiver operating characteristic (ROC) analysis, a comparison with the Geriatric Trauma Outcome Score (GTOS) and the Revised Injury Severity Classification II (RISC-II) Score was performed. A total of 58,055 geriatric trauma patients (mean age: 77 years) were included. Univariable analysis led to the following variables: age ≥ 80 years, need for packed red blood cells (PRBC) transfusion prior to intensive care unit (ICU), American Society of Anesthesiologists (ASA) score ≥ 3, Glasgow Coma Scale (GCS) ≤ 13, Abbreviated Injury Scale (AIS) in any body region ≥ 4. The maximum GERtality score was 5 points. A mortality rate of 72.4% was calculated in patients with the maximum GERtality score. Mortality rates of 65.1 and 47.5% were encountered in patients with GERtality scores of 4 and 3 points, respectively. The area under the curve (AUC) of the novel GERtality score was 0.803 (GTOS: 0.784; RISC-II: 0.879). The novel GERtality score is a simple and feasible score that enables an adequate prediction of the probability of mortality in polytraumatized geriatric patients by using only five specific parameters.

Radiomics-based machine learning model to predict risk of death within 5-years in clear cell renal cell carcinoma patients

PURPOSE

The aim of this study was to develop radiomics-based machine learning models based on extracted radiomic features and clinical information to predict the risk of death within 5 years for prognosis of clear cell renal cell carcinoma (ccRCC) patients.

METHODS

According to image quality and clinical data availability, we eventually selected 70 ccRCC patients that underwent CT scans. Manual volume-of-interest (VOI) segmentation of each image was performed by an experienced radiologist using the 3D slicer software package. Prior to feature extraction, image pre-processing was performed on CT images to extract different image features, including wavelet, Laplacian of Gaussian, and resampling of the intensity values to 32, 64 and 128 bin levels. Overall, 2544 3D radiomics features were extracted from each VOI for each patient. Minimum Redundancy Maximum Relevance (MRMR) algorithm was used as feature selector. Four classification algorithms were used, including Generalized Linear Model (GLM), Support Vector Machine (SVM), K-nearest Neighbor (KNN) and XGBoost. We used the Bootstrap resampling method to create validation sets. Area under the receiver operating characteristic (ROC) curve (AUROC), accuracy, sensitivity, and specificity were used to assess the performance of the classification models.

RESULTS

The best single performance among 8 different models was achieved by the XGBoost model using a combination of radiomic features and clinical information (AUROC, accuracy, sensitivity, and specificity with 95% confidence interval were 0.95-0.98, 0.93-0.98, 0.93-0.96 and ~1.0, respectively).

CONCLUSIONS

We developed a robust radiomics-based classifier that is capable of accurately predicting overall survival of RCC patients for prognosis of ccRCC patients. This signature may help identifying high-risk patients who require additional treatment and follow up regimens.

Survivability modelling using Bayesian network for patients with first and secondary primary cancers

BACKGROUND AND OBJECTIVE

Multiple primary cancers significantly threat patient survivability. Predicting the survivability of patients with two cancers is challenging because its stochastic pattern relates with numerous variables.

METHODS

In this study, a Bayesian network (BN) model was proposed to describe the occurrence of two primary cancers and predict the five-year survivability of patients using probabilistic evidence. Eleven types of major primary cancers and contingent occurrences of secondary cancers were investigated. A nationwide two-cancer database involving 7,845 patients in Taiwan was investigated. The BN topology is rigorously examined and imbalanced dataset is processed by the synthetic minority oversampling technique. The proposed BN survivability prognosis model was compared with benchmark approaches.

RESULTS

The proposed model significantly outperformed the back-propagation neural network, logistic regression, support vector machine, and naïve Bayes in terms of sensitivity, which is a critical performance index for the non-survival group.

CONCLUSIONS

Using the proposed BN model, one can estimate the posterior probabilities for every query provided appropriate prior evidences. The potential survivability information of patients, treatment effects, and socio-demographics factor effects predicted by the proposed model can help in cancer treatment assessment and cancer development monitoring.

Functional Outcome Prediction in Ischemic Stroke: A Comparison of Machine Learning Algorithms and Regression Models

Background and Purpose: Stroke-related functional risk scores are used to predict patients' functional outcomes following a stroke event. We evaluate the predictive accuracy of machine-learning algorithms for predicting functional outcomes in acute ischemic stroke patients after endovascular treatment.

Methods: Data were from the Precise and Rapid Assessment of Collaterals with Multi-phase CT Angiography (PROVE-IT), an observational study of 614 ischemic stroke patients. Regression and machine learning models, including random forest (RF), classification and regression tree (CART), C5.0 decision tree (DT), support vector machine (SVM), adaptive boost machine (ABM), least absolute shrinkage and selection operator (LASSO) logistic regression, and logistic regression models were used to train and predict the 90-day functional impairment risk, which is measured by the modified Rankin scale (mRS) score > 2. The models were internally validated using split-sample cross-validation and externally validated in the INTERRSeCT cohort study. The accuracy of these models was evaluated using the area under the receiver operating characteristic curve (AUC), Matthews Correlation Coefficient (MCC), and Brier score.

Results: Of the 614 patients included in the training data, 249 (40.5%) had 90-day functional impairment (i.e., mRS > 2). The median and interquartile range (IQR) of age and baseline NIHSS scores were 77 years (IQR = 69–83) and 17 (IQR = 11–22), respectively. Both logistic regression and machine learning models had comparable predictive accuracy when validated internally (AUC range = [0.65–0.72]; MCC range = [0.29–0.42]) and externally (AUC range = [0.66–0.71]; MCC range = [0.34–0.42]).

Conclusions: Machine learning algorithms and logistic regression had comparable predictive accuracy for predicting stroke-related functional impairment in stroke patients.

Medical idioms for clinical Bayesian network development

Bayesian Networks (BNs) are graphical probabilistic models that have proven popular in medical applications. While numerous medical BNs have been published, most are presented fait accompli without explanation of how the network structure was developed or justification of why it represents the correct structure for the given medical application. This means that the process of building medical BNs from experts is typically ad hoc and offers little opportunity for methodological improvement. This paper proposes generally applicable and reusable medical reasoning patterns to aid those developing medical BNs. The proposed method complements and extends the idiom-based approach introduced by Neil, Fenton, and Nielsen in 2000. We propose instances of their generic idioms that are specific to medical BNs. We refer to the proposed medical reasoning patterns as medical idioms. In addition, we extend the use of idioms to represent interventional and counterfactual reasoning. We believe that the proposed medical idioms are logical reasoning patterns that can be combined, reused and applied generically to help develop medical BNs. All proposed medical idioms have been illustrated using medical examples on coronary artery disease. The method has also been applied to other ongoing BNs being developed with medical experts. Finally, we show that applying the proposed medical idioms to published BN models results in models with a clearer structure.

Predicting seizure freedom after epilepsy surgery, a challenge in clinical practice

OBJECTIVE

The objective of this study was to compare the accuracy of clinical judgment in predicting seizure outcome after resective epilepsy surgery relative to two recently published statistical tools [the Epilepsy Surgery Nomogram (ESN) and the modified Seizure-Freedom score (m-SFS)].

METHODS

Details of presurgical evaluations of 20 patients who underwent epilepsy surgery were presented to 20 epilepsy experts. The final surgical treatment was also disclosed. The clinicians were asked to predict the likelihood of a good outcome (Engel 1) at 2 and 5 years in each case. The ESN and the m-SFS predictions were calculated with the data provided to the clinicians. The discriminative ability of clinical judgment, ESN, and m-SFS was assessed by calculating a concordance index (C-index). Expert opinion, the m-SFS and the ESN performances were compared using a Receiver Operating Characteristic (ROC) curve analysis.

RESULTS

The mean age at surgery was 29 years (standard deviation [SD] = 14); 40% were male; 70% were right-handed, and thirteen (65%) had an Engel outcome 1 at 2 and 5 years. The mean C-index for the mean physician's prediction was 0.478 with a variance of 0.012. The ESN had an area under the curve (AUC) of 0.528 and 0.533 for the 2-year and 5-year predictions in comparison with the clinicians' predictions that was 0.476, and 0.466, respectively. For the m-SFS, the AUC at 2 years and 5 years was 0.539 and 0.539, respectively. No statistical difference was noted between the ESN and the clinicians or between m-SFS and the ESN, but there is a moderate statistical difference favoring the m-SFS to the clinicians (p 0.0960 and 0.0514, for 2 and 5 years).

SIGNIFICANCE

Clinical judgment was not superior to the ESN nor to the m-SFS. Together with the interphysician's prediction variability, our findings reinforce the need for better tools to predict postoperative outcomes.

[Steps to personalized therapy of multiple sclerosis: predicting safety of treatment using mathematical modeling]

AIM

To construct a mathematical model capable of predicting the drug safety of a patient receiving multiple sclerosis disease modifying drugs (DMD), on a model of flu-like syndrome (FLS).

MATERIAL AND METHODS

The study included 457 patients with multiple sclerosis (MS), aged from 18 to 68 years, mean 38.79 years, the mean duration of disease 122.58 months. All patients received first-line injections drug (interferon-beta). The sample included data from a three-year prospective dynamic observation with a frequency of observation of 1 every 6 months, with only the data of those examinations for which the presence or absence of FLS was known for the next 6 months (1203 cases). At the first step, the frequency of factors in the compared groups using the W Wald-Wolkovitz test, then the prognostic coefficients (PC) and the Kulbak informativity coefficient (CI) were calculated for each factor gradation. To determine the predictive ability of signs, the Spearman's R criterion was used. At the second step, a model of a two-layer neural network was constructed based on the data obtained.

RESULTS

A simple static model and algorithm were developed to assess the risks of the onset and persistence of FLS during the next 6 months of interferon beta therapy. An attempt was also made to create an active model using neural network technology. Both models showed good sensitivity and specificity - 81.2% and 80.6% for the neural network, and 73.4 and 71.6% for the static model.

CONCLUSION

Using of these algorithms allows to significantly increase the possibility of predicting the occurrence of AE at the time of drug prescribing. From the mathematical point of view, for the first time the mechanism and possibilities of using a neural network in conditions of incomplete initial information were determined.

A Bayesian Network Interpretation of the Cox's Proportional Hazard Model

Cox's proportional hazards (CPH) model is quite likely the most popular modeling technique in survival analysis. While the CPH model is able to represent a relationship between a collection of risks and their common effect, Bayesian networks have become an attractive alternative with an increased modeling power and far broader applications. Our paper focuses on a Bayesian network interpretation of the CPH model (BN-Cox). We provide a method of encoding knowledge from existing CPH models in the process of knowledge engineering for Bayesian networks. This is important because in practice we often have CPH models available in the literature and no access to the original data from which they have been derived. We compare the accuracy of the resulting BN-Cox model to the original CPH model, Kaplan-Meier estimate, and Bayesian networks learned from data, including Naive Bayes, Tree Augmented Naive Bayes, Noisy-Max, and parameter learning by means of the EM algorithm. BN-Cox model came out as the most accurate of all BN approaches and very close to the original CPH model. We study two approaches for simplifying the BN-Cox model for the sake of representational and computational efficiency: (1) parent divorcing and (2) removing less important risk factors. We show that removing less important risk factors leads to smaller loss of accuracy.

Dynamic models to predict health outcomes: current status and methodological challenges

BACKGROUND

Disease populations, clinical practice, and healthcare systems are constantly evolving. This can result in clinical prediction models quickly becoming outdated and less accurate over time. A potential solution is to develop 'dynamic' prediction models capable of retaining accuracy by evolving over time in response to observed changes. Our aim was to review the literature in this area to understand the current state-of-the-art in dynamic prediction modelling and identify unresolved methodological challenges.

METHODS

MEDLINE, Embase and Web of Science were searched for papers which used or developed dynamic clinical prediction models. Information was extracted on methods for model updating, choice of update windows and decay factors and validation of models. We also extracted reported limitations of methods and recommendations for future research.

RESULTS

We identified eleven papers that discussed seven dynamic clinical prediction modelling methods which split into three categories. The first category uses frequentist methods to update models in discrete steps, the second uses Bayesian methods for continuous updating and the third, based on varying coefficients, explicitly describes the relationship between predictors and outcome variable as a function of calendar time. These methods have been applied to a limited number of healthcare problems, and few empirical comparisons between them have been made.

CONCLUSION

Dynamic prediction models are not well established but they overcome one of the major issues with static clinical prediction models, calibration drift. However, there are challenges in choosing decay factors and in dealing with sudden changes. The validation of dynamic prediction models is still largely unexplored terrain.

Risk analysis for the reintroduction and transmission of measles in the post-elimination period in the Americas

Objective.

To propose and test a model for analyzing municipalities’ level of risk of reintroduction and transmission of the measles virus in the post-elimination period in the Americas.

Methods.

An ecological-analytical study was conducted using data on the measles epidemic that occurred in 2013–2015 in northeastern Brazil. The variables for analysis were selected after an extensive review of scientific literature on the risk of importation of measles cases. A univariate analysis considering the presence or absence of confirmed cases of measles in 184 municipalities in the state of Ceará, Brazil, was carried out to evaluate the association between the dependent variable and 23 independent variables, grouped into four categories: 1) characteristics of the municipalities; 2) quality indicators for immunization programs and epidemiological surveillance; 3) organizational structure for the public health response; and 4) selected impact indicators. A P value < 0.05 was considered significant. All variables with P < 0.200 were analyzed using multivariate logistic regression. Based on the results, the municipalities were categorized by four levels of risk (“low,” “medium,” “high,” and “very high”).

Results.

The model sensitivity was 95% for concordance between municipalities classified as “high risk” and “very high risk” and those that had an epidemic between 2013 and 2015 in Ceará. Of the 38 municipalities that had an epidemic, 76% (29/38) were classified as “high risk” and “very high risk”; 146 municipalities did not report cases (P < 0.0002).

Conclusions.

Given the imminent risk of reintroduction of measles circulation in the post-elimination period in the Americas, this model may be useful in identifying areas at greater risk for reintroduction and continued transmission of measles. Knowledge of vulnerable areas could trigger appropriate surveillance and monitoring to prevent sustained transmission.

Performance of Firth-and logF-type penalized methods in risk prediction for small or sparse binary data

BACKGROUND

When developing risk models for binary data with small or sparse data sets, the standard maximum likelihood estimation (MLE) based logistic regression faces several problems including biased or infinite estimate of the regression coefficient and frequent convergence failure of the likelihood due to separation. The problem of separation occurs commonly even if sample size is large but there is sufficient number of strong predictors. In the presence of separation, even if one develops the model, it produces overfitted model with poor predictive performance. Firth-and logF-type penalized regression methods are popular alternative to MLE, particularly for solving separation-problem. Despite the attractive advantages, their use in risk prediction is very limited. This paper evaluated these methods in risk prediction in comparison with MLE and other commonly used penalized methods such as ridge.

METHODS

The predictive performance of the methods was evaluated through assessing calibration, discrimination and overall predictive performance using an extensive simulation study. Further an illustration of the methods were provided using a real data example with low prevalence of outcome.

RESULTS

The MLE showed poor performance in risk prediction in small or sparse data sets. All penalized methods offered some improvements in calibration, discrimination and overall predictive performance. Although the Firth-and logF-type methods showed almost equal amount of improvement, Firth-type penalization produces some bias in the average predicted probability, and the amount of bias is even larger than that produced by MLE. Of the logF(1,1) and logF(2,2) penalization, logF(2,2) provides slight bias in the estimate of regression coefficient of binary predictor and logF(1,1) performed better in all aspects. Similarly, ridge performed well in discrimination and overall predictive performance but it often produces underfitted model and has high rate of convergence failure (even the rate is higher than that for MLE), probably due to the separation problem.

CONCLUSIONS

The logF-type penalized method, particularly logF(1,1) could be used in practice when developing risk model for small or sparse data sets.

Probabilistic modeling of short survivability in patients with brain metastasis from lung cancer

The prediction of substantially short survivability in patients is extremely risky. In this study, we proposed a probabilistic model using Bayesian network (BN) to predict the short survivability of patients with brain metastasis from lung cancer. A nationwide cancer patient database from 1996 to 2010 in Taiwan was used. The cohort consisted of 438 patients with brain metastasis from lung cancer. We utilized synthetic minority over-sampling technique (SMOTE) to solve the imbalanced property embedded in the problem. The proposed BN was compared with three competitive models, namely, naive Bayes (NB), logistic regression (LR), and support vector machine (SVM). Statistical analysis showed that performances of BN, LR, NB, and SVM were statistically the same in terms of all indices with low sensitivity when these models were applied on an imbalanced data set. Results also showed that SMOTE can improve the performance of the four models in terms of sensitivity, while keeping high accuracy and specificity. Further, the proposed BN is more effective as compared with NB, LR, and SVM from two perspectives: the transparency and ability to show the relation of factors affecting brain metastasis from lung cancer; it allows decision makers to find the probability despite incomplete evidence and information; and the sensitivity of the proposed BN is the highest among all standard machine learning methods.

An alternative view on the task of prognosis

Prognosis is central to clinical medical practice. In oncology, an accurate prognosis is a key requirement for making the right therapeutical decisions. Prognosis is even more important when medicine extends its predictive capacities using genetic data. Based on statistics and probability, indices, survival curves and prognostic scores are established. Intending to be an objective and neutral description of reality, this kind of prognosis and the corresponding practices of medical care however carry and imply a particular conception of human life and destiny. Medical Humanities can help to develop reflexivity regarding prognostic practices. This paper intends to clarify which particular assumptions are implied in the current way of doing prognosis in medicine, and highlights its advantages and limitations. Prognosis not only describes but also affects present and future patient experience. An alternative view on the task of prognosis is then developed with a broader understanding of the prognostic act. Based on a phenomenology of time, distinguishing between a quantitative chronological understanding of time (chronos) and an experienced qualitative time (kairos), the article contrasts a conception of the prognosis as forecast ('probabilistic prognosis') with a conception of prognosis as perspective ('hermeneutic prognosis'). In probabilistic prognosis, the future is seen as something that can be read from presently accessible signs, as something that we can anticipate within ranges of uncertainty. The patient's lifetime, which has been open toward the future, becomes closed through this kind of prognosis. In a hermeneutic conception of prognosis as a perspective on the future, which is thought of as an open space of possibilities, the future as not-yet is what can be envisioned without being known. A hermeneutic approach emphasizes the meanings of the experience of illness for the patient, helps to improve the practice of prognosis today, and offers caregivers and patients an opportunity for living better.

Aggregating published prediction models with individual participant data: a comparison of different approaches

During the recent decades, interest in prediction models has substantially increased, but approaches to synthesize evidence from previously developed models have failed to keep pace. This causes researchers to ignore potentially useful past evidence when developing a novel prediction model with individual participant data (IPD) from their population of interest. We aimed to evaluate approaches to aggregate previously published prediction models with new data. We consider the situation that models are reported in the literature with predictors similar to those available in an IPD dataset. We adopt a two-stage method and explore three approaches to calculate a synthesis model, hereby relying on the principles of multivariate meta-analysis. The former approach employs a naive pooling strategy, whereas the latter accounts for within-study and between-study covariance. These approaches are applied to a collection of 15 datasets of patients with traumatic brain injury, and to five previously published models for predicting deep venous thrombosis. Here, we illustrated how the generally unrealistic assumption of consistency in the availability of evidence across included studies can be relaxed. Results from the case studies demonstrate that aggregation yields prediction models with an improved discrimination and calibration in a vast majority of scenarios, and result in equivalent performance (compared with the standard approach) in a small minority of situations. The proposed aggregation approaches are particularly useful when few participant data are at hand. Assessing the degree of heterogeneity between IPD and literature findings remains crucial to determine the optimal approach in aggregating previous evidence into new prediction models.

Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessment

Purpose

The aim of our study was to explore, using an innovative method, the effect of temporal changes in the mortality prediction performance of an existing model on the quality of care assessment. The prognostic model (rSAPS-II) was a recalibrated Simplified Acute Physiology Score-II model developed for very elderly Intensive Care Unit (ICU) patients.

Methods

The study population comprised all 12,143 consecutive patients aged 80 years and older admitted between January 2004 and July 2009 to one of the ICUs of 21 Dutch hospitals. The prospective dataset was split into 30 equally sized consecutive subsets. Per subset, we measured the model’s discrimination [area under the curve (AUC)], accuracy (Brier score), and standardized mortality ratio (SMR), both without and after repeated recalibration. All performance measures were considered to be stable if <2 consecutive points fell outside the green zone [mean ± 2 standard deviation (SD)] and none fell outside the yellow zone (mean ± 4SD) of pre-control charts. We compared proportions of hospitals with SMR>1 without and after repeated recalibration for the year 2009.

Results

For all subsets, the AUCs were stable, but the Brier scores and SMRs were not. The SMR was downtrending, achieving levels significantly below 1. Repeated recalibration rendered it stable again. The proportions of hospitals with SMR>1 and SMR<1 changed from 15 versus 85% to 35 versus 65%.

Conclusions

Variability over time may markedly vary among different performance measures, and infrequent model recalibration can result in improper assessment of the quality of care in many hospitals. We stress the importance of the timely recalibration and repeated validation of prognostic models over time.

Electronic supplementary material

The online version of this article (doi:10.1007/s00134-011-2390-2) contains supplementary material, which is available to authorized users.

Statistical process control for validating a classification tree model for predicting mortality--a novel approach towards temporal validation

Prediction models are postulated as useful tools to support tasks such as clinical decision making and benchmarking. In particular, classification tree models have enjoyed much interest in the Biomedical Informatics literature. However, their prospective predictive performance over the course of time has not been investigated. In this paper we suggest and apply statistical process control methods to monitor over more than 5 years the prospective predictive performance of TM80+, one of the few classification-tree models published in the clinical literature. TM80+ is a model for predicting mortality among very elderly patients in the intensive care based on a multi-center dataset. We also inspect the predictive performance at the tree's leaves. This study provides important insights into patterns of (in)stability of the tree's performance and its "shelf life". The study underlies the importance of continuous validation of prognostic models over time using statistical tools and the timely recalibration of tree models.

Prediction of clinical conditions after coronary bypass surgery using dynamic data analysis

This work studies the impact of using dynamic information as features in a machine learning algorithm for the prediction task of classifying critically ill patients in two classes according to the time they need to reach a stable state after coronary bypass surgery: less or more than 9 h. On the basis of five physiological variables (heart rate, systolic arterial blood pressure, systolic pulmonary pressure, blood temperature and oxygen saturation), different dynamic features were extracted, namely the means and standard deviations at different moments in time, coefficients of multivariate autoregressive models and cepstral coefficients. These sets of features served subsequently as inputs for a Gaussian process and the prediction results were compared with the case where only admission data was used for the classification. The dynamic features, especially the cepstral coefficients (aROC: 0.749, Brier score: 0.206), resulted in higher performances when compared to static admission data (aROC: 0.547, Brier score: 0.247). The differences in performance are shown to be significant. In all cases, the Gaussian process classifier outperformed to logistic regression.

Learning patient-specific predictive models from clinical data

We introduce an algorithm for learning patient-specific models from clinical data to predict outcomes. Patient-specific models are influenced by the particular history, symptoms, laboratory results, and other features of the patient case at hand, in contrast to the commonly used population-wide models that are constructed to perform well on average on all future cases. The patient-specific algorithm uses Markov blanket (MB) models, carries out Bayesian model averaging over a set of models to predict the outcome for the patient case at hand, and employs a patient-specific heuristic to locate a set of suitable models to average over. We evaluate the utility of using a local structure representation for the conditional probability distributions in the MB models that captures additional independence relations among the variables compared to the typically used representation that captures only the global structure among the variables. In addition, we compare the performance of Bayesian model averaging to that of model selection. The patient-specific algorithm and its variants were evaluated on two clinical datasets for two outcomes. Our results provide support that the performance of an algorithm for learning patient-specific models can be improved by using a local structure representation for MB models and by performing Bayesian model averaging.

A tree-based decision model to support prediction of the severity of asthma exacerbations in children

This paper describes the development of a tree-based decision model to predict the severity of pediatric asthma exacerbations in the emergency department (ED) at 2 h following triage. The model was constructed from retrospective patient data abstracted from the ED charts. The original data was preprocessed to eliminate questionable patient records and to normalize values of age-dependent clinical attributes. The model uses attributes routinely collected in the ED and provides predictions even for incomplete observations. Its performance was verified on independent validating data (split-sample validation) where it demonstrated AUC (area under ROC curve) of 0.83, sensitivity of 84%, specificity of 71% and the Brier score of 0.18. The model is intended to supplement an asthma clinical practice guideline, however, it can be also used as a stand-alone decision tool.

Discovery and integration of univariate patterns from daily individual organ-failure scores for intensive care mortality prediction

OBJECTIVES

The current established mortality predictive models in the intensive care rely only on patient information gathered within the first 24 hours of admission. Recent research demonstrated the added prognostic value residing in the sequential organ-failure assessment (SOFA) score which quantifies on each day the cumulative patient organ derangement. The objective of this paper is to develop and study predictive models that also incorporate univariate patterns of the six individual organ systems underlining the SOFA score. A model for a given day d predicts the probability of in-hospital mortality.

MATERIALS AND METHODS

We use the logistic framework to combine a summary statistic of the historic SOFA information for a patient together with selected dummy variables indicating the occurrence of univariate frequent temporal patterns of individual organ system functioning. We demonstrate the application of our method to a large real-life data set from an intensive care unit (ICU) in a teaching hospital. Model performance is tested in terms of the AUC and the Brier score.

RESULTS

An algorithm for categorization, discovery, and selection of univariate patterns of individual organ scores and the induction of predictive models. The case-study resulted in six daily models corresponding to days 2-7. Their AUC ranged between 0.715 and 0.794 and the Brier scores between 0.161 and 0.216. Models using only admission data but recalibrated for days 2-7 generated AUC ranging between 0.643 and 0.761 and Brier scores ranged between 0.175 and 0.230.

CONCLUSIONS

The results show that temporal organ-failure episodes improve predictions' quality in terms of both discrimination and calibration. In addition, they enhance the interpretability of models. Our approach should be applicable to many other medical domains where severity scores and sub-scores are collected.

Dynamic Bayesian networks as prognostic models for clinical patient management

Prognostic models in medicine are usually been built using simple decision rules, proportional hazards models, or Markov models. Dynamic Bayesian networks (DBNs) offer an approach that allows for the incorporation of the causal and temporal nature of medical domain knowledge as elicited from domain experts, thereby allowing for detailed prognostic predictions. The aim of this paper is to describe the considerations that must be taken into account when constructing a DBN for complex medical domains and to demonstrate their usefulness in practice. To this end, we focus on the construction of a DBN for prognosis of carcinoid patients, compare performance with that of a proportional hazards model, and describe predictions for three individual patients. We show that the DBN can make detailed predictions, about not only patient survival, but also other variables of interest, such as disease progression, the effect of treatment, and the development of complications. Strengths and limitations of our approach are discussed and compared with those offered by traditional methods.

Discovery and inclusion of SOFA score episodes in mortality prediction

Predicting the survival status of Intensive Care patients at the end of their hospital stay is useful for various clinical and organizational tasks. Current models for predicting mortality use logistic regression models that rely solely on data collected during the first 24h of patient admission. These models do not exploit information contained in daily organ failure scores which nowadays are being routinely collected in many Intensive Care Units. We propose a novel method for mortality prediction that, in addition to admission-related data, takes advantage of daily data as well. The method is characterized by the data-driven discovery of temporal patterns, called episodes, of the organ failure scores and by embedding them in the familiar logistic regression framework for prediction. Our method results in a set of D logistic regression models, one for each of the first D days of Intensive Care Unit stay. A model for day d<or=D is trained on the patient subpopulation that stayed at least d days in the Intensive Care Unit and predicts the probability of death at the end of hospital stay for such patients. We implemented our method, with a specific form of episodes, called aligned episodes, on a large dataset of Intensive Care Unit patients for the first 5 days of stay (D=5) in the unit. We compared our models with ones that were developed on the same patient subpopulations but which did not use the episodes. The new models show improved performance on each of the five days. They also provide insight in the effect of the various selected episodes on mortality.

Application of the Naïve Bayesian Classifier to optimize treatment decisions

BACKGROUND AND PURPOSE

To study the accuracy, specificity and sensitivity of the Naïve Bayesian Classifier (NBC) in the assessment of individual risk of cancer relapse or progression after radiotherapy (RT).

MATERIALS AND METHODS

Data of 142 brain tumour patients irradiated from 2000 to 2005 were analyzed. Ninety-six attributes related to disease, patient and treatment were chosen. Attributes in binary form consisted of the training set for NBC learning. NBC calculated an individual conditional probability of being assigned to: relapse or progression (1), or no relapse or progression (0) group. Accuracy, attribute selection and quality of classifier were determined by comparison with actual treatment results, leave-one-out and cross validation methods, respectively. Clinical setting test utilized data of 35 patients. Treatment results at classification were unknown and were compared with classification results after 3 months.

RESULTS

High classification accuracy (84%), specificity (0.87) and sensitivity (0.80) were achieved, both for classifier training and in progressive clinical evaluation.

CONCLUSIONS

NBC is a useful tool to support the assessment of individual risk of relapse or progression in patients diagnosed with brain tumour undergoing RT postoperatively.

Temporal abstraction for feature extraction: a comparative case study in prediction from intensive care monitoring data

OBJECTIVES

To compare two temporal abstraction procedures for the extraction of meta features from monitoring data. Feature extraction prior to predictive modeling is a common strategy in prediction from temporal data. A fundamental dilemma in this strategy, however, is the extent to which the extraction should be guided by domain knowledge, and to which extent it should be guided by the available data. The two temporal abstraction procedures compared in this case study differ in this respect.

METHODS AND MATERIAL

The first temporal abstraction procedure derives symbolic descriptions from the data that are predefined using existing concepts from the medical language. In the second procedure, a large space of numerical meta features is searched through to discover relevant features from the data. These procedures were applied to a prediction problem from intensive care monitoring data. The predictive value of the resulting meta features were compared, and based on each type of features, a class probability tree model was developed.

RESULTS

The numerical meta features extracted by the second procedure were found to be more informative than the symbolic meta features of the first procedure in the case study, and a superior predictive performance was observed for the associated tree model.

CONCLUSION

The findings indicate that for prediction from monitoring data, induction of numerical meta features from data is preferable to extraction of symbolic meta features using existing clinical concepts.

Prognostic Bayesian networks I: rationale, learning procedure, and clinical use

Prognostic models are tools to predict the future outcome of disease and disease treatment, one of the fundamental tasks in clinical medicine. This article presents the prognostic Bayesian network (PBN) as a new type of prognostic model that builds on the Bayesian network methodology, and implements a dynamic, process-oriented view on prognosis. A PBN describes the mutual relationships between variables that come into play during subsequent stages of a care process and a clinical outcome. A dedicated procedure for inducing these networks from clinical data is presented. In this procedure, the network is composed of a collection of local supervised learning models that are recursively learned from the data. The procedure optimizes performance of the network's primary task, outcome prediction, and handles the fact that patients may drop out of the process in earlier stages. Furthermore, the article describes how PBNs can be applied to solve a number of information problems that are related to medical prognosis.

[Age and survival likelihood of polytrauma patients. "Local tailoring" of the DGU prognosis model]

INTRODUCTION

Age is one of five prognostic parameters identified based on data of the trauma registry of the German Association for Trauma Surgery (DGU). We asked ourselves if the suggested prognostic model provides the same predictive power of data from an independent hospital. Furthermore, we investigated whether age itself or age-associated comorbidity causes an unfavorable prognostic effect.

METHODS

The investigation was based on data of 103 multiply injured patients (67 male, 36 female, mean age 35,4+/-SD 19,0 years, ISS 36,8+/-10,9). Data were collected prospectively following the guidelines of the trauma registry of the German Association for Trauma Surgery. Based on documented comorbidities, a risk calculation was performed using the ASA classification. Correlation between age and ASA was analyzed using Spearman's method. The prognostic value of the original model in our patient pool with or without ASA classification, possible interactions, and the discriminatory power of the model were estimated using logistic regression.

RESULTS

Attributable mortality was 31,7% (95% CI 22,7-41,7%). Age, ISS, GCS and ASA were included into the final logistic model. Odds ratios of the origin model were reproducible nearly identical in our patinet pool (OR: age 1,048; ISS 1,066; GCS 0,822). In spite of the fact that we have found a strong correlation between age and ASA-Classification (rho=0,60, p<0,0001) there was no prognostic value of comorbidity.

CONCLUSION

The suggested prognostic model based on multicenter data evaluation can be applied to a single center with only minimal loss of discriminatory power. In this context, age seems to have a prognostic value independent of comorbidity.

Mortality assessment in intensive care units via adverse events using artificial neural networks

OBJECTIVE

This work presents a novel approach for the prediction of mortality in intensive care units (ICUs) based on the use of adverse events, which are defined from four bedside alarms, and artificial neural networks (ANNs). This approach is compared with two logistic regression (LR) models: the prognostic model used in most of the European ICUs, based on the simplified acute physiology score (SAPS II), and a LR that uses the same input variables of the ANN model.

MATERIALS AND METHODS

A large dataset was considered, encompassing forty two ICUs of nine European countries. The recorded features of each patient include the final outcome, the case mix (e.g. age) and the intermediate outcomes, defined as the daily averages of the out of range values of four biometrics (e.g. heart rate). The SAPS II score requires 17 static variables (e.g. serum sodium), which are collected within the first day of the patient's admission. A nonlinear least squares method was used to calibrate the LR models while the ANNs are made up of multilayer perceptrons trained by the RPROP algorithm. A total of 13,164 adult patients were randomly divided into training (66%) and test (33%) sets. The two methods were evaluated in terms of receiver operator characteristic (ROC) curves.

RESULTS

The event based models predicted the outcome more accurately than the currently used SAPS II model (P<0.05), with ROC areas within the ranges 83.9-87.1% (ANN) and 82.6-85.2% (LR) versus 80% (LR SAPS II). When using the same inputs, the ANNs outperform the LR (improvement of 1.3-2%).

CONCLUSION

Better prognostic models can be achieved by adopting low cost and real-time intermediate outcomes rather than static data.

Temporal reasoning for decision support in medicine

OBJECTIVE

Handling time-related concepts is essential in medicine. During diagnosis it can make a substantial difference to know the temporal order in which some symptoms occurred or for how long they lasted. During prognosis the potential evolutions of a disease are conceived as a description of events unfolding in time. In therapy planning the different steps of treatment must be applied in a precise order, with a given frequency and for a certain span of time in order to be effective. This article offers a survey on the use of temporal reasoning for decision support-related tasks in medicine.

MATERIAL AND METHODS

Key publications of the area, mainly circumscribed to the latest two decades, are reviewed and classified according to three important stages of patient treatment requiring decision support: diagnosis, prognosis and therapy planning/management. Other complementary publications, like those on time-centered information storage and retrieval, are also considered as they provide valuable support to the above mentioned three stages.

RESULTS

Key areas are highlighted and used to organize the latest contributions. The survey of previous research is followed by an analysis of what can still be improved and what is needed to make the next generation of decision support systems for medicine more effective.

CONCLUSIONS

It can be observed that although the area has been considerably developed, there are still areas where more research is needed to make time-based systems of widespread use in decision support-related areas of medicine. Several suggestions for further exploration are proposed as a result of the survey.

Integrating classification trees with local logistic regression in Intensive Care prognosis

Health care effectiveness and efficiency are under constant scrutiny especially when treatment is quite costly as in the Intensive Care (IC). Currently there are various international quality of care programs for the evaluation of IC. At the heart of such quality of care programs lie prognostic models whose prediction of patient mortality can be used as a norm to which actual mortality is compared. The current generation of prognostic models in IC are statistical parametric models based on logistic regression. Given a description of a patient at admission, these models predict the probability of his or her survival. Typically, this patient description relies on an aggregate variable, called a score, that quantifies the severity of illness of the patient. The use of a parametric model and an aggregate score form adequate means to develop models when data is relatively scarce but it introduces the risk of bias. This paper motivates and suggests a method for studying and improving the performance behavior of current state-of-the-art IC prognostic models. Our method is based on machine learning and statistical ideas and relies on exploiting information that underlies a score variable. In particular, this underlying information is used to construct a classification tree whose nodes denote patient sub-populations. For these sub-populations, local models, most notably logistic regression ones, are developed using only the total score variable. We compare the performance of this hybrid model to that of a traditional global logistic regression model. We show that the hybrid model not only provides more insight into the data but also has a better performance. We pay special attention to the precision aspect of model performance and argue why precision is more important than discrimination ability.