Evaluation of two outcome prediction models on an independent database

Predictive Modeling Using Artificial Intelligence and Machine Learning Algorithms on Electronic Health Record Data: Advantages and Challenges

The rapid adoption of electronic health record (EHR) systems in US hospitals from 2008 to 2014 produced novel data elements for analysis. Concurrent innovations in computing architecture and machine learning (ML) algorithms have made rapid consumption of health data feasible and a powerful engine for clinical innovation. In critical care research, the net convergence of these trends has resulted in an exponential increase in outcome prediction research. In the following article, we explore the history of outcome prediction in the intensive care unit (ICU), the growing use of EHR data, and the rise of artificial intelligence and ML (AI) in critical care.

The Norton scale is an important predictor of in-hospital mortality in internal medicine patients

BACKGROUND

The Norton scale, a marker of patient frailty used to predict the risk of pressure ulcers, but the predictive value of the Norton scale for in-hospital mortality after adjustment for a wide range of demographic, and abnormal admission laboratory test results shown in themselves to have a high predictive value for in-hospital mortality is unclear.

AIM

The study aims to determine the value of the Norton scale and the presence of a urinary catheter in predicting in hospital mortality.

METHODS

The study population included all acutely admitted adult patients in 2020 through October 2021 to one of three internal medicine departments at the Laniado Hospital, a regional hospital with 400 beds in Israel. The main objective was to (a) identify the variables associated with the Norton Scale and (b) determine whether it predicts in-hospital mortality after adjustment for these variables.

RESULTS

The Norton scale was associated with an older age, female gender, presence of a urinary catheter, and abnormal laboratory tests. The odds of in-hospital mortality in those with intermediate, high, and very high Norton scale risk groups were 3.10 (2.23-3.56), 6.48 (4.02-10.46), and 12.27 (7.37-20.44), respectively, after adjustment for the remaining predictors. Adding the Norton scale and the presence of a urinary catheter to the prediction logistic regression model that included age, gender, and abnormal laboratory test results increased the c-statistic from 0.870 (0.864-0.876) to 0.908 (0.902-0.913).

CONCLUSIONS

The Norton scale and presence of a urinary catheter are important predictors of in-hospital mortality in acutely hospitalized adults in internal medicine departments.

Predictive outcomes of APACHE II and expanded SAPS II mortality scoring systems in coronary care unit

OBJECTIVE

We investigated the predictive values of the expanded Simplified Acute Physiology Score (SAPS) II and Acute Physiologic Score and Chronic Health Evaluation (APACHE) II score in predicting in-hospital mortality in coronary care unit (CCU) patients.

METHODS

In this study, expanded SAPS II and APACHE II scores were calculated in the CCU of a single-center tertiary hospital. Patients admitted to CCU with any cardivascular indication were included in the study. Both scores were calculated according to previously determined criteria. Calibration and discrimination abilities of the scores in predicting in-hospital mortality were tested with Hosmer-Lemeshow goodness-of-fit C chi-square and receiver operating characteristics (ROC) curve analyses.

RESULTS

A total of 871 patients were included in the analysis. The goodness-of-fit C chi-square test showed that both scores have a good performance in predicting survivors and nonsurvivors in CCU. Expanded SAPS II score has a sensitivity of 80% and a specificity of 91.8% with the cut-off value of 5.55, while APACHE II has a sensitivity of 75.9% and a specificity of 87.4% with the cut-off value of 16.5 in predicting mortality.

CONCLUSION

Expanded SAPS II and APACHE II scores have good ability to predict in-hospital mortality in CCU patients. Therefore, they can be used as a tool to predict short-term mortality in cardiovascular emergencies.

The presence of a urinary catheter is an important predictor of in-hospital mortality in internal medicine patients

RATIONALE AND OBJECTIVE

Mortality rates are used to assess the quality of hospital care after appropriate adjustment for case-mix. Urinary catheters are frequent in hospitalized adults and might be a marker of patient frailty and illness severity. However, we know of no attempts to estimate the predictive value of indwelling catheters for specific patient outcomes. The objective of the present study was to (a) identify the variables associated with the presence of a urinary catheter and (b) determine whether it predicts in-hospital mortality after adjustment for these variables.

METHODS

The study population included all acutely admitted adult patients in 2020 (exploratory cohort) and January-October 2021 (validation cohort) to internal medicine, cardiology and intensive care departments at the Laniado Hospital, a regional hospital with 400 beds in Israel. There were no exclusion criteria. The predictor variables were the presence of a urinary catheter on admission, age, gender, comorbidities and admission laboratory test results. We used bivariate and multivariate logistic regression to test the associations between the presence of a urinary catheter and mortality after adjustment for the remaining independent variables on admission.

RESULTS

The presence of a urinary catheter was associated with other independent variables. In 2020, the odds of in-hospital mortality in patients with a urinary catheter before and after adjustment for the remaining predictors were 14.3 (11.6-17.7) and 6.05 (4.78-7.65), respectively. Adding the presence of a urinary catheter to the prediction logistic regression model increased its c-statistic from 0.887 (0.880-0.894) to 0.907 (0.901-0.913). The results of the validation cohort reduplicated those of the exploratory cohort.

CONCLUSIONS

The presence of a urinary catheter on admission is an important and independent predictor of in-hospital mortality in acutely hospitalized adults in internal medicine departments.

A simple index predicting mortality in acutely hospitalized patients

BACKGROUND

Mortality rates used to evaluate and improve the quality of hospital care are adjusted for comorbidity and disease severity. Comorbidity, measured by International Classification of Diseases codes, do not reflect the severity of the medical condition, that requires clinical assessments not available in electronic databases, and/or laboratory data with clinically relevant ranges to permit extrapolation from one setting to the next.

AIM

To propose a simple index predicting mortality in acutely hospitalized patients.

DESIGN

Retrospective cohort study with internal and external validation.

METHODS

The study populations were all acutely admitted patients in 2015-16, and in January 2019-November 2019 to internal medicine, cardiology and intensive care departments at the Laniado Hospital in Israel, and in 2002-19, at St. James Hospital, Ireland. Predictor variables were age and admission laboratory tests. The outcome variable was in-hospital mortality. Using logistic regression of the data in the 2015-16 Israeli cohort, we derived an index that included age groups and significant laboratory data.

RESULTS

In the Israeli 2015-16 cohort, the index predicted mortality rates from 0.2% to 32.0% with a c-statistic (area under the receiver operator characteristic curve) of 0.86. In the Israeli 2019 validation cohort, the index predicted mortality rates from 0.3% to 38.9% with a c-statistic of 0.87. An abbreviated index performed similarly in the Irish 2002-19 cohort.

CONCLUSIONS

Hospital mortality can be predicted by age and selected admission laboratory data without acquiring information from the patient's medical records. This permits an inexpensive comparison of performance of hospital departments.

Bioinformatic analysis of the potential molecular mechanism of PAK7 expression in glioblastoma

The present study aimed to determine the potential molecular mechanisms underlying p21 (RAC1)-activated kinase 7 (PAK7) expression in glioblastoma (GBM) and evaluate candidate prognosis biomarkers for GBM. Gene expression data from patients with GBM, including 144 tumor samples and 5 normal brain samples, were downloaded. Long non-coding RNAs (lncRNAs) and messenger RNAs (mRNAs) were explored via re-annotation. The differentially expressed genes (DEGs), including differentially expressed mRNAs and differentially expressed lncRNAs, were investigated and subjected to pathway analysis via gene set enrichment analysis. The miRNA-lncRNA-mRNA interaction [competing endogenous RNA (ceRNA)] network was investigated and survival analysis, including of overall survival (OS), was performed on lncRNAs/mRNAs to reveal prognostic markers for GBM. A total of 954 upregulated and 1,234 downregulated DEGs were investigated between GBM samples and control samples. These DEGs, including PAK7, were mainly enriched in pathways such as axon guidance. ceRNA network analysis revealed several outstanding ceRNA relationships, including miR-185-5p-LINC00599-PAK7. Moreover, paraneoplastic antigen Ma family member 5 (PNMA5) and somatostatin receptor 1 (SSTR1) were the two outstanding prognostic genes associated with OS. PAK7 may participate in the tumorigenesis of GBM by regulating axon guidance, and miR-185-5p may play an important role in GBM progression by sponging LINC00599 to prevent interactions with PAK7. PNMA5 and SSTR1 may serve as novel prognostic markers for GBM.

Defining a High-Performance ICU System for the 21st Century: A Position Paper

In the fall of 1997 George D. Lundberg and John E. Wennberg wrote an editorial in JAMA calling for comprehensive quality improvement programs to become the driver of the American health care system. The suggestion came during the Second European Forum on Quality Improvement in Health Care held in Paris, France, in April 1997 and was based on comments made by Donald Berwick. The concept was to focus on an organized response to problem identification and proposed solutions to improve patient care and protect the health of the public. Critical care medicine represents a large segment of health care and is undergoing dramatic changes during our managed care revolution. General ICU severity of illness models have been developed, tested, and shown to provide a useful estimate of hospital mortality for populations of critically ill patients. These systems have captured the imagination of clinical researchers and have become an integral component of a large number of publications as well as a part of many ICU databases. These risk adjustment severity models are remarkably robust for heterogeneous patient populations but the models have not been shown to validate well in new settings. We feel that by focusing on the episode of critical illness rather than each individual ICU admission and by going beyond the traditional acute hospital discharge to determine whether the patient lives or dies, we can better evaluate critical care system performance and cost-effectiveness. The incentives for high quality/low cost should favor integrated comprehensive critical care delivery systems. Programs that score well should be identified as high quality and be honored as medallion level 1 ICUs. We challenge national and international critical care societies to evaluate and then debate the described definitions and recommendations as a call to action.

The predictive performance of the SAPS II and SAPS 3 scoring systems: A retrospective analysis

PURPOSE

The purpose was to analyze and compare the performance of Simplified Acute Physiology Score (SAPS) II and SAPS 3 (North Europe Logit) in an intensive care unit (ICU) for internal disorders at a German university hospital.

MATERIALS AND METHODS

This retrospective study was conducted at a single-center 12-bed ICU sector for Internal Medicine in Essen, Germany, within an 18-month period. Data for adult ICU patients (N = 548) were evaluated. SAPS II and SAPS 3 scores were assessed along with the predicted mortality rates. Discrimination was evaluated by calculating the area under the receiver operating characteristic curve, and calibration was evaluated using the Hosmer-Lemeshow goodness-of-fit C-test. The ratios of observed-to-expected deaths (standardized mortality ratio, SMR) were calculated along with the 95% confidence intervals (95% CIs).

RESULTS

The in-hospital mortality rate was 22.6%, which provided an SMR of 0.91 (95% CI, 0.77-0.99) for SAPS II and 0.62 (95% CI, 0.52-0.71) for SAPS 3. Both SAPS II and SAPS 3 exhibited acceptable discrimination, with an area under the receiver operating characteristic curve of 0.84 (95% CI, 0.79-0.89) and 0.73 (95% CI, 0.67-0.79), respectively. However, SAPS II demonstrated superior SMR-based discrimination, which was closer to the observed mortality rate, compared with SAPS 3. Calibration curves exhibited similar performance based on the Hosmer-Lemeshow goodness-of-fit C-test results: χ(2) = 7.10 with P = .525 for SAPS II and χ(2) = 3.10 with P = .876 for SAPS 3. Interestingly, both scores overpredicted mortality.

CONCLUSIONS

In this study, SAPS 3 overestimated mortality and therefore appears less suitable for risk evaluation in comparison to SAPS II.

Mortality Prediction in Patients Undergoing Non-Invasive Ventilation in Intermediate Care

Background

Intermediate Care Units (ImCU) have become an alternative scenario to perform Non-Invasive Ventilation (NIV). The limited number of prognostic studies in this population support the need of mortality prediction evaluation in this context.

Objective

The objective of this study is to analyze the performance of Simplified Acute Physiology Score (SAPS) II and 3 in patients undergoing NIV in an ImCU. Additionally, we searched for new variables that could be useful to customize these scores, in order to improve mortality prediction.

Design

Cohort study with prospectively collected data from all patients admitted to a single center ImCU who received NIV. The SAPS II and 3 scores with their respective predicted mortality rates were calculated. Discrimination and calibration were evaluated by calculating the area under the receiver operating characteristic curve (AUC) and with the Hosmer-Lemeshow goodness of fit test for the models, respectively. Binary logistic regression was used to identify new variables to customize the scores for mortality prediction in this setting.

Patients

The study included 241 patients consecutively admitted to an ImCU staffed by hospitalists from April 2006 to December 2013.

Key Results

The observed in-hospital mortality was 32.4% resulting in a Standardized Mortality Ratio (SMR) of 1.35 for SAPS II and 0.68 for SAPS 3. Mortality discrimination based on the AUC was 0.73 for SAPS II and 0.69 for SAPS 3. Customized models including immunosuppression, chronic obstructive pulmonary disease (COPD), acute pulmonary edema (APE), lactic acid, pCO2 and haemoglobin levels showed better discrimination than old scores with similar calibration power.

Conclusions

These results suggest that SAPS II and 3 should be customized with additional patient-risk factors to improve mortality prediction in patients undergoing NIV in intermediate care.

Feasibility of transitioning from APACHE II to SAPS III as prognostic model in a Brazilian general intensive care unit. A retrospective study

CONTEXT AND OBJECTIVE

Prognostic models reflect the population characteristics of the countries from which they originate. Predictive models should be customized to fit the general population where they will be used. The aim here was to perform external validation on two predictive models and compare their performance in a mixed population of critically ill patients in Brazil.

DESIGN AND SETTING

Retrospective study in a Brazilian general intensive care unit (ICU).

METHODS

This was a retrospective review of all patients admitted to a 41-bed mixed ICU from August 2011 to September 2012. Calibration (assessed using the Hosmer-Lemeshow goodness-of-fit test) and discrimination (assessed using area under the curve) of APACHE II and SAPS III were compared. The standardized mortality ratio (SMR) was calculated by dividing the number of observed deaths by the number of expected deaths.

RESULTS

A total of 3,333 ICU patients were enrolled. The Hosmer-Lemeshow goodness-of-fit test showed good calibration for all models in relation to hospital mortality. For in-hospital mortality there was a worse fit for APACHE II in clinical patients. Discrimination was better for SAPS III for in-ICU and in-hospital mortality (P = 0.042). The SMRs for the whole population were 0.27 (confidence interval [CI]: 0.23 - 0.33) for APACHE II and 0.28 (CI: 0.22 - 0.36) for SAPS III.

CONCLUSIONS

In this group of critically ill patients, SAPS III was a better prognostic score, with higher discrimination and calibration power.

Validation of the revised injury severity classification score in patients with moderate-to-severe traumatic brain injury

INTRODUCTION

By analysing risk-adjusted mortality ratios, weaknesses in the process of care might be identified. Traumatic brain injury (TBI) is the main cause of death in trauma, and thus it is crucial that trauma prediction models are valid for TBI patients. Accordingly, we assessed the validity of the RISC score in TBI patients by internal and external validation analyses.

METHODS

Patients with moderate-to-severe TBI admitted to the TraumaRegister DGU® (TR-DGU) and the trauma registry of Helsinki University Hospital (TR-THEL) in 2006-2011 were included in this retrospective open cohort study. Definition of moderate-to-severe TBI was head abbreviated injury scale of 3 or higher. Subgroup analysis for patients with isolated and polytrauma TBI was performed. The performance of the RISC score was evaluated by assessing its discrimination (area under the curve, AUC) and calibration (Hosmer-Lemeshow [H-L] test).

RESULTS

Among the 9106 and 809 patients with moderate-to-severe TBI admitted to TR-DGU and TR-THEL, unadjusted mortality was 26% and 23%, respectively. Internal and external validation of the RISC score showed good discrimination (TR-DGU AUC 0.89, 95% confidence interval [CI] 0.88-0.90 and TR-THEL AUC 0.84, 95% CI 0.81-0.87), but poor calibration (p<0.001) in patients with moderate-to-severe TBI. Subgroup analysis found the discrimination only to be modest in isolated TBI (AUC 0.76) and calibration to be particularly poor in polytrauma TBI (TR-DGU H-L=4356, p<0.001; TR-THEL H-L 112, p<0.001).

CONCLUSION

The RISC score was found to be of limited predictive value in patients with moderate-to-severe TBI. A new general trauma scoring system that includes TBI specific prognostic factors is warranted.

SAPS 3 is not superior to SAPS 2 in cardiac surgery patients

OBJECTIVES

Cardiac surgery patients are excluded from SAPS2 but included in SAPS3. Neither score is evaluated for this exclusive population; however, they are used daily. We hypothesized that SAPS3 may be superior to SAPS2 in outcome prediction in cardiac surgery patients.

DESIGN

All consecutive patients undergoing cardiac surgery between January 2007 and December 2010 were included in our prospective study. Both models were tested with calibration and discrimination statistics. We compared the AUC of the ROC curves by DeLong's method and calculated OCC values.

RESULTS

A total of 5207 patients with mean age of 67.2 ± 10.9 years were admitted to the ICU. The mean length of ICU stay was 4.6 ± 7.0 days and the ICU mortality was 5.9%. The two tested models had acceptable discriminatory power (AUC: SAPS2: 0.777-0.875; SAPS3: 0.757-893). SAPS3 had a low AUC and poor calibration on admission day. SAPS2 had poor calibration on Days 1-6 and 8.

CONCLUSIONS

Despite including cardiac surgery patients, SAPS3 was not superior to SAPS2 in our analysis. In this large cohort of ICU cardiac surgery patients, performance of both SAPS models was generally poor. In this subset of patients, neither scoring system is recommended.

Performance of SAPS II and SAPS 3 in intermediate care

Objective

The efficacy and reliability of prognostic scores has been described extensively for intensive care, but their role for predicting mortality in intermediate care patients is uncertain. To provide more information in this field, we have analyzed the performance of the Simplified Acute Physiology Score (SAPS) II and SAPS 3 in a single center intermediate care unit (ImCU).

Materials and Methods

Cohort study with prospectively collected data from all patients admitted to a single center ImCU in Pamplona, Spain, from April 2006 to April 2012. The SAPS II and SAPS 3 scores with respective predicted mortality rates were calculated according to standard coefficients. Discrimination was evaluated by calculating the area under receiver operating characteristic curve (AUROC) and calibration with the Hosmer-Lemeshow goodness of fit test. Standardized mortality ratios (SMR) with 95% confidence interval (95% CI) were calculated for each model.

Results

The study included 607 patients. The observed in-hospital mortality was 20.1% resulting in a SMR of 0.87 (95% CI 0.73-1.04) for SAPS II and 0.56 (95% CI 0.47-0.67) for SAPS 3. Both scores showed acceptable discrimination, with an AUROC of 0.76 (95% CI 0.71-0.80) for SAPS II and 0.75 (95% CI 0.71- 0.80) for SAPS 3. Calibration curves showed similar performance based on Hosmer-Lemeshow goodness of fit C-test: (X²=12.9, p=0.113) for SAPS II and (X²=4.07, p=0.851) for SAPS 3.

Conclusions

Although both scores overpredicted mortality, SAPS II showed better discrimination for patients admitted to ImCU in terms of SMR.

Prognostic performance of the Simplified Acute Physiology Score II in major Croatian hospitals: a prospective multicenter study

Aim

To perform an external validation of the original Simplified Acute Physiology Score II (SAPS II) system and to assess its performance in a selected group of patients in major Croatian hospitals.

Methods

A prospective, multicenter study was conducted in five university hospitals and one general hospital during a six-month period between November 1, 2007 and May 1, 2008. Standardized hospital mortality ratio (SMR) was calculated from the mean predicted mortality of all the 2756 patients and the actual mortality for the same group of patients. The validation of SAPS II was made using the area under receiver operating characteristic curve (AUC), 2 × 2 classification tables, and Hosmer-Lemeshow tests.

Results

The predicted mortality was as low as 14.6% due to a small proportion of medical patients and the SMR being 0.89 (95% confidence interval [CI], 0.78-0.98). The SAPS II system demonstrated a good discriminatory power as measured by the AUC (0.85; standard error [SE] = 0.012; 95% CI = 0.840-0.866; P < 0.001). This system significantly overestimated the actual mortality (Hosmer-Lemeshow goodness-of-fit H statistic: χ² = 584.4; P < 0.001 and C statistics: χ²₈ = 313.0; P < 0.001) in the group of patients included in the study.

Conclusion

The SAPS II had a good discrimination, but it significantly overestimated the observed mortality in comparison with the predicted mortality in this group of patients in Croatia. Therefore, caution is required when an evaluation is performed at the individual level.

The role of ω-3 fatty acid supplemented parenteral nutrition in critical illness in adults: a systematic review and meta-analysis. Crit Care Med. 2013;41:307-316. [PMID: 23128380 DOI: 10.1097/CCM.0b013e3182657578]

OBJECTIVE

To determine whether the supplementation of parenteral nutrition with ω-3 fatty acids confers treatment benefits to critically ill adult patients.

DATA SOURCE

We performed computerized searches for relevant articles from 1996 to June 2011 on MEDLINE, EMBASE, and the Cochrane register of controlled trials and abstracts of scientific meetings from 2005 to 2011.

STUDY SELECTION

Randomized controlled trials of ω-3 fatty acid supplemented parenteral nutrition in critically ill adult patients admitted to the intensive therapy unit, given in addition to their routine care, compared with parenteral nutrition without ω-3 fatty acid supplementation.

DATA SYNTHESIS

Five fully published trials and three trials published in abstract form with 391 participants have been included. Overall trial quality was poor. Mortality data were pooled from eight studies with 391 participants. No differences were found with a risk ratio for death of 0.83 (95% confidence interval 0.57, 1.20; p = 0.32). Data for infectious complications were available from five studies with 337 participants. No differences were found, with a risk ratio for infection of 0.78 (95% confidence interval 0.43, 1.41; p = 0.41). Data for intensive therapy unit and hospital length of stay were available from six and three studies with 305 and 117 participants, respectively. With respect to intensive therapy unit length of stay, no differences were observed with a mean difference of 0.57 days in favor of the ω-3 fatty acid group (95% confidence interval -5.05, 3.90; p = 0.80). A significant reduction in hospital length of stay of 9.49 days (95% confidence interval -16.51, -2.47; p = 0.008) was observed for those receiving ω-3 fatty acid supplemented parenteral nutrition, but results were strongly influenced by one small study.

CONCLUSIONS

On the basis of this systematic review, it can be concluded that ω-3 fatty acid supplementation of parenteral nutrition does not improve mortality, infectious complications, and intensive therapy unit length of stay in comparison with standard parenteral nutrition. Although ω-3 fatty acids appear to reduce hospital length of stay, the poor methodology of the included studies and the absence of other outcome improvements mean they cannot be presently recommended.

Relationship between mortality and first-day events index from routinely gathered physiological variables in ICU patients

OBJECTIVE

To test the hypothesis that the degree and duration of alterations in physiological variables routinely gathered by intensive care unit (ICU) monitoring systems during the first day of admission to the ICU, together with a few additional routinely recorded data, yield information similar to that obtained by traditional mortality prediction systems.

DESIGN

A prospective observational multicenter study (EURICUS II) was carried out.

SETTING

Fifty-five European ICUs.

PATIENTS

A total of 17,598 consecutive patients admitted to the ICU over a 10-month period.

INTERVENTIONS

None.

MAIN VARIABLES OF INTEREST

Hourly data were manually gathered on alterations or "events" in systolic blood pressure, heart rate and oxygen saturation throughout ICU stay to construct an events index and mortality prediction models.

RESULTS

The mean first-day events index was 6.37±10.47 points, and was significantly associated to mortality (p<0.001), with a discrimination capacity for hospital mortality of 0.666 (area under the ROC curve). A second index included this first-day events index, age, pre-admission location, and the Glasgow coma score. A model constructed with this second index plus diagnosis upon admission was validated by using the Jackknife method (Hosmer-Lemeshow, H=13.8554, insignificant); the area under ROC curve was 0.818.

CONCLUSIONS

A prognostic index with performance very similar to that of habitual systems can be constructed from routine ICU data with only a few patient characteristics. These results may serve as a guide for the possible automated construction of ICU prognostic indexes.

Is the SAPS II score valid in surgical intensive care unit patients?

AIMS AND OBJECTIVES

We investigated the performance of the simplified acute physiology score II (SAPS II) in a large cohort of surgical intensive care unit (ICU) patients and tested the hypothesis that customization of the score would improve the uniformity of fit in subgroups of surgical ICU patients.

METHODS

Retrospective analysis of prospectively collected data from all 12,938 patients admitted to a postoperative ICU between January 2004 and January 2009. Probabilities of hospital death were calculated for original and customized (C1-SAPS II and C2-SAPS II) scores. A priori subgroups were defined according to age, probability of death according to the SAPS II score, ICU length of stay (LOS), surgical procedures and type of admission.

RESULTS

The median ICU LOS was 1 (1-3) day. ICU and hospital mortality rates were 5.8% and 10.3%, respectively. Discrimination of the SAPS II was moderate [area under receiver operating characteristic curve (aROC) = 0.76 (0.75-0.78)], but calibration was poor. This model markedly overestimated hospital mortality rates [standardized mortality rate: 0.35 (0.33-0.37)]. First-level customization (C1-SAPS II) did not improve discrimination in the whole cohort or the subgroups, but calibration improved in some subgroups. Second-level customization (C2-SAPS II) improved discrimination in the whole cohort [aROC = 0.82 (0.79-0.85)] and most of the subgroups (aROC range 0.65-86). Calibration in this model (C2-SAPS II) improved in the whole cohort and in subgroups except in patients with ICU LOS 4-14 days and those undergoing neuro- or gastrointestinal surgery.

CONCLUSIONS

In this large cohort of surgical ICU patients, performance of the original SAPS II model was generally poor. Although second-level customization improved discrimination and calibration in the whole cohort and most of the subgroups, it failed to simultaneously improve calibration in the subgroups stratified according to the type of surgery, age or ICU LOS.

Prognostic value of Sequential Organ Failure Assessment and Simplified Acute Physiology II Score compared with trauma scores in the outcome of multiple-trauma patients

BACKGROUND

Prospective data regarding the prognostic value of the Sequential Organ Failure Assessment (SOFA) score in comparison with the Simplified Acute Physiology Score (SAPS II) and trauma scores on the outcome of multiple-trauma patients are lacking.

METHODS

Single-center evaluation (n = 237, Injury Severity Score [ISS] >16; mean ISS = 29). Uni- and multivariate analysis of SAPS II, SOFA, revised trauma, polytrauma, and trauma and ISS scores (TRISS) was performed.

RESULTS

The 30-day mortality was 22.8% (n = 54). SOFA day 1 was significantly higher in nonsurvivors compared with survivors (P < .001) and correlated well with the length of intensive care unit stay (r = .50, P < .001). Logistic regression revealed SAPS II to have the best predictive value of 30-day mortality (area under the receiver operating characteristic = .86 +/- .03). The SOFA score significantly added prognostic information with regard to mortality to both SAPS II and TRISS.

CONCLUSIONS

The combination of critically ill and trauma scores may increase the accuracy of mortality prediction in multiple-trauma patients.

Outcome prediction in critical care: the Acute Physiology and Chronic Health Evaluation models

PURPOSE OF REVIEW

A new generation of predictive models for critically ill patients was described between 2005 and 2008. This review will give details of the latest version of the Acute Physiology and Chronic Health Evaluation (APACHE) predictive models, and discuss it in relation to recent critical care outcome studies. We also compare APACHE IV with other systems and address the issue of model complexity.

RECENT FINDINGS

APACHE IV required the remodeling of over 40 equations. These new models calibrate better to contemporary data than older versions of APACHE and there is good predictive accuracy within diagnostic subgroups. Physiology accounts for 66% and diagnosis for 17% of the APACHE IV mortality model's predictive power. Thus, physiology and diagnosis account for 83% of the accuracy of APACHE IV.

SUMMARY

Predictive models have a modest window of applicability, and therefore must be revalidated frequently. This was shown to be true for APACHE III, and hence a major reestimation of models was carried out to generate APACHE IV. Although overall model accuracy is important, it is also imperative that predictive models work well within diagnostic subgroups.

Outcome prediction in critical care: the Mortality Probability Models

PURPOSE OF REVIEW

The comparison of morbidity, mortality, and length-of-stay outcomes in patients receiving critical care requires adjustment based on their presenting illness. These adjustments are made with severity-of-illness models. These models must be periodically updated to reflect current medical practices. This article will review the history of the Mortality Probability Model (MPM), discuss why and how it was recently updated, and outline examples of MPM use.

RECENT FINDINGS

All severity-of-illness models have limitations, especially if a unit's patient population becomes highly specialized. In these situations, customized models may provide better accuracy. The MPMs include those calculated at admission (MPM0) and additional models at 24, 48, and 72 h (MPM 24, MPM 48, and MPM 72). The model is now in its third iteration (MPM 0-III). Length of stay (LOS) and subgroup models have also been developed.

SUMMARY

Understanding appropriate application of models such as MPM is important as transparency in healthcare drives demand for severity-adjusted outcomes data.

Outcome prediction in critical care: the Simplified Acute Physiology Score models

PURPOSE OF REVIEW

Outcome prediction models measuring severity of illness of patients admitted to the intensive care unit should predict hospital mortality. This review describes the state-of-the-art of Simplified Acute Physiology Score models from the clinical and managerial perspectives. Methodological issues concerning the effects of differences between new samples and original databases in which the models were developed are considered.

RECENT FINDINGS

The progressive lack of fit of the Simplified Acute Physiology Score II in independent intensive care unit populations induced investigators to propose customizations and expansions as potential evolutions for Simplified Acute Physiology Score II. We do not know whether those solutions did solve the issue because there are no demonstrations of consistent good fit in new databases. The recently developed Simplified Acute Physiology Score 3 Admission Score with customization for geographical areas is discussed. The points shared by the Simplified Acute Physiology Score models and the pros and cons for each of them are introduced.

SUMMARY

Comparisons of intensive care unit performance should take into account not only the patient severity of illness, but also the effect of the 'intensive care unit variable', that is, differences in human resources, structure, equipment, management and organization of the intensive care unit. In the future, moving from patient and geographical area adjustment to resource use could allow the user to adjust for differences in healthcare provision.

Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit

BACKGROUND

The Simplified Acute Physiology Score (SAPS) 3 has recently been developed, but not yet validated in surgical intensive care unit (ICU) patients. We compared the performance of SAPS 3 with SAPS II and the Acute Physiology and Chronic Health Evaluation (APACHE) II score in surgical ICU patients.

METHODS

Prospectively collected data from all patients admitted to a German university hospital postoperative ICU between August 2004 and December 2005 were analysed. The probability of ICU mortality was calculated for SAPS II, APACHE II, adjusted APACHE II (adj-APACHE II), SAPS 3, and SAPS 3 customized for Europe [C-SAPS3 (Eu)] using standard formulas. To improve calibration of the prognostic models, a first-level customization was performed, using logistic regression on the original scores, and the corresponding probability of ICU death was calculated for the customized scores (C-SAPS II, C-SAPS 3, and C-APACHE II).

RESULTS

The study included 1851 patients. Hospital mortality was 9%. Hosmer and Lemeshow statistics showed poor calibration for SAPS II, APACHE II, adj-APACHE II, SAPS 3, and C-SAPS 3 (Eu), but good calibration for C-SAPS II, C-APACHE II, and C-SAPS 3. Discrimination was generally good for all models [area under the receiver operating characteristic curve ranged from 0.78 (C-APACHE II) to 0.89 (C-SAPS 3)]. The C-SAPS 3 score appeared to have the best calibration curve on visual inspection.

CONCLUSIONS

In this group of surgical ICU patients, the performance of SAPS 3 was similar to that of APACHE II and SAPS II. Customization improved the calibration of all prognostic models.

Sepsis and organ system failure are major determinants of post-intensive care unit mortality

PURPOSE

The aim of the study was to investigate predictors of post-intensive care unit (ICU) in-hospital mortality with special emphasis on the impact of sepsis and organ system failure.

METHODS

This study is a subanalysis of the database from the observational Sepsis Occurrence in Acutely Ill Patients study conducted in 198 ICUs in 24 European countries between May 1 and May 15, 2002. Potential predictors of post-ICU mortality were considered at 3 levels: admission status, procedures and therapy during the ICU stay, and status at ICU discharge.

RESULTS

Of the 3147 patients included in the Sepsis Occurrence in Acutely Ill Patients study, 1729 (54.9%) were discharged to the general floor (study group) and 125 of these died (overall post-ICU hospital mortality rate, 4%); 26 (20.8%) died already the first day on the floor. Nonsurvivors were older, had higher incidence of hematologic cancer and cirrhosis, and greater Simplified Acute Physiology Score II and Sequential Organ Failure Assessment score on ICU admission; they were also more likely to have been admitted for medical reasons than survivors. In a multivariate forward stepwise logistic regression analysis, age, hematologic cancer, cirrhosis, simplified acute physiology score II on admission, medical admission, sepsis at any time during ICU stay, and organ dysfunction at ICU discharge were all independently associated with a greater risk of post-ICU death.

CONCLUSIONS

This large international study identified not only age, medical admission, and preexisting comorbidities on ICU admission but also sepsis and organ system failure as important independent risk factors for in-hospital post-ICU death.

Severity scoring in the ICU: a review

BACKGROUND

Patients in the intensive care unit (ICU) require huge resources because of the dysfunction of several of their vital organs. The heterogeneity and complexity of the ICU patient have generated interest in systems able to measure severity of illness as a method of predicting outcome, comparing quality-of-care and stratification for clinical trials.

METHODS

By searching Medline and EMBASE for publications describing scoring systems in the ICU, the most frequently used systems, defined as resulting in more than 50 references, are included in this review. Scoring systems belong to one of four classes prognostic, single-organ failure, trauma scores and organ dysfunction (OD). The different systems are described and discussed.

RESULTS

Three different prognostic scoring systems, including several versions, four single OD scores and three OD scores, were included in this review.

CONCLUSION

Different forms of scoring systems are frequently used in the ICU. They have become a necessary tool to describe ICU populations and to explain differences in mortality. As there are several pitfalls related to the interpretation of the numbers supplied by the systems, they should not be used without knowledge on the science of severity scoring.

External Validity of a Mortality Prediction Model in Patients After Open Abdominal Aortic Aneurysm Repair Using Multi-level Methodology

OBJECTIVES

Evaluation of the prognostic ability of the APACHE-AAA model in an independent group of post-operative (open) Abdominal Aortic Aneurysm (AAA) patients.

METHODS

The model was applied to predict in-hospital mortality in 541 patients (325 elective and 216 emergencies; 489 from Oxford; 52 from Lewisham). Multi-level modelling was used to adjust for both the local structure and process of care and patient case-mix. Model performance was assessed using goodness-of-fit and subgroup analyses.

RESULTS

The model's predictive ability to discriminate between dead and alive patients was very good (ROC area=0.84). The model achieved a good fit across all strata of risk (Hosmer-Lemeshow C-test (8, N=476)=7.777, p=0.456) and in all subgroups. The model was able to rank the ICUs according to their performance independently of the patient case-mix.

CONCLUSION

The APACHE-AAA model accurately predicted in-hospital mortality in a population of patients independent of the one used to develop it, confirming its validity. The multi-level methodology employed has shown that patient outcome is not only a function of the patient case-mix but instead predictive models should also adjust for the individual hospital-related factors (structure and process of care).

Comparison of Mortality Prediction Models after Open Abdominal Aortic Aneurysm Repair

OBJECTIVES

Comparison of the accuracy of prediction of contemporary mortality prediction models after open Abdominal Aortic Aneurysm (AAA) surgery.

METHODS

Post-operative data were collected from AAA patients from 2 UK Intensive Care Units (ICU). POSSUM and VBHOM based models were compared to the APACHE-AAA model which was able to adjust for the hospital-related effect on outcome. Model performance was assessed using measures of calibration, discrimination and subgroup analysis.

RESULTS

541 patients were studied. The in-hospital mortality rate for elective AAA repair (325 patients) was: 6.2% (95% confidence interval (c.i.) 3.5 to 8.8) and for emergency repair (216 patients) was: 28.7% (95% c.i. 22.5-34.9). The APACHE-based model had the best overall fit to the whole population of AAA patients, and also separately in elective and emergency patients. The V-POSSUM physiology-only (p<0.001) and VBHOM (p=0.011) models had a poor fit in elective patients. The RAAA-POSSUM physiology-only (p<0.001) and VBHOM models (p=0.010) had a poor fit in emergency patients.

CONCLUSIONS

The APACHE-AAA model with its ability to adjust for both the hospital-related "effect" as well as the patient case-mix, was a more accurate risk stratification model than other contemporary models, in the post-operative AAA patient managed in ICU.

Assessment of three different mortality prediction models in four well-defined critical care patient groups at two points in time: a prospective cohort study

BACKGROUND AND OBJECTIVE

Mortality prediction systems have been calculated and validated from large mixed ICU populations. However, in daily practice it is often more important to know how a model performs in a patient subgroup at a specific ICU. Thus, we assessed the performance of three mortality prediction models in four well-defined patient groups in one centre.

METHODS

A total of 960 consecutive adult patients with either severe head injury (n = 299), multiple injuries (n = 208), abdominal aortic aneurysm (n = 267) or spontaneous subarachnoid haemorrhage (n = 186) were included. Calibration, discrimination and standardized mortality ratios were determined for Simplified Acute Physiology Score II, Mortality Probability Model II (at 0 and 24 h) and Injury Severity Score. Effective mortality was assessed at hospital discharge and after 1 yr.

RESULTS

Eight hundred and fifty-five (89%) patients survived until hospital discharge. Over all four patient groups, Mortality Probability Model II (24 h) had the best predictive accuracy (standardized mortality ratio 0.62) and discrimination (area under the receiver operating characteristic curve 0.9), but Simplified Acute Physiology Score II performed well for patients with subarachnoid haemorrhage. Overall calibration was poor for all models (Hosmer-Lemeshow Type C-values between 20 and 26). Injury Severity Score had the worst discrimination in trauma patients. All models over-estimated hospital mortality in all four patient groups, and these estimates were more like the mortality after 1 yr.

CONCLUSIONS

In our surgical ICU, Mortality Probability Model II (24 h) performed slightly better than Simplified Acute Physiology Score II in terms of overall mortality prediction and discrimination; Injury Severity Score was the worst model for mortality prediction in trauma patients.

Analysis of physiologic alterations in intensive care unit patients and their relationship with mortality

PURPOSE

To analyze patient physiologic alterations (events) and multiple organ failure during intensive care unit (ICU) stay and examine their relationship with ICU mortality.

MATERIAL AND METHODS

A total of 17598 consecutive patients were studied for 10 months (1997-1998) in 55 European ICUs (EURICUS-II). Hourly data were collected on critical and noncritical systolic blood pressure, heart rate, oxygen saturation, and urinary events throughout ICU stay. Sepsis-related Organ Failure Assessment (SOFA) score was collected daily (6409 patients).

RESULTS

SAPS-II was 31.2 +/- 18.4 and ICU mortality 13.9%. There were 3.4 +/- 9.2 noncritical (duration, 3.9 +/- 11.4 hours) and 2 +/- 7.5 critical (3.8 +/- 13.1 hours) systolic blood pressure events per patient. Heart rate, oxygen saturation, and urinary events had similar values. Nonsurvivors had significantly more and longer physiologic alterations vs survivors. Mortality was significantly related to mean daily duration of events and mean and maximum daily SOFA. Discrimination capacity to predict ICU mortality was measured using various models: with SAPS II, area under the receiver operating characteristic curve was 0.80; with APACHE III-classified diagnosis added, 0.84; with mean duration of events/ICU day, 0.91; and with mean and maximum SOFA scores, 0.95.

CONCLUSION

Routinely gathered ICU data on physiologic variables and multiple organ failure can offer considerable complementary information not provided by usual mortality prediction systems; and their weight in daily care policy decisions may need to be revisited.

Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients

OBJECTIVE

To improve the accuracy of the Acute Physiology and Chronic Health Evaluation (APACHE) method for predicting hospital mortality among critically ill adults and to evaluate changes in the accuracy of earlier APACHE models.

DESIGN

: Observational cohort study.

SETTING

A total of 104 intensive care units (ICUs) in 45 U.S. hospitals.

PATIENTS

A total of 131,618 consecutive ICU admissions during 2002 and 2003, of which 110,558 met inclusion criteria and had complete data.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We developed APACHE IV using ICU day 1 information and a multivariate logistic regression procedure to estimate the probability of hospital death for randomly selected patients who comprised 60% of the database. Predictor variables were similar to those in APACHE III, but new variables were added and different statistical modeling used. We assessed the accuracy of APACHE IV predictions by comparing observed and predicted hospital mortality for the excluded patients (validation set). We tested discrimination and used multiple tests of calibration in aggregate and for patient subgroups. APACHE IV had good discrimination (area under the receiver operating characteristic curve = 0.88) and calibration (Hosmer-Lemeshow C statistic = 16.9, p = .08). For 90% of 116 ICU admission diagnoses, the ratio of observed to predicted mortality was not significantly different from 1.0. We also used the validation data set to compare the accuracy of APACHE IV predictions to those using APACHE III versions developed 7 and 14 yrs previously. There was little change in discrimination, but aggregate mortality was systematically overestimated as model age increased. When examined across disease, predictive accuracy was maintained for some diagnoses but for others seemed to reflect changes in practice or therapy.

CONCLUSIONS

APACHE IV predictions of hospital mortality have good discrimination and calibration and should be useful for benchmarking performance in U.S. ICUs. The accuracy of predictive models is dynamic and should be periodically retested. When accuracy deteriorates they should be revised and updated.

Recalibration of risk prediction models in a large multicenter cohort of admissions to adult, general critical care units in the United Kingdom*

OBJECTIVE

To assess the performance of published risk prediction models in common use in adult critical care in the United Kingdom and to recalibrate these models in a large representative database of critical care admissions.

DESIGN

Prospective cohort study.

SETTING

A total of 163 adult general critical care units in England, Wales, and Northern Ireland, during the period of December 1995 to August 2003.

PATIENTS

A total of 231,930 admissions, of which 141,106 met inclusion criteria and had sufficient data recorded for all risk prediction models.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

The published versions of the Acute Physiology and Chronic Health Evaluation (APACHE) II, APACHE II UK, APACHE III, Simplified Acute Physiology Score (SAPS) II, and Mortality Probability Models (MPM) II were evaluated for discrimination and calibration by means of a combination of appropriate statistical measures recommended by an expert steering committee. All models showed good discrimination (the c index varied from 0.803 to 0.832) but imperfect calibration. Recalibration of the models, which was performed by both the Cox method and re-estimating coefficients, led to improved discrimination and calibration, although all models still showed significant departures from perfect calibration.

CONCLUSIONS

Risk prediction models developed in another country require validation and recalibration before being used to provide risk-adjusted outcomes within a new country setting. Periodic reassessment is beneficial to ensure calibration is maintained.

Do we have reliable biochemical markers to predict the outcome of critical illness?

Current outcome prediction in critically ill patients relies on the art of clinical judgement and/or the science of prognostication using illness severity scores. The biochemical processes underlying critical illness have increasingly been unravelled. Several biochemical markers reflecting the process of inflammation, immune dysfunction, impaired tissue oxygenation and endocrine alterations have been evaluated for their predictive power in small subpopulations of critically ill patients. However, none of these parameters has been validated in large populations of unselected ICU patients as has been done for the illness severity and organ failure scores. A simple biochemical predictor of ICU mortality will probably remain elusive because the processes underlying critical illness are very complex and heterogeneous. Future prognostic models will need to be far more sophisticated.

Prediction of hospital mortality rates by admission laboratory tests

BACKGROUND

The aim of this study was to explore whether electronically retrieved laboratory data can predict mortality in internal medicine departments in a regional hospital.

METHODS

All 10,308 patients hospitalized in internal medicine departments over a 1-year period were included in the cohort. Nearly all patients had a complete blood count and basic clinical chemistries on admission. We used logistic regression analysis to predict the 573 deaths (5.6%), including all variables that added significantly to the model.

RESULTS

Eight laboratory variables and age significantly and independently contributed to a logistic regression model (area under the ROC curve, 88.7%). The odds ratio for the final model per quartile of risk was 6.44 (95% confidence interval, 5.42-7.64), whereas for age alone, the odds ratio per quartile was 2.01 (95% confidence interval, 1.84-2.19).

CONCLUSIONS

A logistic regression model including only age and electronically retrieved laboratory data highly predicted mortality in internal medicine departments in a regional hospital, suggesting that age and routine admission laboratory tests might be used to ensure a fair comparison when using mortality monitoring for hospital quality control.

SAPS 3--From evaluation of the patient to evaluation of the intensive care unit. Part 1: Objectives, methods and cohort description

OBJECTIVE

Risk adjustment systems now in use were developed more than a decade ago and lack prognostic performance. Objective of the SAPS 3 study was to collect data about risk factors and outcomes in a heterogeneous cohort of intensive care unit (ICU) patients, in order to develop a new, improved model for risk adjustment.

DESIGN

Prospective multicentre, multinational cohort study.

PATIENTS AND SETTING

A total of 19,577 patients consecutively admitted to 307 ICUs from 14 October to 15 December 2002.

MEASUREMENTS AND RESULTS

Data were collected at ICU admission, on days 1, 2 and 3, and the last day of the ICU stay. Data included sociodemographics, chronic conditions, diagnostic information, physiological derangement at ICU admission, number and severity of organ dysfunctions, length of ICU and hospital stay, and vital status at ICU and hospital discharge. Data reliability was tested with use of kappa statistics and intraclass-correlation coefficients, which were >0.85 for the majority of variables. Completeness of the data was also satisfactory, with 1 [0-3] SAPS II parameter missing per patient. Prognostic performance of the SAPS II was poor, with significant differences between observed and expected mortality rates for the overall cohort and four (of seven) defined regions, and poor calibration for most tested subgroups.

CONCLUSIONS

The SAPS 3 study was able to provide a high-quality multinational database, reflecting heterogeneity of current ICU case-mix and typology. The poor performance of SAPS II in this cohort underscores the need for development of a new risk adjustment system for critically ill patients.

Epidemiology and natural history of acute renal failure in the ICU

This article focuses on the incidence, risk factors, and mortality of acute renal failure (ARF) in critically ill patients. Accurate epidemiologic assessment of ARF is still a problem; as long as there is neither a uniquely accepted definition of ARF nor definitions for end points to measure, results will remain heterogeneous and hard to compare. Mortality of ARF has remained high throughout the last decades, despite further development of modern treatment modalities. This indicates that ARF is not just a matter of loss of organ function that can easily be replaced easily by extracorporeal therapies, but is a condition additionally accompanied by systemic consequences which significantly impact on prognosis.

Operational performance of validated physiologic scoring systems for predicting in-hospital mortality among critically ill emergency department patients*

OBJECTIVES

New Simplified Acute Physiology Score (SAPS) II, Morbidity Probability Model at admission (MPM0 II), and Logistic Organ Dysfunction System (LODS) have all demonstrated high accuracy for predicting mortality in intensive care unit populations. We tested the prognostic accuracy of these instruments for predicting mortality among a cohort of critically ill emergency department patients.

DESIGN

Secondary analysis of a randomized controlled trial.

SETTING

Urban, tertiary emergency department, census >100,000.

PATIENTS

Nontrauma emergency department patients admitted to an intensive care unit, aged >17 yrs, with initial emergency department vital signs consistent with shock (systolic blood pressure <100 mm Hg or shock index >1.0), and with agreement of two independent observers for at least one sign and symptom of inadequate tissue perfusion.

INTERVENTIONS

Emergency department variables needed for calculation of each scoring system were prospectively collected, and published formulas were used to calculate the probability of in-hospital death for each scoring system. The main outcome was actual in-hospital mortality. The area under the receiver operating characteristic curve was used to evaluate the predictive ability of each scoring system.

MEASUREMENTS AND MAIN RESULTS

Ninety-one of 202 patients (45%) were included. The mean age was 56 +/- 16 yrs, 42% were female, the mean initial systolic blood pressure was 84 +/- 13 mm Hg, and the average length of stay in the emergency department was 4.2 +/- 2.0 hrs. The in-hospital mortality rate was 21%. The area under the receiver operating characteristic curve for calculated probability of in-hospital mortality for SAPS II was 0.72 (95% confidence interval, 0.57-0.87), for MPM0 II 0.69 (95% confidence interval, 0.54-0.84), and for LODS 0.60 (95% confidence interval, 0.45-0.76).

CONCLUSIONS

Using variables available in the emergency department, three previously validated intensive care unit scoring systems demonstrated moderate accuracy for predicting in-hospital mortality.

Adverse effect on a referral intensive care unitʼs performance of accepting patients transferred from another intensive care unit*

OBJECTIVE

To determine whether observed and predicted mortality for intensive care unit (ICU) transfer admissions is different from non-ICU transfer admissions and how that might affect ICU performance evaluation.

DESIGN, SETTING, AND PATIENTS

We retrospectively analyzed the charts of 3,416 patients admitted to our tertiary referral ICU from January 1995 to December 2001 and evaluated the effect on our performance (based on the Simplified Acute Physiology Score II risk model) of accepting patients transferred from another hospital's ICU.

MAIN RESULTS

During the study period, 597 patients (17%) had been transferred from a non-ICU setting in another hospital (hospital transfer) and 408 (12%) from another hospital's ICU (ICU transfer). ICU mortality and standardized mortality ratios were significantly higher for ICU-transfer patients than for hospital-transfer or directly admitted patients: 34% vs. 23% vs. 17% (p < .0001) and 0.95 (95% confidence interval, 0.83-1.08), 0.82 (95% confidence interval, 0.71-0.95), and 0.62 (95% confidence interval, 0.55-0.68), respectively. ICU-transfer patients had 3.6-fold longer mean ICU stays and 1.9-fold longer durations of mechanical ventilation than directly admitted patients. Hospital-transfer (odds ratio = 1.89) and ICU-transfer patients (odds ratio = 2.41) had significantly higher mortality rates, even after adjustment for case mix and disease severity. Consequently, a benchmarking program adjusting only for these latter variables, but not admission source, would penalize our ICU by 39 excess deaths per 1,000 admissions as compared with another ICU admitting no transfer patients. Finally, patients transferred from the ward of another hospital had significantly higher mortality rates (odds ratio = 1.56) as compared with patients directly admitted from the ward of our hospital, confirming the "transfer effect" for this homogeneous patients' subgroup.

CONCLUSIONS

Admission source remains a strong and independent predictor of ICU death, despite adjustment for case mix and disease severity at ICU admission. Specifically, accepting numerous ICU-transfer patients, for whom the probability of ICU death is the most underestimated by a system adjusting only for case mix and disease severity, can adversely affect the evaluation of referral centers' performance. Future benchmarking and profiling systems should evaluate and adequately account for the ICU-transfer factor to provide healthcare payers and consumers with more accurate and valid information on the true performance of referral centers.

Re-evaluation and modification of the Stuivenberg Hospital Acute Renal Failure (SHARF) scoring system for the prognosis of acute renal failure: an independent multicentre, prospective study

BACKGROUND

A prognostic scoring system for hospital mortality in acute renal failure (Stuivenberg Hospital Acute Renal Failure, SHARF score) was developed in a single-centre study. The scoring system consists of two scores, for the time of diagnosis of acute renal failure (ARF) and for 48 h later, each originally based on four parameters (age, serum albumin, prothrombin time and heart failure). The scoring system was now tested and adapted in a prospective study.

METHODS

The study involved eight intensive care units. We studied 293 consecutive patients with ARF in 6 months. Their mortality was 50.5%. The causes of ARF were medical in 184 (63%) patients and surgical in 108 (37%). In the latter group, 74 (69%) patients underwent cardiac and 19 (18%) vascular surgery.

RESULTS

As the performance of the original SHARF scores was much lower in the multicentre study than in the original single-centre study, we re-analysed the multicentre data to customize the original model for the population studied. The independent variables were the score developed in the original study plus all additonal parameters that were significant on univariate analysis. The new multivariate analysis revealed an additional subset of three parameters for inclusion in the model (serum bilirubin, sepsis and hypotension). For the modified SHARF II score, r(2) was 0.27 at 0 and 0.33 at 48 h, respectively, the receiver operating characteristic (ROC) values were 0.82 and 0.83, and the Hosmer-Lemeshow goodness-of-fit P values were 0.19 and 0.05.

CONCLUSION

After customizing and by using two scoring moments, this prediction model for hospital mortality in ARF is useful in different settings for comparing groups of patients and centres, quality assessment and clinical trials. We do not recommend its use for individual patient prognosis.

[Scoring systems in the intensive care unit]

Scoring systems represent classification systems or point systems which have been designed for making quantitative statements regarding the severity of a disease, its prognosis, and its course. Furthermore, scores may serve the purposes of assessing therapies, of quality control and of quality assurance, and of an economic evaluation of intensive care. Like all measuring methods, scores are susceptible to failures and systematic mistakes. The clinical user should be well aware of these limitations. Generally, one would recommend only using scores which have been rigorously tested for their reliability, validity, and practicability. These include, but are not limited to, the updated versions of the APACHE, the SAPS, and the MPM. Although great strides have been made concerning development, verification, and clinical applicability, scores still exhibit a level of uncertainty which precludes their use in individual patients. Frequently, it may be of benefit to combine the more general scores with one or several organ dysfunction scores to determine the extent of functional impairment of specific organs. If, however, well-trained medical personnel apply tried and tested scoring systems, intensive care units will definitely gain a lot from it.

Long-term survival following intensive care: subgroup analysis and comparison with the general population

This study aimed to compare the very long-term survival of critically ill patients with that of the general population, and examine the association among age, sex, admission diagnosis, APACHE II score and mortality. In a retrospective observational cohort study of prospectively gathered data, 2104 adult patients admitted to the intensive care unit (ICU) of a teaching hospital in Glasgow from 1985 to 1992, were followed until 1997. Vital status at five years was compared with that of an age- and sex-matched Scottish population. Five-year mortality for the ICU patients was 47.1%, 3.4 times higher than that of the general population. For those surviving intensive care the five-year mortality was 33.4%. Mortality was greater than that of the general population for four years following intensive care unit admission (95% confidence interval included 1.0 at four years). Multivariate analysis showed that risk factors for mortality in those admitted to ICU were age, APACHE II score on admission and diagnostic category. Mortality was higher for those admitted with haematological (87.5%) and neurological diseases (61.7%) and septic shock (62.9%). A risk score was produced: Risk Score = 10 (age hazard ratio + APACHE II hazard ratio + diagnosis hazard ratio). None of the patients with a risk score > 100 survived more than five years and for those who survived to five years the mean risk score was 57. Long-term survival following intensive care is not only related to age and severity of illness but also diagnostic category. The risk of mortality in survivors of critical illness matches that of the normal population after four years. Age, severity of illness and diagnosis can be combined to provide an estimate of five-year survival.

External validation of the SAPS II, APACHE II and APACHE III prognostic models in South England: a multicentre study

OBJECTIVE

External validation of three prognostic models in adult intensive care patients in South England. DESIGN. Prospective cohort study.

SETTING

Seventeen intensive care units (ICU) in the South West Thames Region in South England.

PATIENTS AND PARTICIPANTS

Data of 16646 patients were analysed.

INTERVENTIONS

None.

MEASUREMENTS AND RESULTS

We compared directly the predictive accuracy of three prognostic models (SAPS II, APACHE II and III), using formal tests of calibration and discrimination. The external validation showed a similar pattern for all three models tested: good discrimination, but imperfect calibration. The areas under the receiver operating characteristics (ROC) curves, used to test discrimination, were 0.835 and 0.867 for APACHE II and III, and 0.852 for the SAPS II model. Model calibration was assessed by Lemeshow-Hosmer C-statistics and was Chi(2 )=232.1 for APACHE II, Chi(2 )=443.3 for APACHE III and Chi(2 )=287.5 for SAPS II.

CONCLUSIONS

Disparity in case mix, a higher prevalence of outcome events and important unmeasured patient mix factors are possible sources for the decay of the models' predictive accuracy in our population. The lack of generalisability of standard prognostic models requires their validation and re-calibration before they can be applied with confidence to new populations. Customisation of existing models may become an important strategy to obtain authentic information on disease severity, which is a prerequisite for reliably measuring and comparing the quality and cost of intensive care.

Identifying quality outliers in a large, multiple-institution database by using customized versions of the Simplified Acute Physiology Score II and the Mortality Probability Model II0

OBJECTIVE

To assess whether customized versions of the Simplified Acute Physiology Score (SAPS) II and the Mortality Probability Model (MPM) II0 agree on the identity of intensive care unit quality outliers within a multiple-center database.

DESIGN

Retrospective database analysis.

SETTING AND PATIENTS

Patient subset of the Project IMPACT database consisting of 39,617 adult patients admitted to surgical, medical, and mixed surgical-medical intensive care units at 54 hospitals between 1995 and 1999 who met inclusion criteria for SAPS II and MPM II0.

INTERVENTIONS

Customized versions of SAPS II and MPM II0 were obtained by fitting new logistic regressions to the data by using the risk score as the independent variable and outcome at hospital discharge as the dependent variable. The data set was divided randomly into a training set and a validation set. Each model was customized by using the training set; model performance was then assessed in the validation set by using the area under the receiver operating characteristic curve and the Hosmer-Lemeshow statistic. The final models were based on the entire data set. The level of agreement between the customized models on the identity of quality outliers was evaluated by using kappa analysis.

MEASUREMENTS AND MAIN RESULTS

Both customized models exhibited good discrimination and good calibration in this database. The area under the receiver operating characteristic curve was 0.83 for MPM II0 and 0.872 for SAPS II following model customization. The Hosmer-Lemeshow statistic was 12.3 ( >.14) for MPM II0, and 8.17 (p >.42) for SAPS II, after customization. Kappa analysis showed only fair agreement between the two customized models with regard to the identity of the quality outliers: kappa = 0.44 (95% confidence interval, 0.24, 0.65).

CONCLUSIONS

Customization of SAPS II and MPM II0 to the Project IMPACT database resulted in well-calibrated models. Despite this, the models exhibited only a moderate level of agreement in which hospitals were designated as quality outliers. Seventeen of the 54 hospitals were categorized differently depending on which of the two scoring systems was used. Therefore, the rating of quality of care appears, in part, to be a function of the prediction model used.

Rating the quality of intensive care units: is it a function of the intensive care unit scoring system?

OBJECTIVE

Intensive care units (ICUs) use severity-adjusted mortality measures such as the standardized mortality ratio to benchmark their performance. Prognostic scoring systems such as Acute Physiology and Chronic Health Evaluation (APACHE) II, Simplified Acute Physiology Score II, and Mortality Probability Model II0 permit performance-based comparisons of ICUs by adjusting for severity of disease and case mix. Whether different risk-adjustment methods agree on the identity of ICU quality outliers within a single database has not been previously investigated. The objective of this study was to determine whether the identity of ICU quality outliers depends on the ICU scoring system used to calculate the standardized mortality ratio.

DESIGN, SETTING, PATIENTS

Retrospective cohort study of 16,604 patients from 32 hospitals based on the outcomes database (Project IMPACT) created by the Society of Critical Care Medicine. The ICUs were a mixture of medical, surgical, and mixed medical-surgical ICUs in urban and nonurban settings. Standardized mortality ratios for each ICU were calculated using APACHE II, Simplified Acute Physiology Score II, and Mortality Probability Model II. ICU quality outliers were defined as ICUs whose standardized mortality ratio was statistically different from 1. Kappa analysis was used to determine the extent of agreement between the scoring systems on the identity of hospital quality outliers. The intraclass correlation coefficient was calculated to estimate the reliability of standardized mortality ratios obtained using the three risk-adjustment methods.

MEASUREMENTS AND MAIN RESULTS

Kappa analysis showed fair to moderate agreement among the three scoring systems in identifying ICU quality outliers; the intraclass correlation coefficient suggested moderate to substantial agreement between the scoring systems. The majority of ICUs were classified as high-performance ICUs by all three scoring systems. All three scoring systems exhibited good discrimination and poor calibration in this data set.

CONCLUSION

APACHE II, Simplified Acute Physiology Score II, and Mortality Probability Model II0 exhibit fair to moderate agreement in identifying quality outliers. However, the finding that most ICUs in this database were judged to be high-performing units limits the usefulness of these models in their present form for benchmarking.