New issues in severity scoring: interfacing the ICU and evaluating it

Development and validation of prognostic machine learning models for short- and long-term mortality among acutely admitted patients based on blood tests

Several scores predicting mortality at the emergency department have been developed. However, all with shortcomings either simple and applicable in a clinical setting, with poor performance, or advanced, with high performance, but clinically difficult to implement. This study aimed to explore if machine learning algorithms could predict all-cause short- and long-term mortality based on the routine blood test collected at admission.

METHODS

We analyzed data from a retrospective cohort study, including patients > 18 years admitted to the Emergency Department (ED) of Copenhagen University Hospital Hvidovre, Denmark between November 2013 and March 2017. The primary outcomes were 3-, 10-, 30-, and 365-day mortality after admission. PyCaret, an automated machine learning library, was used to evaluate the predictive performance of fifteen machine learning algorithms using the area under the receiver operating characteristic curve (AUC).

RESULTS

Data from 48,841 admissions were analyzed, of these 34,190 (70%) were randomly divided into training data, and 14,651 (30%) were in test data. Eight machine learning algorithms achieved very good to excellent results of AUC on test data in a of range 0.85-0.93. In prediction of short-term mortality, lactate dehydrogenase (LDH), leukocyte counts and differentials, Blood urea nitrogen (BUN) and mean corpuscular hemoglobin concentration (MCHC) were the best predictors, whereas prediction of long-term mortality was favored by age, LDH, soluble urokinase plasminogen activator receptor (suPAR), albumin, and blood urea nitrogen (BUN).

CONCLUSION

The findings suggest that measures of biomarkers taken from one blood sample during admission to the ED can identify patients at high risk of short-and long-term mortality following emergency admissions.

Scoring systems in critically ill: Which one to use in cancer patients?

BACKGROUND

Scoring systems have not been evaluated in oncology patients. We aimed to assess the performance of Acute Physiology and Chronic Health Evaluation (APACHE) II, APACHE III, APACHE IV, Simplified Acute Physiology Score (SAPS) II, SAPS III, Mortality Probability Model (MPM) II₀ and Sequential Organ Failure Assessment (SOFA) score in critically ill oncology patients.

AIM

To compare the efficacy of seven commonly employed scoring systems to predict outcomes of critically ill cancer patients.

METHODS

We conducted a retrospective analysis of 400 consecutive cancer patients admitted in the medical intensive care unit over a two-year period. Primary outcome was hospital mortality and the secondary outcome measure was comparison of various scoring systems in predicting hospital mortality.

RESULTS

In our study, the overall intensive care unit and hospital mortality was 43.5% and 57.8%, respectively. All of the seven tested scores underestimated mortality. The mortality as predicted by MPM II₀ predicted death rate (PDR) was nearest to the actual mortality followed by that predicted by APACHE II, with a standardized mortality rate (SMR) of 1.305 and 1.547, respectively. The best calibration was shown by the APACHE III score (χ² = 4.704, P = 0.788). On the other hand, SOFA score (χ² = 15.966, P = 0.025) had the worst calibration, although the difference was not statistically significant. All of the seven scores had acceptable discrimination with good efficacy however, SAPS III PDR and MPM II₀ PDR (AUROC = 0.762), had a better performance as compared to others. The correlation between the different scoring systems was significant (P < 0.001).

CONCLUSION

All the severity scores were tested under-predicted mortality in the present study. As the difference in efficacy and performance was not statistically significant, the choice of scoring system used may depend on the ease of use and local preferences.

A machine learning approach for mortality prediction only using non-invasive parameters

At present, the traditional scoring methods generally utilize laboratory measurements to predict mortality. It results in difficulties of early mortality prediction in the rural areas lack of professional laboratorians and medical laboratory equipment. To improve the efficiency, accuracy, and applicability of mortality prediction in the remote areas, a novel mortality prediction method based on machine learning algorithms is proposed, which only uses non-invasive parameters readily available from ordinary monitors and manual measurement. A new feature selection method based on the Bayes error rate is developed to select valuable features. Based on non-invasive parameters, four machine learning models were trained for early mortality prediction. The subjects contained in this study suffered from general critical diseases including but not limited to cancer, bone fracture, and diarrhea. Comparison tests among five traditional scoring methods and these four machine learning models with and without laboratory measurement variables are performed. Only using the non-invasive parameters, the LightGBM algorithms have an excellent performance with the largest accuracy of 0.797 and AUC of 0.879. There is no apparent difference between the mortality prediction performance with and without laboratory measurement variables for the four machine learning methods. After reducing the number of feature variables to no more than 50, the machine learning models still outperform the traditional scoring systems, with AUC higher than 0.83. The machine learning approaches only using non-invasive parameters achieved an excellent mortality prediction performance and can equal those using extra laboratory measurements, which can be applied in rural areas and remote battlefield for mortality risk evaluation. Graphical abstract.

Data Quality Improvement in Clinical Databases Using Statistical Quality Control: Review and Case Study

Ensuring the quality of data being collected in clinical and medical contexts is a concern for data managers and users. Quality assurance frameworks, systematic audits, and correction procedures have been proposed to enhance the accuracy and completeness of databases. Following an overview of the undertaken approaches, particularly statistical methods, the authors promote acceptance sampling plans (ASPs) and statistical process control (SPC) tools, including control charts and root cause analysis, as the technical core of the data quality improvement mechanism. They review ASP and SPC techniques and discuss their implementation in data quality evaluation and improvement. Two case studies are presented in which the authors apply some of the techniques to databases maintained by a local hospital. Finally, guidelines are proposed for which techniques are appropriate with regard to dataflow and database specifications.

Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit

BACKGROUND

The Simplified Acute Physiology Score (SAPS) 3 has recently been developed, but not yet validated in surgical intensive care unit (ICU) patients. We compared the performance of SAPS 3 with SAPS II and the Acute Physiology and Chronic Health Evaluation (APACHE) II score in surgical ICU patients.

METHODS

Prospectively collected data from all patients admitted to a German university hospital postoperative ICU between August 2004 and December 2005 were analysed. The probability of ICU mortality was calculated for SAPS II, APACHE II, adjusted APACHE II (adj-APACHE II), SAPS 3, and SAPS 3 customized for Europe [C-SAPS3 (Eu)] using standard formulas. To improve calibration of the prognostic models, a first-level customization was performed, using logistic regression on the original scores, and the corresponding probability of ICU death was calculated for the customized scores (C-SAPS II, C-SAPS 3, and C-APACHE II).

RESULTS

The study included 1851 patients. Hospital mortality was 9%. Hosmer and Lemeshow statistics showed poor calibration for SAPS II, APACHE II, adj-APACHE II, SAPS 3, and C-SAPS 3 (Eu), but good calibration for C-SAPS II, C-APACHE II, and C-SAPS 3. Discrimination was generally good for all models [area under the receiver operating characteristic curve ranged from 0.78 (C-APACHE II) to 0.89 (C-SAPS 3)]. The C-SAPS 3 score appeared to have the best calibration curve on visual inspection.

CONCLUSIONS

In this group of surgical ICU patients, the performance of SAPS 3 was similar to that of APACHE II and SAPS II. Customization improved the calibration of all prognostic models.

Review article: scoring systems for assessing prognosis in critically ill adult cirrhotics

BACKGROUND

Cirrhotic patients admitted to intensive care units (ICU) still have poor outcomes. Some current ICU prognostic models [Acute Physiology and Chronic Health Evaluation (APACHE), Organ System Failure (OSF) and Sequential Organ Failure Assessment (SOFA)] were used to stratify cirrhotics into risk categories, but few cirrhotics were included in the original model development. Liver-specific scores [Child-Turcotte-Pugh (CTP) and model for end-stage liver disease (MELD)] could be useful in this setting.

AIM

To evaluate whether ICU prognostic models perform better compared with liver-disease specific ones in cirrhotics admitted to ICU.

METHODS

We performed a structured literature review identifying clinical studies focusing on prognosis and risk factors for mortality in adult cirrhotics admitted to ICU.

RESULTS

We found 21 studies (five solely dealing with gastrointestinal bleeding) published during the last 20 years (54-420 patients in each). APACHE II and III, SOFA and OSF had better discrimination for correctly predicting death compared with the CTP score. The MELD score was evaluated only in one study and had good predictive accuracy [receiver operator characteristic (ROC) curve: 0.81). Organ dysfunction models (OSF, SOFA) were superior compared with APACHE II and III (ROC curve: range 0.83-0.94 vs. 0.66-0.88 respectively). Cardiovascular, liver and renal system dysfunction were more frequently independently associated with mortality.

CONCLUSIONS

General-ICU models had better performance in cirrhotic populations compared with CTP score; OSF and SOFA had the best predictive ability. Further prospective and validation studies are needed.

A nationwide survey of intensive care unit discharge practices

OBJECTIVE

To describe intensive care unit (ICU) discharge practices, examine factors associated with physicians' discharge decisions, and explore ICU and hospital characteristics and clinical determinants associated with the discharge process.

DESIGN

Survey in adult ICUs affiliated with the Swiss Society of Intensive Care Medicine.

INTERVENTIONS

Questionnaire inquiring about ICU structure and organization mailed to 73 medical directors. Level of monitoring, intravenous medications, and physiological variables were proposed as elements of discharge decision. Five clinical situations were presented with request to assign a discharge disposition.

MEASUREMENTS AND RESULTS

Fifty-five ICUs participated, representing 75% of adult Swiss ICUs. Responsibility for patient management was assigned in 91% to the ICU team directing patient care. Only 22% of responding centers used written discharge guidelines. One-half of the respondents considered at least 10 of 15 proposed criteria to decide patient discharge. ICUs in central referral hospitals used fewer criteria than community and private hospitals. The availability of intermediate care units was significantly greater in university hospitals. The ICU director's level of experience was not associated with the number of criteria used. In the five clinical scenarios there was wide variation in discharge decision.

CONCLUSIONS

Our data indicate that there is marked heterogeneity in ICUs discharge practices, and that discharge decisions may be influenced by institutional factors. University teaching hospitals had more intermediate care facilities available. Written discharge guidelines were not widely used.

SAPS�II revisited

OBJECTIVE

To construct and validate an update of the Simplified Acute Physiology Score II (SAPS II) for the evaluation of clinical performance of Intensive Care Units (ICU).

DESIGN AND SETTING

Retrospective analysis of prospectively collected multicenter data in 32 ICUs located in the Paris area belonging to the Cub-Rea database and participating in a performance evaluation project.

PATIENTS

33,471 patients treated between 1999 and 2000.

MEASUREMENTS AND RESULTS

Two logistic regression models based on SAPS II were developed to estimate in-hospital mortality among ICU patients. The second model comprised reevaluation of original items of SAPS II and integration of the preadmission location and chronic comorbidity. Internal and external validation were performed. In the two validation samples the most complex model had better calibration than the original SAPS II for in-hospital mortality but its discrimination was not significantly higher (area under ROC curve 0.89 vs. 0.87 for SAPS II). Second-level customization and integration of new items improved uniformity of fit for various categories of patients except for diagnosis-related groups. The rank order of ICUs was modified according to the model used.

CONCLUSIONS

The overall performance of SAPS II derived models was good, even in the context of a community cohort and routinely gathered data. However, one-half the variation of outcome remains unexplained after controlling for admission characteristics, and uniformity of prediction across diagnostic subgroups was not achieved. Differences in case-mix still limit comparisons of quality of care.