Outcome prediction in critical care: physicians' prognoses vs. scoring systems

Accuracy of physician and nurse predictions for 28-day prognosis in ICU: a single center prospective study

The proportion of correctly predicted prognoses and factors associated with prediction accuracy are unknown. The objective of this study was to explore the accuracy of physician and nurse predictions of 28-day mortality in the ICU. This was a prospective observational single-center study. All medical staff in the ICU have access to patient data, can communicate with patients or clients, and can independently predict the prognosis of patients within 24 h of patient admission. The only question of the questionnaire survey was: What is the patient's outcome on day 28 (alive or death)? There were 2155 questionnaires completed by 18 physicians and 1916 submitted by 15 nurses. In the 312 patients included, the 28-day mortality rates were predicted by physicians and nurses. The overall proportion of correct prognosis prediction was 90.1% for physicians and 64.4% for nurses (P = 0.000). There was no significant difference in the overall correct proportion and average correct proportion among physicians with different seniority levels. The overall correct proportion and average correct proportion increased among nurses with seniority. Physicians in the ICU can moderately predict 28-day mortality in critically ill patients. Nurses with a seniority of less than 10 years in ICU cannot accurately predict 28-day mortality in critically ill patients. However, the accuracy of nurses' prediction of patients' 28-day prognosis increased with their seniority in the ICU.

Comparing In-Hospital Mortality Prediction by Senior Emergency Resident's Judgment and Prognostic Models in the Emergency Department

Background

A comparison of emergency residents' judgments and two derivatives of the Sequential Organ Failure Assessment (SOFA), namely, the mSOFA and the qSOFA, was conducted to determine the accuracy of predicting in-hospital mortality among critically ill patients in the emergency department (ED).

Methods

A prospective cohort research was performed on patients over 18 years of age presented to the ED. We used logistic regression to develop a model for predicting in-hospital mortality by using qSOFA, mSOFA, and residents' judgment scores. We compared the accuracy of prognostic models and residents' judgment in terms of the overall accuracy of the predicted probabilities (Brier score), discrimination (area under the ROC curve), and calibration (calibration graph). Analyses were carried out using R software version R-4.2.0.

Results

In the study, 2,205 patients with median age of 64 (IQR: 50-77) years were included. There were no significant differences between the qSOFA (AUC 0.70; 95% CI: 0.67-0.73) and physician's judgment (AUC 0.68; 0.65-0.71). Despite this, the discrimination of mSOFA (AUC 0.74; 0.71-0.77) was significantly higher than that of the qSOFA and residents' judgments. Additionally, the AUC-PR of mSOFA, qSOFA, and emergency resident's judgments was 0.45 (0.43-0.47), 0.38 (0.36-0.40), and 0.35 (0.33-0.37), respectively. The mSOFA appears stronger in terms of overall performance: 0.13 vs. 0.14 and 0.15. All three models showed good calibration.

Conclusion

The performance of emergency residents' judgment and the qSOFA was the same in predicting in-hospital mortality. However, the mSOFA predicted better-calibrated mortality risk. Large-scale studies should be conducted to determine the utility of these models.

Predicting the Risk of Mortality in Children using a Fuzzy-Probabilistic Hybrid Model

Introduction

The mortality risk in children admitted to Pediatric Intensive Care Units (PICU) is usually estimated by means of validated scales, which only include objective data among their items. Human perceptions may also add relevant information to prognosticate the risk of death, and the tool to use this subjective data is fuzzy logic. The objective of our study was to develop a mathematical model to predict mortality risk based on the subjective perception of PICU staff and to evaluate its accuracy compared to validated scales.

Methods

A prospective observational study in two PICUs (one in Spain and another in Latvia) was performed. Children were consecutively included regardless of the cause of admission along a two-year period. A fuzzy set program was developed for the PICU staff to record the subjective assessment of the patients' mortality risk expressed through a short range and a long range, both between 0% and 100%. Pediatric Index of Mortality 2 (PIM2) and Therapeutic Intervention Scoring System 28 (TISS28) were also prospectively calculated for each patient. Subjective and objective predictions were compared using the logistic regression analysis. To assess the prognostication ability of the models a stratified B-random K-fold cross-validation was performed.

Results

Five hundred ninety-nine patients were included, 308 in Spain (293 survivors, 15 nonsurvivors) and 291 in Latvia (282 survivors, 9 nonsurvivors). The best logistic classification model for subjective information was the one based on MID (midpoint of the short range), whereas objective information was the one based on PIM2. Mortality estimation performance was 86.3% for PIM2, 92.6% for MID, and the combination of MID and PIM2 reached 96.4%.

Conclusions

Subjective assessment was as useful as validated scales to estimate the risk of mortality. A hybrid model including fuzzy information and probabilistic scales (PIM2) seems to increase the accuracy of prognosticating mortality in PICU.

The simple observational critical care studies: estimations by students, nurses, and physicians of in-hospital and 6-month mortality

Background

Prognostic assessments of the mortality of critically ill patients are frequently performed in daily clinical practice and provide prognostic guidance in treatment decisions. In contrast to several sophisticated tools, prognostic estimations made by healthcare providers are always available and accessible, are performed daily, and might have an additive value to guide clinical decision-making. The aim of this study was to evaluate the accuracy of students’, nurses’, and physicians’ estimations and the association of their combined estimations with in-hospital mortality and 6-month follow-up.

Methods

The Simple Observational Critical Care Studies is a prospective observational single-center study in a tertiary teaching hospital in the Netherlands. All patients acutely admitted to the intensive care unit were included. Within 3 h of admission to the intensive care unit, a medical or nursing student, a nurse, and a physician independently predicted in-hospital and 6-month mortality. Logistic regression was used to assess the associations between predictions and the actual outcome; the area under the receiver operating characteristics (AUROC) was calculated to estimate the discriminative accuracy of the students, nurses, and physicians.

Results

In 827 out of 1,010 patients, in-hospital mortality rates were predicted to be 11%, 15%, and 17% by medical students, nurses, and physicians, respectively. The estimations of students, nurses, and physicians were all associated with in-hospital mortality (OR 5.8, 95% CI [3.7, 9.2], OR 4.7, 95% CI [3.0, 7.3], and OR 7.7 95% CI [4.7, 12.8], respectively). Discriminative accuracy was moderate for all students, nurses, and physicians (between 0.58 and 0.68). When more estimations were of non-survival, the odds of non-survival increased (OR 2.4 95% CI [1.9, 3.1]) per additional estimate, AUROC 0.70 (0.65, 0.76). For 6-month mortality predictions, similar results were observed.

Conclusions

Based on the initial examination, students, nurses, and physicians can only moderately predict in-hospital and 6-month mortality in critically ill patients. Combined estimations led to more accurate predictions and may serve as an example of the benefit of multidisciplinary clinical care and future research efforts.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13054-021-03809-w.

SURvival PRediction In SEverely Ill Patients Study-The Prediction of Survival in Critically Ill Patients by ICU Physicians

The surprise question, "Would I be surprised if this patient died in the next 12 months?" is a tool to identify patients at high risk of death in the next year. Especially in the situation of an ICU admission, it is important to recognize patients who could and could not have the benefits of an intensive treatment in the ICU department.

Design and Setting

A single-center, prospective, observational cohort study was conducted between April 2013 and April 2018, in ICU Gelre hospitals, location Apeldoorn.

Patients

A total of 3,140 patients were included (57% male) with a mean age of 63.5 years. Seven-hundred thirteen patients (23%) died within 1 year.

Interventions

The physician answered three different surprise question's with either "yes" or "no": "I expect that the patient is going to survive the ICU admission" (surprise question 1), "I expect that the patient is going to survive the hospital stay" (surprise question 2), and "I expect that the patient is going to survive one year after ICU admission" (surprise question 3). We tested positive and negative predicted values of the surprise questions, the mean accuracy of the surprise questions, and kappa statistics.

Measurements and Main Results

The positive and negative predictive values of the surprise questions for ICU admission, hospital admission, and 1-year survival were, respectively, 64%/94%, 59%/92%, and 60%/86%. Accordingly, the mean accuracy and kappa statistics were 93% (95% CI, 92-94%), κ equals to 0.43, 89% (95% CI, 88-90%), κ equals to 0.40, and 81% (95% CI, 80-82%), κ equals to 0.43.

Conclusions

The frequently overlooked simple and cheap surprise question is probably an useful tool to evaluate the prognosis of acutely admitted critically ill patients.

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.

SCORING SYSTEMS DO NOT ACCURATELY PREDICT OUTCOME FOLLOWING ABDOMINAL AORTIC ANEURYSM REPAIR

BACKGROUND

Abdominal aortic aneurysm repair is associated with significant morbidity and mortality. This study aims to evaluate the efficiency of scoring systems in a group of patients undergoing abdominal aortic aneurysm repair.

METHODS

A prospective study of 152 patients undergoing aneurysm repair was conducted. Each patient was scored according to the Acute Physiology and Chronic Health Evaluation II, Physiological and Operative Severity Score for the enUmeration of Mortality and Morbidity and Simplified Acute Physiology Score II systems. The predicted mortality for each patient was calculated. Chi(2) analysis was carried out to determine the accuracy of mortality predictions. Receiver-operator curves were drawn to compare scoring systems in terms of sensitivity and specificity.

RESULTS

In the elective aneurysm repair group, all scoring systems tended to overestimate mortality. Receiver-operator curves showed inaccuracies in identifying patients who were at high risk from surgery. In contrast, predicted mortalities underestimated the true death rate among the ruptured aneurysm group. Receiver-operator curves showed better efficiency of scoring systems in the ruptured aneurysm group than in the elective repair group. There was no significant correlation between predicted and observed mortalities in either group.

CONCLUSION

In this study, all systems showed significant errors when predicting mortality. Therefore, although useful as an audit tool, scoring systems should not be used on an individual basis to guide treatment and assess prognosis after surgery.