Prognostic models based on administrative data alone inadequately predict the survival outcomes for critically ill patients at 180 days post-hospital discharge

Early prediction of mortality upon intensive care unit admission

BACKGROUND

We aimed to develop and validate models for predicting intensive care unit (ICU) mortality of critically ill adult patients as early as upon ICU admission.

METHODS

Combined data of 79,657 admissions from two teaching hospitals' ICU databases were used to train and validate the machine learning models to predict ICU mortality upon ICU admission and at 24 h after ICU admission by using logistic regression, gradient boosted trees (GBT), and deep learning algorithms.

RESULTS

In the testing dataset for the admission models, the ICU mortality rate was 7%, and 38.4% of patients were discharged alive or dead within 1 day of ICU admission. The area under the receiver operating characteristic curve (0.856, 95% CI 0.845-0.867) and area under the precision-recall curve (0.331, 95% CI 0.323-0.339) were the highest for the admission GBT model. The ICU mortality rate was 17.4% in the 24-hour testing dataset, and the performance was the highest for the 24-hour GBT model.

CONCLUSION

The ADM models can provide crucial information on ICU mortality as early as upon ICU admission. 24 H models can be used to improve the prediction of ICU mortality for patients discharged more than 1 day after ICU admission.

Reliability of comorbidity scores derived from administrative data in the tertiary hospital intensive care setting: a cross-sectional study

Background

Hospital reporting systems commonly use administrative data to calculate comorbidity scores in order to provide risk-adjustment to outcome indicators.

Objective

We aimed to elucidate the level of agreement between administrative coding data and medical chart review for extraction of comorbidities included in the Charlson Comorbidity Index (CCI) and Elixhauser Index (EI) for patients admitted to the intensive care unit of a university-affiliated hospital.

Method

We conducted an examination of a random cross-section of 100 patient episodes over 12 months (July 2012 to June 2013) for the 19 CCI and 30 EI comorbidities reported in administrative data and the manual medical record system. CCI and EI comorbidities were collected in order to ascertain the difference in mean indices, detect any systematic bias, and ascertain inter-rater agreement.

Results

We found reasonable inter-rater agreement (kappa (κ) coefficient ≥0.4) for cardiorespiratory and oncological comorbidities, but little agreement (κ<0.4) for other comorbidities. Comorbidity indices derived from administrative data were significantly lower than from chart review: −0.81 (95% CI − 1.29 to − 0.33; p=0.001) for CCI, and −2.57 (95% CI −4.46 to −0.68; p=0.008) for EI.

Conclusion

While cardiorespiratory and oncological comorbidities were reliably coded in administrative data, most other comorbidities were under-reported and an unreliable source for estimation of CCI or EI in intensive care patients. Further examination of a large multicentre population is required to confirm our findings.

Perinatal Risk Factors and Outcome Coding in Clinical and Administrative Databases

BACKGROUND AND OBJECTIVES

Administrative databases may allow true population-based studies and quality improvement endeavors, but the accuracy of billing codes for capturing key risk factors and outcomes needs to be assessed. We sought to describe the performance of a statewide administrative database and the clinical database from the California Perinatal Quality Care Collaborative (CPQCC).

METHODS

This population-based retrospective cohort study linked key perinatal risk factors and outcomes from the 133-unit CPQCC database to relevant billing codes from administrative maternal and newborn inpatient discharge records, for 50 631 infants born from 2006 to 2012. Using the CPQCC record as the gold standard, we calculated the positive predictive value, negative predictive value, and Matthews correlation coefficient for each item, then evaluated comparative performance across units.

RESULTS

The Matthews correlation coefficient was highest (>0.7; strong positive correlation) for multiple delivery, Cesarean delivery, very low birth weight, maternal hypertension, maternal diabetes, patent ductus arteriosus, in-hospital death, patent ductus arteriosus and retinopathy of prematurity surgeries, extracorporeal life support, and intraventricular hemorrhage. Maternal chorioamnionitis, fetal distress, retinopathy of prematurity staging, chronic lung disease, and pneumothorax were the least reliably coded. Maternal factors and delivery details were more reliably coded in the maternal inpatient record than the newborn inpatient record.

CONCLUSIONS

Several important perinatal risk factors and outcomes are highly congruent between these administrative and clinical databases. Several subjective risk factors and outcomes are appropriate targets for data improvement initiatives. The ability for timely extraction of administrative inpatient data will be key to their usefulness in quality metrics.

Assessing contemporary intensive care unit outcome: development and validation of the Australian and New Zealand Risk of Death admission model

The Australian and New Zealand Risk of Death (ANZROD) model currently used for benchmarking intensive care units (ICUs) in Australia and New Zealand utilises physiological data collected up to 24 hours after ICU admission to estimate the risk of hospital mortality. This study aimed to develop the Australian and New Zealand Risk of Death admission (ANZROD₀) model to predict hospital mortality using data available at presentation to ICU and compare its performance with the ANZROD in Australian and New Zealand hospitals. Data pertaining to all ICU admissions between 1 January 2006 and 31 December 2015 were extracted from the Australian and New Zealand Intensive Care Society Adult Patient Database. Hospital mortality was modelled using logistic regression with development (two-thirds) and validation (one-third) datasets. All predictor variables available at ICU admission were considered for inclusion in the ANZROD₀ model. Model performance was assessed using Brier score, standardised mortality ratio and area under the receiver operating characteristic curve. The relationship between ANZROD₀ and ANZROD predicted risk of death was assessed using linear regression. After standard exclusions, 1,097,416 patients were available for model development and validation. Observed mortality was 9.5%. Model performance measures (Brier score, standardised mortality ratio and area under the receiver operating characteristic curve) for the ANZROD₀ and ANZROD in the validation dataset were 0.069, 1.0 and 0.853; 0.057, 1.0 and 0.909, respectively. There was a strong positive correlation between the mortality predictions with an overall R² of 0.73. We found that the ANZROD₀ model had acceptable calibration and discrimination. Predictions from the models had high correlations in all major diagnostic groups, with the exception of cardiac surgery and possibly trauma and sepsis.

Predicting intensive care and hospital outcome with the Dalhousie Clinical Frailty Scale: a pilot assessment

Frailty may help to predict intensive care unit (ICU) patient outcome. The Dalhousie Clinical Frailty Scale (DCFS) is validated to assess frailty in ambulatory settings but has not been investigated in Australian ICUs. We conducted a prospective three-month study of patients admitted to a tertiary level ICU. Within 24 hours of ICU admission, the next of kin or nurse in charge assigned a DCFS score to the patient. Data were obtained to assess the association between frailty and patient outcome. The DCFS score was completed in 205 of 348 (59%) of eligible patient admissions. The mean DCFS score was 3.2 (±1.6). Overall frailty (DCFS>4) occurred in 28 of 205 patients (13%, confidence interval 9% to 17%), 13 of 93 (15%, confidence interval 10% to 25%) in patients aged >65 years and 5 of 11 (45%, confidence interval 21% to 71%) in those>85 years. Patients with chronic liver disease (P<0.001) and end-stage renal failure (P=0.009) were more likely to be frail. The DCFS score was not significantly associated with ICU or hospital mortality: odds ratio 0.98 (95% confidence interval 0.6 to 1.6) and odds ratio 1.07 (95% confidence interval 0.8 to 1.4), respectively. However, after adjustment for illness severity and requirement for palliative care, the DCFS score was significantly associated with increased (log) hospital length-of-stay (P=0.04) and age (P=0.001). Approximately 1 in 10 ICU patients were frail and this frequency increased with age. The DCFS was associated with patient age and comorbidities and potentially predicts increased hospital length-of-stay but not other outcomes. Strategies to improve compliance with DCFS completion are needed.

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.

Addressing electronic clinical information in the construction of quality measures

Electronic health records (EHR) and registries play a central role in health care and provide access to detailed clinical information at the individual, institutional, and population level. Use of these data for clinical quality/performance improvement and cost management has been a focus of policy initiatives over the past decade. The Children's Health Insurance Program Reauthorization Act of 2009 (CHIPRA)-mandated Pediatric Quality Measurement Program supports development and testing of quality measures for children on the basis of electronic clinical information, including de novo measures and respecification of existing measures designed for other data sources. Drawing on the experience of Centers of Excellence, we review both structural and pragmatic considerations in e-measurement. The presence of primary observations in EHR-derived data make it possible to measure outcomes in ways that are difficult with administrative data alone. However, relevant information may be located in narrative text, making it difficult to interpret. EHR systems are collecting more discrete data, but the structure, semantics, and adoption of data elements vary across vendors and sites. EHR systems also differ in ability to incorporate pediatric concepts such as variable dosing and growth percentiles. This variability complicates quality measurement, as do limitations in established measure formats, such as the Quality Data Model, to e-measurement. Addressing these challenges will require investment by vendors, researchers, and clinicians alike in developing better pediatric content for standard terminologies and data models, encouraging wider adoption of technical standards that support reliable quality measurement, better harmonizing data collection with clinical work flow in EHRs, and better understanding the behavior and potential of e-measures.

The utility of imputed matched sets. Analyzing probabilistically linked databases in a low information setting

OBJECTIVE

To compare results from high probability matched sets versus imputed matched sets across differing levels of linkage information.

METHODS

A series of linkages with varying amounts of available information were performed on two simulated datasets derived from multiyear motor vehicle crash (MVC) and hospital databases, where true matches were known. Distributions of high probability and imputed matched sets were compared against the true match population for occupant age, MVC county, and MVC hour. Regression models were fit to simulated log hospital charges and hospitalization status.

RESULTS

High probability and imputed matched sets were not significantly different from occupant age, MVC county, and MVC hour in high information settings (p > 0.999). In low information settings, high probability matched sets were significantly different from occupant age and MVC county (p < 0.002), but imputed matched sets were not (p > 0.493). High information settings saw no significant differences in inference of simulated log hospital charges and hospitalization status between the two methods. High probability and imputed matched sets were significantly different from the outcomes in low information settings; however, imputed matched sets were more robust.

CONCLUSIONS

The level of information available to a linkage is an important consideration. High probability matched sets are suitable for high to moderate information settings and for situations involving case-specific analysis. Conversely, imputed matched sets are preferable for low information settings when conducting population-based analyses.

Text mining and medicine: usefulness in respiratory diseases

It is increasingly common to have medical information in electronic format. This includes scientific articles as well as clinical management reviews, and even records from health institutions with patient data. However, traditional instruments, both individual and institutional, are of little use for selecting the most appropriate information in each case, either in the clinical or research field. So-called text or data «mining» enables this huge amount of information to be managed, extracting it from various sources using processing systems (filtration and curation), integrating it and permitting the generation of new knowledge. This review aims to provide an overview of text and data mining, and of the potential usefulness of this bioinformatic technique in the exercise of care in respiratory medicine and in research in the same field.

The Dutch hospital standardised mortality ratio (HSMR) method and cardiac surgery: benchmarking in a national cohort using hospital administration data versus a clinical database

Objective

To compare the accuracy of data from hospital administration databases and a national clinical cardiac surgery database and to compare the performance of the Dutch hospital standardised mortality ratio (HSMR) method and the logistic European System for Cardiac Operative Risk Evaluation, for the purpose of benchmarking of mortality across hospitals.

Methods

Information on all patients undergoing cardiac surgery between 1 January 2007 and 31 December 2010 in 10 centres was extracted from The Netherlands Association for Cardio-Thoracic Surgery database and the Hospital Discharge Registry. The number of cardiac surgery interventions was compared between both databases. The European System for Cardiac Operative Risk Evaluation and hospital standardised mortality ratio models were updated in the study population and compared using the C-statistic, calibration plots and the Brier-score.

Results

The number of cardiac surgery interventions performed could not be assessed using the administrative database as the intervention code was incorrect in 1.4–26.3%, depending on the type of intervention. In 7.3% no intervention code was registered. The updated administrative model was inferior to the updated clinical model with respect to discrimination (c-statistic of 0.77 vs 0.85, p<0.001) and calibration (Brier Score of 2.8% vs 2.6%, p<0.001, maximum score 3.0%). Two average performing hospitals according to the clinical model became outliers when benchmarking was performed using the administrative model.

Conclusions

In cardiac surgery, administrative data are less suitable than clinical data for the purpose of benchmarking. The use of either administrative or clinical risk-adjustment models can affect the outlier status of hospitals. Risk-adjustment models including procedure-specific clinical risk factors are recommended.

Risk prediction of hospital mortality for adult patients admitted to Australian and New Zealand intensive care units: development and validation of the Australian and New Zealand Risk of Death model

PURPOSE

The purpose of this study is to develop and validate a new mortality prediction model (Australian and New Zealand Risk of Death [ANZROD]) for Australian and New Zealand intensive care units (ICUs) and compare its performance with the existing Acute Physiology and Chronic Health Evaluation (APACHE) III-j.

MATERIALS AND METHODS

All ICU admissions from 2004 to 2009 were extracted from the Australian and New Zealand Intensive Care Society Adult Patient Database. Hospital mortality was modeled using logistic regression with training (two third) and validation (one third) data sets. Predictor variables included APACHE III score components, source of admission to ICU and hospital, lead time, elective surgery, treatment limitation, ventilation status, and APACHE III diagnoses. Model performance was assessed by standardized mortality ratio, Hosmer-Lemeshow C and H statistics, Brier score, Cox calibration regression, area under the receiver operating characteristic curve, and calibration curves.

RESULTS

There were 456605 patients available for model development and validation. Observed mortality was 11.3%. Performance measures (standardized mortality ratio, Hosmer-Lemeshow C and H statistics, and receiver operating characteristic curve) for the ANZROD and APACHE III-j model in the validation data set were 1.01, 104.9 and 111.4, and 0.902; 0.84, 1596.6 and 2087.3, and 0.885, respectively.

CONCLUSIONS

The ANZROD has better calibration; discrimination compared with the APACHE III-j. Further research is required to validate performance over time and in specific subgroups of ICU population.

Predictive data mining on monitoring data from the intensive care unit

The widespread implementation of computerized medical files in intensive care units (ICUs) over recent years has made available large databases of clinical data for the purpose of developing clinical prediction models. The typical intensive care unit has several information sources from which data is electronically collected as time series of varying time resolutions. We present an overview of research questions studied in the ICU setting that have been addressed through the automatic analysis of these large databases. We focus on automatic learning methods, specifically data mining approaches for predictive modeling based on these time series of clinical data. On the one hand we examine short and medium term predictions, which have as ultimate goal the development of early warning or decision support systems. On the other hand we examine long term outcome prediction models and evaluate their performance with respect to established scoring systems based on static admission and demographic data.