Development of an automated, real time surveillance tool for predicting readmissions at a community hospital

Improvement of scoring system used before discharge to predict 30-day all-cause unplanned readmission in geriatric population: a prospective cohort study

BACKGROUND

Data taken from tertiary referral hospitals in Indonesia suggested readmission rate in older population ranging between 18.1 and 36.3%. Thus, it is crucial to identify high risk patients who were readmitted. Our previous study found several important predictors, despite unsatisfactory discrimination value.

METHODS

We aimed to investigate whether comprehensive geriatric assessment (CGA) -based modification to the published seven-point scoring system may increase the discrimination value. We conducted a prospective cohort study in July-September 2022 and recruited patients aged 60 years and older admitted to the non-surgical ward and intensive coronary care unit. The ROC curve was made based on the four variables included in the prior study. We conducted bivariate and multivariate analyses, and derived a new scoring system with its discrimination value.

RESULTS

Of 235 subjects, the incidence of readmission was 32.3% (95% CI 26-38%). We established a new scoring system consisting of 4 components. The scoring system had maximum score of 21 and incorporated malignancy (6 points), delirium (4 points), length of stay ≥ 10 days (4 points), and being at risk of malnutrition or malnourished (7 points), with a good calibration test. The C-statistic value was 0.835 (95% CI 0.781-0.880). The optimal cut-off point was ≥ 8 with a sensitivity of 90.8% and a specificity of 54.7%.

CONCLUSIONS

Malignancy, delirium, length of stay ≥ 10 days, and being at risk of malnutrition or malnourished are predictors for 30-day all-cause unplanned readmission. The sensitive scoring system is a strong model to identify whether an individual is at higher risk for readmission. The new CGA-based scoring system had higher discrimination value than that of the previous seven-point scoring system.

Predicting Hospital Readmissions in a Commercially Insured Population over Varying Time Horizons

BACKGROUND

Reducing hospital readmissions is a federal policy priority, and predictive models of hospital readmissions have proliferated in recent years; however, most such models tend to focus on the 30-day readmission time horizon and do not consider readmission over shorter (or longer) windows.

OBJECTIVES

To evaluate the performance of a predictive model of hospital readmissions over three different readmission timeframes in a commercially insured population.

DESIGN

Retrospective multivariate logistic regression with an 80/20 train/test split.

PARTICIPANTS

A total of 2,213,832 commercially insured inpatient admissions from 2016 to 2017 comprising 782,768 unique patients from the Health Care Cost Institute.

MAIN MEASURES

Outcomes are readmission within 14 days, 15-30 days, and 31-60 days from discharge. Predictor variables span six different domains: index admission, condition history, demographic, utilization history, pharmacy, and environmental controls.

KEY RESULTS

Our model generates C-statistics for holdout samples ranging from 0.618 to 0.915. The model's discriminative power declines with readmission time horizon: discrimination for readmission predictions within 14 days following discharge is higher than for readmissions 15-30 days following discharge, which in turn is higher than predictions 31-60 days following discharge. Additionally, the model's predictive power increases nonlinearly with the inclusion of successive risk factor domains: patient-level measures of utilization and condition history add substantially to the discriminative power of the model, while demographic information, pharmacy utilization, and environmental risk factors add relatively little.

CONCLUSION

It is more difficult to predict distant readmissions than proximal readmissions, and the more information the model uses, the better the predictions. Inclusion of utilization-based risk factors add substantially to the discriminative ability of the model, much more than any other included risk factor domain. Our best-performing models perform well relative to other published readmission prediction models. It is possible that these predictions could have operational utility in targeting readmission prevention interventions among high-risk individuals.

Implementation Experience with a 30-Day Hospital Readmission Risk Score in a Large, Integrated Health System: A Retrospective Study

BACKGROUND

Driven by quality outcomes and economic incentives, predicting 30-day hospital readmissions remains important for healthcare systems. The Cleveland Clinic Health System (CCHS) implemented an internally validated readmission risk score in the electronic medical record (EMR).

OBJECTIVE

We evaluated the predictive accuracy of the readmission risk score across CCHS hospitals, across primary discharge diagnosis categories, between surgical/medical specialties, and by race and ethnicity.

DESIGN

Retrospective cohort study.

PARTICIPANTS

Adult patients discharged from a CCHS hospital April 2017-September 2020.

MAIN MEASURES

Data was obtained from the CCHS EMR and billing databases. All patients discharged from a CCHS hospital were included except those from Oncology and Labor/Delivery, patients with hospice orders, or patients who died during admission. Discharges were categorized as surgical if from a surgical department or surgery was performed. Primary discharge diagnoses were classified per Agency for Healthcare Research and Quality Clinical Classifications Software Level 1 categories. Discrimination performance predicting 30-day readmission is reported using the c-statistic.

RESULTS

The final cohort included 600,872 discharges from 11 Northeast Ohio and Florida CCHS hospitals. The readmission risk score for the cohort had a c-statistic of 0.6875 with consistent yearly performance. The c-statistic for hospital sites ranged from 0.6762, CI [0.6634, 0.6876], to 0.7023, CI [0.6903, 0.7132]. Medical and surgical discharges showed consistent performance with c-statistics of 0.6923, CI [0.6807, 0.7045], and 0.6802, CI [0.6681, 0.6925], respectively. Primary discharge diagnosis showed variation, with lower performance for congenital anomalies and neoplasms. COVID-19 had a c-statistic of 0.6387. Subgroup analyses showed c-statistics of > 0.65 across race and ethnicity categories.

CONCLUSIONS

The CCHS readmission risk score showed good performance across diverse hospitals, across diagnosis categories, between surgical/medical specialties, and by patient race and ethnicity categories for 3 years after implementation, including during COVID-19. Evaluating clinical decision-making tools post-implementation is crucial to determine their continued relevance, identify opportunities to improve performance, and guide their appropriate use.

Published models that predict hospital readmission: a critical appraisal

INTRODUCTION

The number of readmission risk prediction models available has increased rapidly, and these models are used extensively for health decision-making. Unfortunately, readmission models can be subject to flaws in their development and validation, as well as limitations in their clinical usefulness.

OBJECTIVE

To critically appraise readmission models in the published literature using Delphi-based recommendations for their development and validation.

METHODS

We used the modified Delphi process to create Critical Appraisal of Models that Predict Readmission (CAMPR), which lists expert recommendations focused on development and validation of readmission models. Guided by CAMPR, two researchers independently appraised published readmission models in two recent systematic reviews and concurrently extracted data to generate reference lists of eligibility criteria and risk factors.

RESULTS

We found that published models (n=81) followed 6.8 recommendations (45%) on average. Many models had weaknesses in their development, including failure to internally validate (12%), failure to account for readmission at other institutions (93%), failure to account for missing data (68%), failure to discuss data preprocessing (67%) and failure to state the model's eligibility criteria (33%).

CONCLUSIONS

The high prevalence of weaknesses in model development identified in the published literature is concerning, as these weaknesses are known to compromise predictive validity. CAMPR may support researchers, clinicians and administrators to identify and prevent future weaknesses in model development.

What Are They Worth? Six 30-Day Readmission Risk Scores for Medical Inpatients Externally Validated in a Swiss Cohort

BACKGROUND

Several clinical risk scores for unplanned 30-day readmission have been published, but there is a lack of external validation and head-to-head comparison.

OBJECTIVE

Retrospective replication of six clinical risk scores (LACE, HOSPITAL, SEMI, RRS, PARA, Tsui et al.)f DESIGN: Models were fitted with the original intercept and beta coefficients as reported. Otherwise, a logistic model was refitted (SEMI and Tsui et al). We performed subgroup analyses on main admission specialty. This report adheres to the TRIPOD statement for reporting of prediction models.

PARTICIPANTS

We used our prospective cohort of 15,639 medical patients from a Swiss tertiary care institution from 2016 through 2018.

MAIN MEASURES

Thirty-day readmission rate and area under the curve (AUC < 0.50 worse than chance, > 0.70 acceptable, > 0.80 excellent) CONCLUSIONS: Among several readmission risk scores, HOSPITAL, PARA, and the score from Tsui et al. showed the best predictive abilities and have high potential to improve patient care. Interventional research is now needed to understand the effects of these scores when used in clinical routine.

KEY RESULTS

Among the six risk scores externally validated, calibration of the models was overall poor with overprediction of events, except for the HOSPITAL and the PARA scores. Discriminative abilities (AUC) were as follows: LACE 0.53 (95% CI 0.50-0.56), HOSPITAL 0.73 (95% CI 0.72-0.74), SEMI 0.47 (95% CI 0.46-0.49), RRS 0.64 (95% CI 0.62-0.66), PARA 0.72 (95% CI 0.72-0.74), and the score from Tsui et al. 0.73 (95% CI 0.72-0.75). Performance in subgroups did not differ from the overall performance, except for oncology patients in the PARA score (0.57, 95% CI 0.54-0.60), and nephrology patients in the SEMI index (0.25, 95% CI 0.18-0.31), respectively.

POLAR Diversion: Using General Practice Data to Calculate Risk of Emergency Department Presentation at the Time of Consultation

OBJECTIVE

This project examined and produced a general practice (GP) based decision support tool (DST), namely POLAR Diversion, to predict a patient's risk of emergency department (ED) presentation. The tool was built using both GP/family practice and ED data, but is designed to operate on GP data alone.

METHODS

GP data from 50 practices during a defined time frame were linked with three local EDs. Linked data and data mapping were used to develop a machine learning DST to determine a range of variables that, in combination, led to predictive patient ED presentation risk scores. Thirteen percent of the GP data was kept as a control group and used to validate the tool.

RESULTS

The algorithm performed best in predicting the risk of attending ED within the 30-day time category, and also in the no ED attendance tests, suggesting few false positives. At 0 to 30 days the positive predictive value (PPV) was 74%, with a sensitivity/recall of 68%. Non-ED attendance had a PPV of 82% and sensitivity/recall of 96%.

CONCLUSION

Findings indicate that the POLAR Diversion algorithm performed better than previously developed tools, particularly in the 0 to 30 day time category. Its utility increases because of it being based on the data within the GP system alone, with the ability to create real-time "in consultation" warnings. The tool will be deployed across GPs in Australia, allowing us to assess the clinical utility, and data quality needs in further iterations.

Combining Contrast Mining with Logistic Regression To Predict Healthcare Utilization in a Managed Care Population

BACKGROUND

Because 5% of patients incur 50% of healthcare expenses, population health managers need to be able to focus preventive and longitudinal care on those patients who are at highest risk of increased utilization. Predictive analytics can be used to identify these patients and to better manage their care. Data mining permits the development of models that surpass the size restrictions of traditional statistical methods and take advantage of the rich data available in the electronic health record (EHR), without limiting predictions to specific chronic conditions.

OBJECTIVE

The objective was to demonstrate the usefulness of unrestricted EHR data for predictive analytics in managed healthcare.

METHODS

In a population of 9,568 Medicare and Medicaid beneficiaries, patients in the highest 5% of charges were compared to equal numbers of patients with the lowest charges. Contrast mining was used to discover the combinations of clinical attributes frequently associated with high utilization and infrequently associated with low utilization. The attributes found in these combinations were then tested by multiple logistic regression, and the discrimination of the model was evaluated by the c-statistic.

RESULTS

Of 19,014 potential EHR patient attributes, 67 were found in combinations frequently associated with high utilization, but not with low utilization (support>20%). Eleven of these attributes were significantly associated with high utilization (p<0.05). A prediction model composed of these eleven attributes had a discrimination of 84%.

CONCLUSIONS

EHR mining reduced an unusably high number of patient attributes to a manageable set of potential healthcare utilization predictors, without conjecturing on which attributes would be useful. Treating these results as hypotheses to be tested by conventional methods yielded a highly accurate predictive model. This novel, two-step methodology can assist population health managers to focus preventive and longitudinal care on those patients who are at highest risk for increased utilization.

Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review

OBJECTIVE

To update previous systematic review of predictive models for 28-day or 30-day unplanned hospital readmissions.

DESIGN

Systematic review.

SETTING/DATA SOURCE

CINAHL, Embase, MEDLINE from 2011 to 2015.

PARTICIPANTS

All studies of 28-day and 30-day readmission predictive model.

OUTCOME MEASURES

Characteristics of the included studies, performance of the identified predictive models and key predictive variables included in the models.

RESULTS

Of 7310 records, a total of 60 studies with 73 unique predictive models met the inclusion criteria. The utilisation outcome of the models included all-cause readmissions, cardiovascular disease including pneumonia, medical conditions, surgical conditions and mental health condition-related readmissions. Overall, a wide-range C-statistic was reported in 56/60 studies (0.21-0.88). 11 of 13 predictive models for medical condition-related readmissions were found to have consistent moderate discrimination ability (C-statistic ≥0.7). Only two models were designed for the potentially preventable/avoidable readmissions and had C-statistic >0.8. The variables 'comorbidities', 'length of stay' and 'previous admissions' were frequently cited across 73 models. The variables 'laboratory tests' and 'medication' had more weight in the models for cardiovascular disease and medical condition-related readmissions.

CONCLUSIONS

The predictive models which focused on general medical condition-related unplanned hospital readmissions reported moderate discriminative ability. Two models for potentially preventable/avoidable readmissions showed high discriminative ability. This updated systematic review, however, found inconsistent performance across the included unique 73 risk predictive models. It is critical to define clearly the utilisation outcomes and the type of accessible data source before the selection of the predictive model. Rigorous validation of the predictive models with moderate-to-high discriminative ability is essential, especially for the two models for the potentially preventable/avoidable readmissions. Given the limited available evidence, the development of a predictive model specifically for paediatric 28-day all-cause, unplanned hospital readmissions is a high priority.

Risk factors and screening instruments to predict adverse outcomes for undifferentiated older emergency department patients: a systematic review and meta-analysis

OBJECTIVES

A significant proportion of geriatric patients experience suboptimal outcomes following episodes of emergency department (ED) care. Risk stratification screening instruments exist to distinguish vulnerable subsets, but their prognostic accuracy varies. This systematic review quantifies the prognostic accuracy of individual risk factors and ED-validated screening instruments to distinguish patients more or less likely to experience short-term adverse outcomes like unanticipated ED returns, hospital readmissions, functional decline, or death.

METHODS

A medical librarian and two emergency physicians conducted a medical literature search of PubMed, EMBASE, SCOPUS, CENTRAL, and ClinicalTrials.gov using numerous combinations of search terms, including emergency medical services, risk stratification, geriatric, and multiple related MeSH terms in hundreds of combinations. Two authors hand-searched relevant specialty society research abstracts. Two physicians independently reviewed all abstracts and used the revised Quality Assessment of Diagnostic Accuracy Studies instrument to assess individual study quality. When two or more qualitatively similar studies were identified, meta-analysis was conducted using Meta-DiSc software. Primary outcomes were sensitivity, specificity, positive likelihood ratio (LR+), and negative likelihood ratio (LR-) for predictors of adverse outcomes at 1 to 12 months after the ED encounters. A hypothetical test-treatment threshold analysis was constructed based on the meta-analytic summary estimate of prognostic accuracy for one outcome.

RESULTS

A total of 7,940 unique citations were identified yielding 34 studies for inclusion in this systematic review. Studies were significantly heterogeneous in terms of country, outcomes assessed, and the timing of post-ED outcome assessments. All studies occurred in ED settings and none used published clinical decision rule derivation methodology. Individual risk factors assessed included dementia, delirium, age, dependency, malnutrition, pressure sore risk, and self-rated health. None of these risk factors significantly increased the risk of adverse outcome (LR+ range = 0.78 to 2.84). The absence of dependency reduces the risk of 1-year mortality (LR- = 0.27) and nursing home placement (LR- = 0.27). Five constructs of frailty were evaluated, but none increased or decreased the risk of adverse outcome. Three instruments were evaluated in the meta-analysis: Identification of Seniors at Risk, Triage Risk Screening Tool, and Variables Indicative of Placement Risk. None of these instruments significantly increased (LR+ range for various outcomes = 0.98 to 1.40) or decreased (LR- range = 0.53 to 1.11) the risk of adverse outcomes. The test threshold for 3-month functional decline based on the most accurate instrument was 42%, and the treatment threshold was 61%.

CONCLUSIONS

Risk stratification of geriatric adults following ED care is limited by the lack of pragmatic, accurate, and reliable instruments. Although absence of dependency reduces the risk of 1-year mortality, no individual risk factor, frailty construct, or risk assessment instrument accurately predicts risk of adverse outcomes in older ED patients. Existing instruments designed to risk stratify older ED patients do not accurately distinguish high- or low-risk subsets. Clinicians, educators, and policy-makers should not use these instruments as valid predictors of post-ED adverse outcomes. Future research to derive and validate feasible ED instruments to distinguish vulnerable elders should employ published decision instrument methods and examine the contributions of alternative variables, such as health literacy and dementia, which often remain clinically occult.

From bench to bed: bridging from informatics theory to practice. An exploratory analysis

BACKGROUND

In 2009, the journal Applied Clinical Informatics (ACI) commenced publication. Focused on applications in clinical informatics, ACI was intended to be a companion journal to METHODS of Information in Medicine (MIM). Both journals are official journals of IMIA, the International Medical Informatics Association.

OBJECTIVES

To explore, after five years, which congruencies and interdependencies exist in publications of these journals and to determine if gaps exist. To achieve this goal, major topics discussed in ACI and in MIM had to be analysed. Finally, we wanted to explore, whether the intention of publishing these companion journals to provide an information bridge from informatics theory to informatics practice and from practice to theory could be supported by this model. In this manuscript we will report on congruencies and interdependencies from practise to theory and on major topis in ACI. Further results will be reported in a second paper.

METHODS

Retrospective, prolective observational study on recent publications of ACI and MIM. All publications of the years 2012 and 2013 from these journals were indexed and analysed.

RESULTS

Hundred and ninety-six publications have been analysed (87 ACI, 109 MIM). In ACI publications addressed care coordination, shared decision support, and provider communication in its importance for complex patient care and safety and quality. Other major themes included improving clinical documentation quality and efficiency, effectiveness of clinical decision support and alerts, implementation of health information technology systems including discussion of failures and succeses. An emerging topic in the years analyzed was a focus on health information technology to predict and prevent hospital admissions and managing population health including the application of mobile health technology. Congruencies between journals could be found in themes, but with different focus in its contents. Interdependencies from practise to theory found in these publications, were only limited.

CONCLUSIONS

Bridging from informatics theory to practise and vice versa remains a major component of successful research and practise as well as a major challenge.