Identifying patients at risk of inhospital death or hospice transfer for early goals of care discussions in a US referral center: the HELPS model derived from retrospective data

Identification of Hospitalized Patients Who May Benefit from a Serious Illness Conversation Using the Readmission Risk Score Combined with the Surprise Question

BACKGROUND

Determining which patients benefit from a serious illness conversation (SIC) is challenging. The authors sought to determine whether Epic's Risk of Readmission Score (RRS), could be combined with a simple, validated, one-question mortality prognostic screen (the surprise question: Would you be surprised if the patient died in the next 12 months?) to identify hospitalized patients with SIC needs.

METHODS

In this retrospective study, the authors randomly selected encounters for patients ≥ 18 years of age to a general medicine service from January 2019 to October 2021 who had an RRS > 28%. Two adjudicators independently performed chart reviews for each encounter to answer the surprise question to create two distinct prognostic groups (yes vs. no). Fisher's exact test was used to assess for statistically significant differences in standardized documentation of SICs between groups.

RESULTS

Out of 2,879 encounters, 202 patient encounters were randomly selected. Adjudicators answered "no" to the surprise question for 156 (77.2%) patients. Patients for whom adjudicators answered "no" were generally older with higher comorbidity and more often had standardized documentation of a SIC (14 [9.0%] vs. 0.[0.0%], p = 0.042) compared to patients for whom adjudicators answered "yes."

CONCLUSION

Approximately three quarters of patients with a high RRS were predicted to have a lifespan of less than a year. Although these patients were significantly more likely to have a SIC, rates of SICs were extremely low. Combining available electronic health record (EHR) data with a simple one-question screening tool may help identify hospitalized patients who require a SIC in quality improvement initiatives.

Who Gets (and Who Should Get) a Serious Illness Conversation in the Hospital? An Analysis of Readmission Risk Score in an Electronic Health Record

Serious Illness Conversations (SICs) explore patients' prognostic awareness, hopes, and worries, and can help establish priorities for their care during and after hospitalization. While identifying patients who benefit from an SIC remains a challenge, this task may be facilitated by use of validated prediction scores available in most commercial electronic health records (EHRs), such as Epic's Readmission Risk Score (RRS). We identified the RRS on admission for all hospital encounters from October 2018 to August 2019 and measured the area under the receiver operating characteristic (AUROC) curve to determine whether RRS could accurately discriminate post discharge 6-month mortality. For encounters with standardized SIC documentation matched in a 1:3 ratio to controls by sex and age (±5 years), we constructed a multivariable, paired logistic regression model and measured the odds of SIC documentation per every 10% absolute increase in RRS. RRS was predictive of 6-month mortality with acceptable discrimination (AUROC .71) and was significantly associated with SIC documentation (adjusted OR 1.42, 95% CI 1.24-1.63). An RRS >28% used to identify patients with post discharge 6-month mortality had a high specificity (89.0%) and negative predictive value (NPV) (97.0%), but low sensitivity (25.2%) and positive predictive value (PPV) (7.9%). RRS may serve as a practical EHR-based screen to exclude patients not requiring an SIC, thereby leaving a smaller cohort to be further evaluated for SIC needs using other validated tools and clinical assessment.

Improving time to palliative care review with predictive modeling in an inpatient adult population: study protocol for a stepped-wedge, pragmatic randomized controlled trial

Background

Palliative care is a medical specialty centered on improving the quality of life (QOL) of patients with complex or life-threatening illnesses. The need for palliative care is increasing and with that the rigorous testing of triage tools that can be used quickly and reliably to identify patients that may benefit from palliative care.

Methods

To that aim, we will conduct a two-armed stepped-wedge cluster randomized trial rolled out to two inpatient hospitals to evaluate whether a machine learning algorithm accurately identifies patients who may benefit from a comprehensive review by a palliative care specialist and decreases time to receiving a palliative care consult in hospital. This is a single-center study which will be conducted from August 2019 to November 2020 at Saint Mary’s Hospital & Methodist Hospital both within Mayo Clinic Rochester in Minnesota. Clusters will be nursing units which will be chosen to be a mix of complex patients from Cardiology, Critical Care, and Oncology and had previously established relationships with palliative medicine. The stepped wedge design will have 12 units allocated to a design matrix of 5 treatment wedges. Each wedge will last 75 days resulting in a study period of 12 months of recruitment unless otherwise specified. Data will be analyzed with Bayesian hierarchical models with credible intervals denoting statistical significance.

Discussion

This intervention offers a pragmatic approach to delivering specialty palliative care to hospital patients in need using machine learning, thereby leading to high value care and improved outcomes. It is not enough for AI to be utilized by simply publishing research showing predictive performance; clinical trials demonstrating better outcomes are critically needed. Furthermore, the deployment of an AI algorithm is a complex process that requires multiple teams with varying skill sets. To evaluate a deployed AI, a pragmatic clinical trial can accommodate the difficulties of clinical practice while retaining scientific rigor.

Trial registration

ClinicalTrials.gov NCT03976297. Registered on 6 June 2019, prior to trial start.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-021-05546-5.

SafeNET: Initial development and validation of a real-time tool for predicting mortality risk at the time of hospital transfer to a higher level of care

Background

Processes for transferring patients to higher acuity facilities lack a standardized approach to prognostication, increasing the risk for low value care that imposes significant burdens on patients and their families with unclear benefits. We sought to develop a rapid and feasible tool for predicting mortality using variables readily available at the time of hospital transfer.

Methods and findings

All work was carried out at a single, large, multi-hospital integrated healthcare system. We used a retrospective cohort for model development consisting of patients aged 18 years or older transferred into the healthcare system from another hospital, hospice, skilled nursing or other healthcare facility with an admission priority of direct emergency admit. The cohort was randomly divided into training and test sets to develop first a 54-variable, and then a 14-variable gradient boosting model to predict the primary outcome of all cause in-hospital mortality. Secondary outcomes included 30-day and 90-day mortality and transition to comfort measures only or hospice care. For model validation, we used a prospective cohort consisting of all patients transferred to a single, tertiary care hospital from one of the 3 referring hospitals, excluding patients transferred for myocardial infarction or maternal labor and delivery. Prospective validation was performed by using a web-based tool to calculate the risk of mortality at the time of transfer. Observed outcomes were compared to predicted outcomes to assess model performance.

The development cohort included 20,985 patients with 1,937 (9.2%) in-hospital mortalities, 2,884 (13.7%) 30-day mortalities, and 3,899 (18.6%) 90-day mortalities. The 14-variable gradient boosting model effectively predicted in-hospital, 30-day and 90-day mortality (c = 0.903 [95% CI:0.891–0.916]), c = 0.877 [95% CI:0.864–0.890]), and c = 0.869 [95% CI:0.857–0.881], respectively). The tool was proven feasible and valid for bedside implementation in a prospective cohort of 679 sequentially transferred patients for whom the bedside nurse calculated a SafeNET score at the time of transfer, taking only 4–5 minutes per patient with discrimination consistent with the development sample for in-hospital, 30-day and 90-day mortality (c = 0.836 [95%CI: 0.751–0.921], 0.815 [95% CI: 0.730–0.900], and 0.794 [95% CI: 0.725–0.864], respectively).

Conclusions

The SafeNET algorithm is feasible and valid for real-time, bedside mortality risk prediction at the time of hospital transfer. Work is ongoing to build pathways triggered by this score that direct needed resources to the patients at greatest risk of poor outcomes.