An online international comparison of thresholds for triggering a negative response to the "Surprise Question": a study protocol

Prospective Comparison of Medical Oncologists and a Machine Learning Model to Predict 3-Month Mortality in Patients With Metastatic Solid Tumors

Importance

To date, oncologist and model prognostic performance have been assessed independently and mostly retrospectively; however, how model prognostic performance compares with oncologist prognostic performance prospectively remains unknown.

Objective

To compare oncologist performance with a model in predicting 3-month mortality for patients with metastatic solid tumors in an outpatient setting.

Design, Setting, and Participants

This prognostic study evaluated prospective predictions for a cohort of patients with metastatic solid tumors seen in outpatient oncology clinics at a National Cancer Institute-designated cancer center and associated satellites between December 6, 2019, and August 6, 2021. Oncologists (57 physicians and 17 advanced practice clinicians) answered a 3-month surprise question (3MSQ) within clinical pathways. A model was trained with electronic health record data from January 1, 2013, to April 24, 2019, to identify patients at high risk of 3-month mortality and deployed silently in October 2019. Analysis was limited to oncologist prognostications with a model prediction within the preceding 30 days.

Exposures

Three-month surprise question and gradient-boosting binary classifier.

Main Outcomes and Measures

The primary outcome was performance comparison between oncologists and the model to predict 3-month mortality. The primary performance metric was the positive predictive value (PPV) at the sensitivity achieved by the medical oncologists with their 3MSQ answers.

Results

A total of 74 oncologists answered 3099 3MSQs for 2041 patients with advanced cancer (median age, 62.6 [range, 18-96] years; 1271 women [62.3%]). In this cohort with a 15% prevalence of 3-month mortality and 30% sensitivity for both oncologists and the model, the PPV of oncologists was 34.8% (95% CI, 30.1%-39.5%) and the PPV of the model was 60.0% (95% CI, 53.6%-66.3%). Area under the receiver operating characteristic curve for the model was 81.2% (95% CI, 79.1%-83.3%). The model significantly outperformed the oncologists in short-term mortality.

Conclusions and Relevance

In this prognostic study, the model outperformed oncologists overall and within the breast and gastrointestinal cancer cohorts in predicting 3-month mortality for patients with advanced cancer. These findings suggest that further studies may be useful to examine how model predictions could improve oncologists' prognostic confidence and patient-centered goal-concordant care at the end of life.

An online international comparison of palliative care identification in primary care using the Surprise Question

BACKGROUND

The Surprise Question ('Would I be surprised if this patient died within 12 months?') identifies patients in the last year of life. It is unclear if 'surprised' means the same for each clinician, and whether their responses are internally consistent.

AIM

To determine the consistency with which the Surprise Question is used.

DESIGN

A cross-sectional online study of participants located in Belgium, Germany, Italy, The Netherlands, Switzerland and UK. Participants completed 20 hypothetical patient summaries ('vignettes'). Primary outcome measure: continuous estimate of probability of death within 12 months (0% [certain survival]-100% [certain death]). A threshold (probability estimate above which Surprise Question responses were consistently 'no') and an inconsistency range (range of probability estimates where respondents vacillated between responses) were calculated. Univariable and multivariable linear regression explored differences in consistency. Trial registration: NCT03697213.

SETTING/PARTICIPANTS

Registered General Practitioners (GPs). Of the 307 GPs who started the study, 250 completed 15 or more vignettes.

RESULTS

Participants had a consistency threshold of 49.8% (SD 22.7) and inconsistency range of 17% (SD 22.4). Italy had a significantly higher threshold than other countries (p = 0.002). There was also a difference in threshold levels depending on age of clinician, for every yearly increase, participants had a higher threshold. There was no difference in inconsistency between countries (p = 0.53).

CONCLUSIONS

There is variation between clinicians regarding the use of the Surprise Question. Over half of GPs were not internally consistent in their responses to the Surprise Question. Future research with standardised terms and real patients is warranted.

The Surprise Question and Serious Illness Conversations: A pilot study

BACKGROUND

Serious Illness Conversations aim to discuss patient goals. However, on acute medicine units, seriously ill patients may undergo distressing interventions until death.

OBJECTIVES

To investigate the feasibility of using the Surprise Question, "Would you be surprised if this patient died within the next year?" to identify patients who would benefit from early Serious Illness Conversations and study any changes in the interdisciplinary team's beliefs, confidence, and engagement as a result of asking the Surprise Question.

DESIGN

A prospective cohort pilot study with two Plan-Do-Study-Act cycles.

PARTICIPANTS/CONTEXT

Fifty-eight healthcare professionals working on Acute Medicine Units participated in pre- and post-intervention questionnaires. The intervention involved asking participants the Surprise Question for each patient. Patient charts were reviewed for Serious Illness Conversation documentation.

ETHICAL CONSIDERATIONS

Ethical approval was granted by the institutions involved.

FINDINGS

Equivocal overall changes in the beliefs, confidence, and engagement of healthcare professionals were observed. Six out of 23 patients were indicated as needing a Serious Illness Conversation; chart review provided some evidence that these patients had more Serious Illness Conversation documentation compared with the 17 patients not flagged for a Serious Illness Conversation. Issues were identified in equating the Surprise Question to a Serious Illness Conversation.

DISCUSSION

Appropriate support for seriously ill patients is both a nursing professional and ethical duty. Flagging patients for conversations may act as a filtering process, allowing healthcare professionals to focus on conversations with patients who need them most. There are ethical and practical issues as to what constitutes a "serious illness" and if answering "no" to the Surprise Question always equates to a conversation.

CONCLUSION

The barriers of time constraints and lack of training call for institutional change in order to prioritise the moral obligation of Serious Illness Conversations.