Development and Validation of a Machine Learning Algorithm Predicting Emergency Department Use and Unplanned Hospitalization in Patients With Head and Neck Cancer

Using artificial intelligence to predict patient outcomes from patient-reported outcome measures: a scoping review

PURPOSE

This scoping review aims to identify and summarise artificial intelligence (AI) methods applied to patient-reported outcome measures (PROMs) for prediction of patient outcomes, such as survival, quality of life, or treatment decisions.

INTRODUCTION

AI models have been successfully applied to predict outcomes for patients using mainly clinically focused data. However, systematic guidance for utilising AI and PROMs for patient outcome predictions is lacking. This leads to inconsistency of model development and evaluation, limited practical implications, and poor translation to clinical practice.

MATERIALS AND METHODS

This review was conducted across Web of Science, IEEE Xplore, ACM, Digital Library, Cochrane Central Register of Controlled Trials, Medline and Embase databases. Adapted search terms identified published research using AI models with patient-reported data for outcome predictions. Papers using PROMs data as input variables in AI models for prediction of patient outcomes were included.

RESULTS

Three thousand and seventy-seven records were screened, 94 of which were included in the analysis. AI models applied to PROMs data for outcome predictions are most commonly used in orthopaedics and oncology. Poor reporting of model hyperparameters and inconsistent techniques of handling class imbalance and missingness in data were found. The absence of external model validation, participants' ethnicity information and stakeholders involvement was common.

CONCLUSION

The results highlight inconsistencies in conducting and reporting of AI research involving PROMs in patients' outcomes predictions, which reduces the reproducibility of the studies. Recommendations for external validation and stakeholders' involvement are given to increase the opportunities for applying AI models in clinical practice.

Use of Patient-Reported Outcomes in Risk Prediction Model Development to Support Cancer Care Delivery: A Scoping Review

PURPOSE

The integration of patient-reported outcomes (PROs) into electronic health records (EHRs) has enabled systematic collection of symptom data to manage post-treatment symptoms. The use and integration of PRO data into routine care are associated with overall treatment success, adherence, and satisfaction. Clinical trials have demonstrated the prognostic value of PROs including physical function and global health status in predicting survival. It is unknown to what extent routinely collected PRO data are used in the development of risk prediction models (RPMs) in oncology care. The objective of the scoping review is to assess how PROs are used to train risk RPMs to predict patient outcomes in oncology care.

METHODS

Using the scoping review methodology outlined in the Joanna Briggs Institute Manual for Evidence Synthesis, we searched four databases (MEDLINE, CINAHL, Embase, and Web of Science) to locate peer-reviewed oncology articles that used PROs as predictors to train models. Study characteristics including settings, clinical outcomes, and model training, testing, validation, and performance data were extracted for analyses.

RESULTS

Of the 1,254 studies identified, 18 met inclusion criteria. Most studies performed retrospective analyses of prospectively collected PRO data to build prediction models. Post-treatment survival was the most common outcome predicted. Discriminative performance of models trained using PROs was better than models trained without PROs. Most studies did not report model calibration.

CONCLUSION

Systematic collection of PROs in routine practice provides an opportunity to use patient-reported data to develop RPMs. Model performance improves when PROs are used in combination with other comprehensive data sources.

The efficacy of the National Surgical Quality Improvement Program surgical risk calculator in head and neck surgery: A meta-analysis

BACKGROUND

The National Surgical Quality Improvement Program surgical risk calculator (SRC) estimates the risk for postoperative complications. This meta-analysis assesses the efficacy of the SRC in the field of head and neck surgery.

METHODS

A systematic review identified studies comparing the SRC's predictions to observed outcomes following head and neck surgeries. Predictive accuracy was assessed using receiver operating characteristic curves (AUCs) and Brier scoring.

RESULTS

Nine studies totaling 1774 patients were included. The SRC underpredicted the risk of all outcomes (including any complication [observed (ob) = 35.9%, predicted (pr) = 21.8%] and serious complication [ob = 28.7%, pr = 17.0%]) except mortality (ob = 0.37%, pr = 1.55%). The observed length of stay was more than twice the predicted length (p < 0.02). Discrimination was acceptable for postoperative pneumonia (AUC = 0.778) and urinary tract infection (AUC = 0.782) only. Predictive accuracy was low for all outcomes (Brier scores ≥0.01) and comparable for patients with and without free-flap reconstructions.

CONCLUSION

The SRC is an ineffective instrument for predicting outcomes in head and neck surgery.

Can prediction models for hospital readmission be improved by incorporating patient-reported outcome measures? A systematic review and narrative synthesis

PURPOSE

To investigate the roles, challenges, and implications of using patient-reported outcome measures (PROMs) in predicting the risk of hospital readmissions.

METHODS

We systematically searched four bibliometric databases for peer-reviewed studies published in English between 1 January 2000 and 15 June 2023 and used validated PROMs to predict readmission risks for adult populations. Reported studies were analysed and narratively synthesised in accordance with the CHARMS and PRISMA guidelines.

RESULTS

Of the 2858 abstracts reviewed, 23 studies met predefined eligibility criteria, representing diverse geographic regions and medical specialties. Among those, 19 identified the positive contributions of PROMs in predicting readmission risks. Seven studies utilised generic PROMs exclusively, eleven used generic and condition-specific PROMs, while 5 focussed solely on condition-specific PROMs. Logistic regression was the most used modelling approach, with 13 studies aiming at predicting 30-day all-cause readmission risks. The c-statistic, ranging from 0.54 to 0.84, was reported in 22/23 studies as a measure of model discrimination. Nine studies reported model calibration in addition to c-statistic. Thirteen studies detailed their approaches to dealing with missing data.

CONCLUSION

Our study highlights the potential of PROMs to enhance predictive accuracy in readmission models, while acknowledging the diversity in data collection methods, readmission definitions, and model evaluation approaches. Recognizing that PROMs serve various purposes beyond readmission reduction, our study supports routine data collection and strategic integration of PROMs in healthcare practices to improve patient outcomes. To facilitate comparative analysis and broaden the use of PROMs in the prediction framework, it is imperative to consider the methodological aspects involved.

An introduction to machine learning and generative artificial intelligence for otolaryngologists-head and neck surgeons: a narrative review

PURPOSE

Despite the robust expansion of research surrounding artificial intelligence (AI) and machine learning (ML) and their applications to medicine, these methodologies often remain opaque and inaccessible to many otolaryngologists. Especially, with the increasing ubiquity of large-language models (LLMs), such as ChatGPT and their potential implementation in clinical practice, clinicians may benefit from a baseline understanding of some aspects of AI. In this narrative review, we seek to clarify underlying concepts, illustrate applications to otolaryngology, and highlight future directions and limitations of these tools.

METHODS

Recent literature regarding AI principles and otolaryngologic applications of ML and LLMs was reviewed via search in PubMed and Google Scholar.

RESULTS

Significant recent strides have been made in otolaryngology research utilizing AI and ML, across all subspecialties, including neurotology, head and neck oncology, laryngology, rhinology, and sleep surgery. Potential applications suggested by recent publications include screening and diagnosis, predictive tools, clinical decision support, and clinical workflow improvement via LLMs. Ongoing concerns regarding AI in medicine include ethical concerns around bias and data sharing, as well as the "black box" problem and limitations in explainability.

CONCLUSIONS

Potential implementations of AI in otolaryngology are rapidly expanding. While implementation in clinical practice remains theoretical for most of these tools, their potential power to influence the practice of otolaryngology is substantial.

LEVEL OF EVIDENCE: 4

Gaps in Depression Symptom Management for Patients With Head and Neck Cancer

OBJECTIVE

To understand practice patterns and identify care gaps within a large-scale depression screening program for patients with head and neck cancer (HNC).

STUDY DESIGN

Retrospective cohort study.

METHODS

This was a population-based study of adults diagnosed with a HNC between January 2007 and October 2020. Each patient was observed from time of first symptom assessment until end of study date, or death. The exposure of interest was a positive depressive symptom screen on the Edmonton Symptom Assessment System (ESAS). Outcomes of interest included psychiatry/psychology assessment, social work referral, or palliative care assessment. Cause specific hazard models with a time-varying exposure were used to investigate the exposure-outcome relationships.

RESULTS

Of 14,054 patients with HNC, 9016 (64.2%) reported depressive symptoms on at least one ESAS assessment. Within 60 days of first reporting depressive symptoms, 223 (2.7%) received a psychiatry assessment, 646 (7.9%) a social work referral, and 1131 (13.9%) a palliative care assessment. Rates of psychiatry/psychology assessment (HR 3.15 [95% CI 2.67-3.72]), social work referral (HR 1.83 [95% CI 1.64-2.02]), and palliative care assessment (HR 2.34 [95% CI 2.19-2.50]) were higher for those screening positive for depression. Certain patient populations were less likely to receive an assessment including the elderly, rural residents, and those without a prior psychiatric history.

CONCLUSION

A high proportion of head and neck patients report depressive symptoms, though this triggers a referral in a small number of cases. These data highlight areas for improvement in depression screening care pathways.

LEVEL OF EVIDENCE

3 Laryngoscope, 133:2638-2646, 2023.

A Machine Learning Approach to Assess Patients with Deep Neck Infection Progression to Descending Mediastinitis: Preliminary Results

BACKGROUND

Deep neck infection (DNI) is a serious infectious disease, and descending mediastinitis is a fatal infection of the mediastinum. However, no study has applied artificial intelligence to assess progression to descending mediastinitis in DNI patients. Thus, we developed a model to assess the possible progression of DNI to descending mediastinitis.

METHODS

Between August 2017 and December 2022, 380 patients with DNI were enrolled; 75% of patients (n = 285) were assigned to the training group for validation, whereas the remaining 25% (n = 95) were assigned to the test group to determine the accuracy. The patients' clinical and computed tomography (CT) parameters were analyzed via the k-nearest neighbor method. The predicted and actual progression of DNI patients to descending mediastinitis were compared.

RESULTS

In the training and test groups, there was no statistical significance (all p > 0.05) noted at clinical variables (age, gender, chief complaint period, white blood cells, C-reactive protein, diabetes mellitus, and blood sugar), deep neck space (parapharyngeal, submandibular, retropharyngeal, and multiple spaces involved, ≥3), tracheostomy performance, imaging parameters (maximum diameter of abscess and nearest distance from abscess to level of sternum notch), or progression to mediastinitis. The model had a predictive accuracy of 82.11% (78/95 patients), with sensitivity and specificity of 41.67% and 87.95%, respectively.

CONCLUSIONS

Our model can assess the progression of DNI to descending mediastinitis depending on clinical and imaging parameters. It can be used to identify DNI patients who will benefit from prompt treatment.

Enhancing Triage Efficiency and Accuracy in Emergency Rooms for Patients with Metastatic Prostate Cancer: A Retrospective Analysis of Artificial Intelligence-Assisted Triage Using ChatGPT 4.0

BACKGROUND

Accurate and efficient triage is crucial for prioritizing care and managing resources in emergency rooms. This study investigates the effectiveness of ChatGPT, an advanced artificial intelligence system, in assisting health providers with decision-making for patients presenting with metastatic prostate cancer, focusing on the potential to improve both patient outcomes and resource allocation.

METHODS

Clinical data from patients with metastatic prostate cancer who presented to the emergency room between 1 May 2022 and 30 April 2023 were retrospectively collected. The primary outcome was the sensitivity and specificity of ChatGPT in determining whether a patient required admission or discharge. The secondary outcomes included the agreement between ChatGPT and emergency medicine physicians, the comprehensiveness of diagnoses, the accuracy of treatment plans proposed by both parties, and the length of medical decision making.

RESULTS

Of the 147 patients screened, 56 met the inclusion criteria. ChatGPT had a sensitivity of 95.7% in determining admission and a specificity of 18.2% in discharging patients. In 87.5% of cases, ChatGPT made the same primary diagnoses as physicians, with more accurate terminology use (42.9% vs. 21.4%, p = 0.02) and more comprehensive diagnostic lists (median number of diagnoses: 3 vs. 2, p < 0.001). Emergency Severity Index scores calculated by ChatGPT were not associated with admission (p = 0.12), hospital stay length (p = 0.91) or ICU admission (p = 0.54). Despite shorter mean word count (169 ± 66 vs. 272 ± 105, p < 0.001), ChatGPT was more likely to give additional treatment recommendations than physicians (94.3% vs. 73.5%, p < 0.001).

CONCLUSIONS

Our hypothesis-generating data demonstrated that ChatGPT is associated with a high sensitivity in determining the admission of patients with metastatic prostate cancer in the emergency room. It also provides accurate and comprehensive diagnoses. These findings suggest that ChatGPT has the potential to assist health providers in improving patient triage in emergency settings, and may enhance both efficiency and quality of care provided by the physicians.