Optimal Triage for COVID-19 Patients Under Limited Health Care Resources With a Parsimonious Machine Learning Prediction Model and Threshold Optimization Using Discrete-Event Simulation: Development Study

Mapping and Summarizing the Research on AI Systems for Automating Medical History Taking and Triage: Scoping Review

BACKGROUND

The integration of artificial intelligence (AI) systems for automating medical history taking and triage can significantly enhance patient flow in health care systems. Despite the promising performance of numerous AI studies, only a limited number of these systems have been successfully integrated into routine health care practice. To elucidate how AI systems can create value in this context, it is crucial to identify the current state of knowledge, including the readiness of these systems, the facilitators of and barriers to their implementation, and the perspectives of various stakeholders involved in their development and deployment.

OBJECTIVE

This study aims to map and summarize empirical research on AI systems designed for automating medical history taking and triage in health care settings.

METHODS

The study was conducted following the framework proposed by Arksey and O'Malley and adhered to the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) guidelines. A comprehensive search of 5 databases-PubMed, CINAHL, PsycINFO, Scopus, and Web of Science-was performed. A detailed protocol was established before the review to ensure methodological rigor.

RESULTS

A total of 1248 research publications were identified and screened. Of these, 86 (6.89%) met the eligibility criteria. Notably, most (n=63, 73%) studies were published between 2020 and 2022, with a significant concentration on emergency care (n=32, 37%). Other clinical contexts included radiology (n=12, 14%) and primary care (n=6, 7%). Many (n=15, 17%) studies did not specify a clinical context. Most (n=31, 36%) studies used retrospective designs, while others (n=34, 40%) did not specify their methodologies. The predominant type of AI system identified was the hybrid model (n=68, 79%), with forecasting (n=40, 47%) and recognition (n=36, 42%) being the most common tasks performed. While most (n=70, 81%) studies included patient populations, only 1 (1%) study investigated patients' views on AI-based medical history taking and triage, and 2 (2%) studies considered health care professionals' perspectives. Furthermore, only 6 (7%) studies validated or demonstrated AI systems in relevant clinical settings through real-time model testing, workflow implementation, clinical outcome evaluation, or integration into practice. Most (n=76, 88%) studies were concerned with the prototyping, development, or validation of AI systems. In total, 4 (5%) studies were reviews of several empirical studies conducted in different clinical settings. The facilitators and barriers to AI system implementation were categorized into 4 themes: technical aspects, contextual and cultural considerations, end-user engagement, and evaluation processes.

CONCLUSIONS

This review highlights current trends, stakeholder perspectives, stages of innovation development, and key influencing factors related to implementing AI systems in health care. The identified literature gaps regarding stakeholder perspectives and the limited research on AI systems for automating medical history taking and triage indicate significant opportunities for further investigation and development in this evolving field.

Investigating ethical tradeoffs in crisis standards of care through simulation of ventilator allocation protocols

INTRODUCTION

Arguments over the appropriate Crisis Standards of Care (CSC) for public health emergencies often assume that there is a tradeoff between saving the most lives, saving the most life-years, and preventing racial disparities. However, these assumptions have rarely been explored empirically. To quantitatively characterize possible ethical tradeoffs, we aimed to simulate the implementation of five proposed CSC protocols for rationing ventilators in the context of the COVID-19 pandemic.

METHODS

A Monte Carlo simulation was used to estimate the number of lives saved and life-years saved by implementing clinical acuity-, comorbidity- and age-based CSC protocols under different shortage conditions. This model was populated with patient data from 3707 adult admissions requiring ventilator support in a New York hospital system between April 2020 and May 2021. To estimate lives and life-years saved by each protocol, we determined survival to discharge and estimated remaining life expectancy for each admission.

RESULTS

The simulation demonstrated stronger performance for age-sensitive protocols. For a capacity of 1 bed per 2 patients, ranking by age bands saves approximately 29 lives and 3400 life-years per thousand patients. Proposed protocols from New York and Maryland which allocated without considering age saved the fewest lives (~13.2 and 8.5 lives) and life-years (~416 and 420 years). Unlike other protocols, the New York and Maryland algorithms did not generate significant disparities in lives saved and life-years saved between White non-Hispanic, Black non-Hispanic, and Hispanic sub-populations. For all protocols, we observed a positive correlation between lives saved and life-years saved, but also between lives saved overall and inequality in the number of lives saved in different race and ethnicity sub-populations.

CONCLUSION

While there is significant variance in the number of lives saved and life-years saved, we did not find a tradeoff between saving the most lives and saving the most life-years. Moreover, concerns about racial discrimination in triage protocols require thinking carefully about the tradeoff between enforcing equality of survival rates and maximizing the lives saved in each sub-population.

Evaluating accuracy and fairness of clinical decision support algorithms when health care resources are limited

OBJECTIVE

Guidance on how to evaluate accuracy and algorithmic fairness across subgroups is missing for clinical models that flag patients for an intervention but when health care resources to administer that intervention are limited. We aimed to propose a framework of metrics that would fit this specific use case.

METHODS

We evaluated the following metrics and applied them to a Veterans Health Administration clinical model that flags patients for intervention who are at risk of overdose or a suicidal event among outpatients who were prescribed opioids (N = 405,817): Receiver - Operating Characteristic and area under the curve, precision - recall curve, calibration - reliability curve, false positive rate, false negative rate, and false omission rate. In addition, we developed a new approach to visualize false positives and false negatives that we named 'per true positive bars.' We demonstrate the utility of these metrics to our use case for three cohorts of patients at the highest risk (top 0.5 %, 1.0 %, and 5.0 %) by evaluating algorithmic fairness across the following age groups: <=30, 31-50, 51-65, and >65 years old.

RESULTS

Metrics that allowed us to assess group differences more clearly were the false positive rate, false negative rate, false omission rate, and the new 'per true positive bars'. Metrics with limited utility to our use case were the Receiver - Operating Characteristic and area under the curve, the calibration - reliability curve, and the precision - recall curve.

CONCLUSION

There is no "one size fits all" approach to model performance monitoring and bias analysis. Our work informs future researchers and clinicians who seek to evaluate accuracy and fairness of predictive models that identify patients to intervene on in the context of limited health care resources. In terms of ease of interpretation and utility for our use case, the new 'per true positive bars' may be the most intuitive to a range of stakeholders and facilitates choosing a threshold that allows weighing false positives against false negatives, which is especially important when predicting severe adverse events.

Investigating Ethical Tradeoffs in Crisis Standards of Care through Simulation of Ventilator Allocation Protocols

Introduction

Arguments over the appropriate Crisis Standards of Care (CSC) for public health emergencies often assume that there is a tradeoff between saving the most lives, saving the most life-years, and preventing racial disparities. However, these assumptions have rarely been explored empirically. To quantitatively characterize possible ethical tradeoffs, we aimed to simulate the implementation of five proposed CSC protocols for rationing ventilators in the context of the COVID-19 pandemic.

Methods

A Monte Carlo simulation was used to estimate the number of lives saved and life-years saved by implementing clinical acuity-, comorbidity- and age-based CSC protocols under different shortage conditions. This model was populated with patient data from 3707 adult admissions requiring ventilator support in a New York hospital system between April 2020 and May 2021. To estimate lives and life-years saved by each protocol, we determined survival to discharge and estimated remaining life expectancy for each admission.

Results

The simulation demonstrated stronger performance for age- and comorbidity-sensitive protocols. For a capacity of 1 bed per 2 patients, ranking by age bands saves approximately 28.7 lives and 3408 life-years per thousand patients, while ranking by Sequential Organ Failure Assessment (SOFA) bands saved the fewest lives (13.2) and life-years (416). For all protocols, we observed a positive correlation between lives saved and life-years saved. For all protocols except lottery and the banded SOFA, significant disparities in lives saved and life-years saved were noted between White non-Hispanic, Black non-Hispanic, and Hispanic sub-populations.

Conclusion

While there is significant variance in the number of lives saved and life-years saved, we did not find a tradeoff between saving the most lives and saving the most life-years. Moreover, concerns about racial discrimination in triage protocols require thinking carefully about the tradeoff between enforcing equality of survival rates and maximizing the lives saved in each sub-population.

Optimal resource allocation model for COVID-19: a systematic review and meta-analysis

BACKGROUND

A lack of health resources is a common problem after the outbreak of infectious diseases, and resource optimization is an important means to solve the lack of prevention and control capacity caused by resource constraints. This study systematically evaluated the similarities and differences in the application of coronavirus disease (COVID-19) resource allocation models and analyzed the effects of different optimal resource allocations on epidemic control.

METHODS

A systematic literature search was conducted of CNKI, WanFang, VIP, CBD, PubMed, Web of Science, Scopus and Embase for articles published from January 1, 2019, through November 23, 2023. Two reviewers independently evaluated the quality of the included studies, extracted and cross-checked the data. Moreover, publication bias and sensitivity analysis were evaluated.

RESULTS

A total of 22 articles were included for systematic review; in the application of optimal allocation models, 59.09% of the studies used propagation dynamics models to simulate the allocation of various resources, and some scholars also used mathematical optimization functions (36.36%) and machine learning algorithms (31.82%) to solve the problem of resource allocation; the results of the systematic review show that differential equation modeling was more considered when testing resources optimization, the optimization function or machine learning algorithm were mostly used to optimize the bed resources; the meta-analysis results showed that the epidemic trend was obviously effectively controlled through the optimal allocation of resources, and the average control efficiency was 0.38(95%CI 0.25-0.51); Subgroup analysis revealed that the average control efficiency from high to low was health specialists 0.48(95%CI 0.37-0.59), vaccines 0.47(95%CI 0.11-0.82), testing 0.38(95%CI 0.19-0.57), personal protective equipment (PPE) 0.38(95%CI 0.06-0.70), beds 0.34(95%CI 0.14-0.53), medicines and equipment for treatment 0.32(95%CI 0.12-0.51); Funnel plots and Egger's test showed no publication bias, and sensitivity analysis suggested robust results.

CONCLUSION

When the data are insufficient and the simulation time is short, the researchers mostly use the constructor for research; When the data are relatively sufficient and the simulation time is long, researchers choose differential equations or machine learning algorithms for research. In addition, our study showed that control efficiency is an important indicator to evaluate the effectiveness of epidemic prevention and control. Through the optimization of medical staff and vaccine allocation, greater prevention and control effects can be achieved.

Development and Validation of a Robust and Interpretable Early Triaging Support System for Patients Hospitalized With COVID-19: Predictive Algorithm Modeling and Interpretation Study

BACKGROUND

Robust and accurate prediction of severity for patients with COVID-19 is crucial for patient triaging decisions. Many proposed models were prone to either high bias risk or low-to-moderate discrimination. Some also suffered from a lack of clinical interpretability and were developed based on early pandemic period data. Hence, there has been a compelling need for advancements in prediction models for better clinical applicability.

OBJECTIVE

The primary objective of this study was to develop and validate a machine learning-based Robust and Interpretable Early Triaging Support (RIETS) system that predicts severity progression (involving any of the following events: intensive care unit admission, in-hospital death, mechanical ventilation required, or extracorporeal membrane oxygenation required) within 15 days upon hospitalization based on routinely available clinical and laboratory biomarkers.

METHODS

We included data from 5945 hospitalized patients with COVID-19 from 19 hospitals in South Korea collected between January 2020 and August 2022. For model development and external validation, the whole data set was partitioned into 2 independent cohorts by stratified random cluster sampling according to hospital type (general and tertiary care) and geographical location (metropolitan and nonmetropolitan). Machine learning models were trained and internally validated through a cross-validation technique on the development cohort. They were externally validated using a bootstrapped sampling technique on the external validation cohort. The best-performing model was selected primarily based on the area under the receiver operating characteristic curve (AUROC), and its robustness was evaluated using bias risk assessment. For model interpretability, we used Shapley and patient clustering methods.

RESULTS

Our final model, RIETS, was developed based on a deep neural network of 11 clinical and laboratory biomarkers that are readily available within the first day of hospitalization. The features predictive of severity included lactate dehydrogenase, age, absolute lymphocyte count, dyspnea, respiratory rate, diabetes mellitus, c-reactive protein, absolute neutrophil count, platelet count, white blood cell count, and saturation of peripheral oxygen. RIETS demonstrated excellent discrimination (AUROC=0.937; 95% CI 0.935-0.938) with high calibration (integrated calibration index=0.041), satisfied all the criteria of low bias risk in a risk assessment tool, and provided detailed interpretations of model parameters and patient clusters. In addition, RIETS showed potential for transportability across variant periods with its sustainable prediction on Omicron cases (AUROC=0.903, 95% CI 0.897-0.910).

CONCLUSIONS

RIETS was developed and validated to assist early triaging by promptly predicting the severity of hospitalized patients with COVID-19. Its high performance with low bias risk ensures considerably reliable prediction. The use of a nationwide multicenter cohort in the model development and validation implicates generalizability. The use of routinely collected features may enable wide adaptability. Interpretations of model parameters and patients can promote clinical applicability. Together, we anticipate that RIETS will facilitate the patient triaging workflow and efficient resource allocation when incorporated into a routine clinical practice.

Artificial intelligence and discrete-event simulation for capacity management of intensive care units during the Covid-19 pandemic: A case study

The Covid-19 pandemic has pushed the Intensive Care Units (ICUs) into significant operational disruptions. The rapid evolution of this disease, the bed capacity constraints, the wide variety of patient profiles, and the imbalances within health supply chains still represent a challenge for policymakers. This paper aims to use Artificial Intelligence (AI) and Discrete-Event Simulation (DES) to support ICU bed capacity management during Covid-19. The proposed approach was validated in a Spanish hospital chain where we initially identified the predictors of ICU admission in Covid-19 patients. Second, we applied Random Forest (RF) to predict ICU admission likelihood using patient data collected in the Emergency Department (ED). Finally, we included the RF outcomes in a DES model to assist decision-makers in evaluating new ICU bed configurations responding to the patient transfer expected from downstream services. The results evidenced that the median bed waiting time declined between 32.42 and 48.03 min after intervention.

Impact of ICU strain on outcomes

PURPOSE OF REVIEW

Acute surge events result in health capacity strain, which can result in deviations from normal care, activation of contingencies and decisions related to resource allocation. This review discusses the impact of health capacity strain on patient centered outcomes.

RECENT FINDINGS

This manuscript discusses the lack of validated metrics for ICU strain capacity and a need for understanding the complex interrelationships of strain with patient outcomes. Recent work through the coronavirus disease 2019 pandemic has shown that acute surge events are associated with significant increase in hospital mortality. Though causal data on the differential impact of surge actions and resource availability on patient outcomes remains limited the overall signal consistently highlights the link between ICU strain and critical care outcomes in both normal and surge conditions.

SUMMARY

An understanding of ICU strain is fundamental to the appropriate clinical care for critically ill patients. Accounting for stain on outcomes in critically ill patients allows for minimization of variation in care and an ability of a given healthcare system to provide equitable, and quality care even in surge scenarios.

Integration of Machine Learning Algorithms and Discrete-Event Simulation for the Cost of Healthcare Resources

A healthcare resource allocation generally plays a vital role in the number of patients treated (p_nt) and the patient waiting time (w_t) in healthcare institutions. This study aimed to estimate p_nt and w_t as output variables by considering the number of healthcare resources employed and analyze the cost of health resources to the hospital depending on the cost coefficient (δ_i) in an emergency department (ED). The integration of the discrete-event simulation (DES) model and machine learning (ML) algorithms, namely random forest (RF), gradient boosting (GB), and AdaBoost (AB), was used to calculate the estimation of the output variables depending on the δ_i of resources cost. The AB algorithm performed best in almost all scenarios based on the results of the analysis. According to the AB algorithm based on the δ_0.0, δ_0.1, δ_0.2, and δ_0.3, the accuracy data were calculated as 0.9838, 0.9843, 0.9838, and 0.9846 for p_nt; 0.9514, 0.9517, 0.9514, and 0.9514 for w_t, respectively in the training stage. The GB algorithm had the best performance value, except for the results of the δ_0.2 (AB had a better accuracy at 0.8709 based on the value of δ_0.2 for p_nt) in the test stage. According to the AB algorithm based on the δ_0.0, δ_0.1, δ_0.2, and δ_0.3, the accuracy data were calculated as 0.7956, 0.9298, 0.8288, and 0.7394 for p_nt; 0.8820, 0.8821, 0.8819, and 0.8818 for w_t in the training phase, respectively. All scenarios created by the δ_i coefficient should be preferred for ED since the income provided by the p_nt value to the hospital was more than the cost of healthcare resources. On the contrary, the w_t estimation results of ML algorithms based on the δ_i coefficient differed. Although w_t values in all ML algorithms with δ_0.0 and δ_0.1 coefficients reduced the cost of the hospital, w_t values based on δ_0.2 and δ_0.3 increased the cost of the hospital.