Development and Prospective Validation of a Deep Learning Algorithm for Predicting Need for Mechanical Ventilation

Machine learning modelling for predicting the utilization of invasive and non-invasive ventilation throughout the ICU duration

The goal of this work is to develop a Machine Learning model to predict the need for both invasive and non-invasive mechanical ventilation in intensive care unit (ICU) patients. Using the Philips eICU Research Institute (ERI) database, 2.6 million ICU patient data from 2010 to 2019 were analyzed. This data was randomly split into training (63%), validation (27%), and test (10%) sets. Additionally, an external test set from a single hospital from the ERI database was employed to assess the model's generalizability. Model performance was determined by comparing the model probability predictions with the actual incidence of ventilation use, either invasive or non-invasive. The model demonstrated a prediction performance with an AUC of 0.921 for overall ventilation, 0.937 for invasive, and 0.827 for non-invasive. Factors such as high Glasgow Coma Scores, younger age, lower BMI, and lower PaCO2 were highlighted as indicators of a lower likelihood for the need for ventilation. The model can serve as a retrospective benchmarking tool for hospitals to assess ICU performance concerning mechanical ventilation necessity. It also enables analysis of ventilation strategy trends and risk-adjusted comparisons, with potential for future testing as a clinical decision tool for optimizing ICU ventilation management.

The premise, promise, and perils of artificial intelligence in critical care cardiology

Artificial intelligence (AI) is an emerging technology with numerous healthcare applications. AI could prove particularly useful in the cardiac intensive care unit (CICU) where its capacity to analyze large datasets in real-time would assist clinicians in making more informed decisions. This systematic review aimed to explore current research on AI as it pertains to the CICU. A PRISMA search strategy was carried out to identify the pertinent literature on topics including vascular access, heart failure care, circulatory support, cardiogenic shock, ultrasound, and mechanical ventilation. Thirty-eight studies were included. Although AI is still in its early stages of development, this review illustrates its potential to yield numerous benefits in the CICU.

Comparative Analysis of Fusion Strategies for Imaging and Non-imaging Data - Use-case of Hospital Discharge Prediction

Accurate prediction of future clinical events such as discharge from hospital can not only improve hospital resource management but also provide an indicator of a patient's clinical condition. Within the scope of this work, we perform a comparative analysis of deep learning based fusion strategies against traditional single source models for prediction of discharge from hospital by fusing information encoded in two diverse but relevant data modalities, i.e., chest X-ray images and tabular electronic health records (EHR). We evaluate multiple fusion strategies including late, early and joint fusion in terms of their efficacy for target prediction compared to EHR-only and Image-only predictive models. Results indicated the importance of merging information from two modalities for prediction as fusion models tended to outperform single modality models and indicate that the joint fusion scheme was the most effective for target prediction. Joint fusion model merges the two modalities through a branched neural network that is jointly trained in an end-to-end fashion to extract target-relevant information from both modalities.

Noninvasive respiratory support in the emergency department: Controversies and state-of-the-art recommendations

Acute respiratory failure is a common reason for emergency department visits and hospital admissions. Diverse underlying physiologic abnormalities lead to unique aspects about the most common causes of acute respiratory failure: acute decompensated heart failure, acute exacerbation of chronic obstructive pulmonary disease, and acute de novo hypoxemic respiratory failure. Noninvasive respiratory support strategies are increasingly used methods to support work of breathing and improve gas exchange abnormalities to improve outcomes relative to conventional oxygen therapy or invasive mechanical ventilation. Noninvasive respiratory support includes noninvasive positive pressure ventilation and nasal high flow, each with unique physiologic mechanisms. This paper will review the physiology of respiratory failure and noninvasive respiratory support modalities and offer data and guideline-driven recommendations in the context of key clinical controversies.

Prognostic analysis of high-flow nasal cannula therapy and non-invasive ventilation in mild to moderate hypoxemia patients and construction of a machine learning model for 48-h intubation prediction-a retrospective analysis of the MIMIC database

Background

This study aims to investigate the clinical outcome between high-flow nasal cannula (HFNC) and non-invasive ventilation (NIV) therapy in mild to moderate hypoxemic patients on the first ICU day and to develop a predictive model of 48-h intubation.

Methods

The study included adult patients from the MIMIC III and IV databases who first initiated HFNC or NIV therapy due to mild to moderate hypoxemia (100 < PaO2/FiO2 ≤ 300). The 48-h and 30-day intubation rates were compared using cross-sectional and survival analysis. Nine machine learning and six ensemble algorithms were deployed to construct the 48-h intubation predictive models, of which the optimal model was determined by its prediction accuracy. The top 10 risk and protective factors were identified using the Shapley interpretation algorithm.

Result

A total of 123,042 patients were screened, of which, 673 were from the MIMIC IV database for ventilation therapy comparison (HFNC n = 363, NIV n = 310) and 48-h intubation predictive model construction (training dataset n = 471, internal validation set n = 202) and 408 were from the MIMIC III database for external validation. The NIV group had a lower intubation rate (23.1% vs. 16.1%, p = 0.001), ICU 28-day mortality (18.5% vs. 11.6%, p = 0.014), and in-hospital mortality (19.6% vs. 11.9%, p = 0.007) compared to the HFNC group. Survival analysis showed that the total and 48-h intubation rates were not significantly different. The ensemble AdaBoost decision tree model (internal and external validation set AUROC 0.878, 0.726) had the best predictive accuracy performance. The model Shapley algorithm showed Sequential Organ Failure Assessment (SOFA), acute physiology scores (APSIII), the minimum and maximum lactate value as risk factors for early failure and age, the maximum PaCO2 and PH value, Glasgow Coma Scale (GCS), the minimum PaO2/FiO2 ratio, and PaO2 value as protective factors.

Conclusion

NIV was associated with lower intubation rate and ICU 28-day and in-hospital mortality. Further survival analysis reinforced that the effect of NIV on the intubation rate might partly be attributed to the other impact factors. The ensemble AdaBoost decision tree model may assist clinicians in making clinical decisions, and early organ function support to improve patients' SOFA, APSIII, GCS, PaCO2, PaO2, PH, PaO2/FiO2 ratio, and lactate values can reduce the early failure rate and improve patient prognosis.

Development and Validation of a Model for Endotracheal Intubation and Mechanical Ventilation Prediction in PICU Patients

OBJECTIVES

To develop and externally validate an intubation prediction model for children admitted to a PICU using objective and routinely available data from the electronic medical records (EMRs).

DESIGN

Retrospective observational cohort study.

SETTING

Two PICUs within the same healthcare system: an academic, quaternary care center (36 beds) and a community, tertiary care center (56 beds).

PATIENTS

Children younger than 18 years old admitted to a PICU between 2010 and 2022.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Clinical data was extracted from the EMR. PICU stays with at least one mechanical ventilation event (≥ 24 hr) occurring within a window of 1-7 days after hospital admission were included in the study. Of 13,208 PICU stays in the derivation PICU cohort, 1,175 (8.90%) had an intubation event. In the validation cohort, there were 1,165 of 17,841 stays (6.53%) with an intubation event. We trained a Categorical Boosting (CatBoost) model using vital signs, laboratory tests, demographic data, medications, organ dysfunction scores, and other patient characteristics to predict the need of intubation and mechanical ventilation using a 24-hour window of data within their hospital stay. We compared the CatBoost model to an extreme gradient boost, random forest, and a logistic regression model. The area under the receiving operating characteristic curve for the derivation cohort and the validation cohort was 0.88 (95% CI, 0.88-0.89) and 0.92 (95% CI, 0.91-0.92), respectively.

CONCLUSIONS

We developed and externally validated an interpretable machine learning prediction model that improves on conventional clinical criteria to predict the need for intubation in children hospitalized in a PICU using information readily available in the EMR. Implementation of our model may help clinicians optimize the timing of endotracheal intubation and better allocate respiratory and nursing staff to care for mechanically ventilated children.

Predicting Hypoxia Using Machine Learning: Systematic Review

Background

Hypoxia is an important risk factor and indicator for the declining health of inpatients. Predicting future hypoxic events using machine learning is a prospective area of study to facilitate time-critical interventions to counter patient health deterioration.

Objective

This systematic review aims to summarize and compare previous efforts to predict hypoxic events in the hospital setting using machine learning with respect to their methodology, predictive performance, and assessed population.

Methods

A systematic literature search was performed using Web of Science, Ovid with Embase and MEDLINE, and Google Scholar. Studies that investigated hypoxia or hypoxemia of hospitalized patients using machine learning models were considered. Risk of bias was assessed using the Prediction Model Risk of Bias Assessment Tool.

Results

After screening, a total of 12 papers were eligible for analysis, from which 32 models were extracted. The included studies showed a variety of population, methodology, and outcome definition. Comparability was further limited due to unclear or high risk of bias for most studies (10/12, 83%). The overall predictive performance ranged from moderate to high. Based on classification metrics, deep learning models performed similar to or outperformed conventional machine learning models within the same studies. Models using only prior peripheral oxygen saturation as a clinical variable showed better performance than models based on multiple variables, with most of these studies (2/3, 67%) using a long short-term memory algorithm.

Conclusions

Machine learning models provide the potential to accurately predict the occurrence of hypoxic events based on retrospective data. The heterogeneity of the studies and limited generalizability of their results highlight the need for further validation studies to assess their predictive performance.

Integrating Multi-Omics Data With EHR for Precision Medicine Using Advanced Artificial Intelligence

With the recent advancement of novel biomedical technologies such as high-throughput sequencing and wearable devices, multi-modal biomedical data ranging from multi-omics molecular data to real-time continuous bio-signals are generated at an unprecedented speed and scale every day. For the first time, these multi-modal biomedical data are able to make precision medicine close to a reality. However, due to data volume and the complexity, making good use of these multi-modal biomedical data requires major effort. Researchers and clinicians are actively developing artificial intelligence (AI) approaches for data-driven knowledge discovery and causal inference using a variety of biomedical data modalities. These AI-based approaches have demonstrated promising results in various biomedical and healthcare applications. In this review paper, we summarize the state-of-the-art AI models for integrating multi-omics data and electronic health records (EHRs) for precision medicine. We discuss the challenges and opportunities in integrating multi-omics data with EHRs and future directions. We hope this review can inspire future research and developing in integrating multi-omics data with EHRs for precision medicine.

Cohort profile: a large EHR-based cohort with linked pharmacy refill and neighbourhood social determinants of health data to assess heart failure medication adherence

PURPOSE

Clinic-based or community-based interventions can improve adherence to guideline-directed medication therapies (GDMTs) among patients with heart failure (HF). However, opportunities for such interventions are frequently missed, as providers may be unable to recognise risk patterns for medication non-adherence. Machine learning algorithms can help in identifying patients with high likelihood of non-adherence. While a number of multilevel factors influence adherence, prior models predicting non-adherence have been limited by data availability. We have established an electronic health record (EHR)-based cohort with comprehensive data elements from multiple sources to improve on existing models. We linked EHR data with pharmacy refill data for real-time incorporation of prescription fills and with social determinants data to incorporate neighbourhood factors.

PARTICIPANTS

Patients seen at a large health system in New York City (NYC), who were >18 years old with diagnosis of HF or reduced ejection fraction (<40%) since 2017, had at least one clinical encounter between 1 April 2021 and 31 October 2022 and active prescriptions for any of the four GDMTs (beta-blocker, ACEi/angiotensin receptor blocker (ARB)/angiotensin receptor neprilysin inhibitor (ARNI), mineralocorticoid receptor antagonist (MRA) and sodium-glucose cotransporter 2 inhibitor (SGLT2i)) during the study period. Patients with non-geocodable address or outside the continental USA were excluded.

FINDINGS TO DATE

Among 39 963 patients in the cohort, the average age was 73±14 years old, 44% were female and 48% were current/former smokers. The common comorbid conditions were hypertension (77%), cardiac arrhythmias (56%), obesity (33%) and valvular disease (33%). During the study period, 33 606 (84%) patients had an active prescription of beta blocker, 32 626 (82%) had ACEi/ARB/ARNI, 11 611 (29%) MRA and 7472 (19%) SGLT2i. Ninety-nine per cent were from urban metropolitan areas.

FUTURE PLANS

We will use the established cohort to develop a machine learning model to predict medication adherence, and to support ancillary studies assessing associates of adherence. For external validation, we will include data from an additional hospital system in NYC.

Methodologic Issues Specific to Prediction Model Development and Evaluation

Developing and evaluating statistical prediction models is challenging, and many pitfalls can arise. This article identifies what the authors believe are some common methodologic concerns that may be encountered. We describe each problem and make suggestions regarding how to address them. The hope is that this article will result in higher-quality publications of statistical prediction models.

Development of an automated, general-purpose prediction tool for postoperative respiratory failure using machine learning: A retrospective cohort study

STUDY OBJECTIVE

Postoperative respiratory failure is a major surgical complication and key quality metric. Existing prediction tools underperform, are limited to specific populations, and necessitate manual calculation. This limits their implementation. We aimed to create an improved, machine learning powered prediction tool with ideal characteristics for automated calculation.

DESIGN, SETTING, AND PATIENTS

We retrospectively reviewed 101,455 anesthetic procedures from 1/2018 to 6/2021. The primary outcome was the Standardized Endpoints in Perioperative Medicine consensus definition for postoperative respiratory failure. Secondary outcomes were respiratory quality metrics from the National Surgery Quality Improvement Sample, Society of Thoracic Surgeons, and CMS. We abstracted from the electronic health record 26 procedural and physiologic variables previously identified as respiratory failure risk factors. We randomly split the cohort and used the Random Forest method to predict the composite outcome in the training cohort. We coined this the RESPIRE model and measured its accuracy in the validation cohort using area under the receiver operating curve (AUROC) analysis, among other measures, and compared this with ARISCAT and SPORC-1, two leading prediction tools. We compared performance in a validation cohort using score cut-offs determined in a separate test cohort.

MAIN RESULTS

The RESPIRE model exhibited superior accuracy with an AUROC of 0.93 (95% CI, 0.92-0.95) compared to 0.82 for both ARISCAT and SPORC-1 (P-for-difference < 0.0001 for both). At comparable 80-90% sensitivities, RESPIRE had higher positive predictive value (11%, 95% CI: 10-12%) and lower false positive rate (12%, 95% CI: 12-13%) compared to 4% and 37% for both ARISCAT and SPORC-1. The RESPIRE model also better predicted the established quality metrics for postoperative respiratory failure.

CONCLUSIONS

We developed a general-purpose, machine learning powered prediction tool with superior performance for research and quality-based definitions of postoperative respiratory failure.

Machine Learning for Benchmarking Critical Care Outcomes

OBJECTIVES

Enhancing critical care efficacy involves evaluating and improving system functioning. Benchmarking, a retrospective comparison of results against standards, aids risk-adjusted assessment and helps healthcare providers identify areas for improvement based on observed and predicted outcomes. The last two decades have seen the development of several models using machine learning (ML) for clinical outcome prediction. ML is a field of artificial intelligence focused on creating algorithms that enable computers to learn from and make predictions or decisions based on data. This narrative review centers on key discoveries and outcomes to aid clinicians and researchers in selecting the optimal methodology for critical care benchmarking using ML.

METHODS

We used PubMed to search the literature from 2003 to 2023 regarding predictive models utilizing ML for mortality (592 articles), length of stay (143 articles), or mechanical ventilation (195 articles). We supplemented the PubMed search with Google Scholar, making sure relevant articles were included. Given the narrative style, papers in the cohort were manually curated for a comprehensive reader perspective.

RESULTS

Our report presents comparative results for benchmarked outcomes and emphasizes advancements in feature types, preprocessing, model selection, and validation. It showcases instances where ML effectively tackled critical care outcome-prediction challenges, including nonlinear relationships, class imbalances, missing data, and documentation variability, leading to enhanced results.

CONCLUSIONS

Although ML has provided novel tools to improve the benchmarking of critical care outcomes, areas that require further research include class imbalance, fairness, improved calibration, generalizability, and long-term validation of published models.

Prediction of prognosis in COVID-19 patients using machine learning: A systematic review and meta-analysis

BACKGROUND

Accurate prediction of prognostic outcomes in patients with COVID-19 could facilitate clinical decision-making and medical resource allocation. However, little is known about the ability of machine learning (ML) to predict prognosis in COVID-19 patients.

OBJECTIVE

This study aimed to systematically examine the prognostic value of ML in patients with COVID-19.

METHODS

A systematic search was conducted in PubMed, Web of Science, Embase, Cochrane Library, and IEEE Xplore up to December 15, 2021. Studies predicting the prognostic outcomes of COVID-19 patients using ML were eligible for inclusion. Risk of bias was evaluated by a tailored checklist based on Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2). Pooled sensitivity, specificity, and area under the receiver operating curve (AUC) were calculated to evaluate model performance.

RESULTS

A total of 33 studies that described 35 models were eligible for inclusion, with 27 models presenting mortality, four intensive care unit (ICU) admission, and four use of ventilation. For predicting mortality, ML gave a pooled sensitivity of 0.86 (95% CI, 0.79-0.90), a specificity of 0.87 (95% CI, 0.80-0.92), and an AUC of 0.93 (95% CI, 0.90-0.95). For the prediction of ICU admission, ML had a sensitivity of 0.86 (95% CI, 0.78-0.92), a specificity of 0.81 (95% CI, 0.66-0.91), and an AUC of 0.91 (95% CI, 0.88-0.93). For the prediction of ventilation, ML had a sensitivity of 0.81 (95% CI, 0.68-0.90), a specificity of 0.78 (95% CI, 0.66-0.87), and an AUC of 0.87 (95% CI, 0.83-0.89). Meta-regression analyses indicated that algorithm, population, study design, and source of dataset influenced the pooled estimate.

CONCLUSION

This meta-analysis demonstrated the satisfactory performance of ML in predicting prognostic outcomes in patients with COVID-19, suggesting the potential value of ML to support clinical decision-making. However, improvements to methodology and validation are still necessary before its application in routine clinical practice.

Development and validation of a prediction model for mechanical ventilation based on comorbidities in hospitalized patients with COVID-19

Background

Timely recognition of respiratory failure and the need for mechanical ventilation is crucial in managing patients with coronavirus disease 2019 (COVID-19) and reducing hospital mortality rate. A risk stratification tool could assist to avoid clinical deterioration of patients with COVID-19 and optimize allocation of scarce resources. Therefore, we aimed to develop a prediction model for early identification of patients with COVID-19 who may require mechanical ventilation.

Methods

We included patients with COVID-19 hospitalized in United States. Demographic and clinical data were extracted from the records of the Healthcare Cost and Utilization Project State Inpatient Database in 2020. Model construction involved the use of the least absolute shrinkage and selection operator and multivariable logistic regression. The model's performance was evaluated based on discrimination, calibration, and clinical utility.

Results

The training set comprised 73,957 patients (5,971 requiring mechanical ventilation), whereas the validation set included 10,428 (887 requiring mechanical ventilation). The prediction model incorporating age, sex, and 11 other comorbidities (deficiency anemias, congestive heart failure, coagulopathy, dementia, diabetes with chronic complications, complicated hypertension, neurological disorders unaffecting movement, obesity, pulmonary circulation disease, severe renal failure, and weight loss) demonstrated moderate discrimination (area under the curve, 0.715; 95% confidence interval, 0.709-0.722), good calibration (Brier score = 0.070, slope = 1, intercept = 0) and a clinical net benefit with a threshold probability ranged from 2 to 34% in the training set. Similar model's performances were observed in the validation set.

Conclusion

A robust prognostic model utilizing readily available predictors at hospital admission was developed for the early identification of patients with COVID-19 who may require mechanical ventilation. Application of this model could support clinical decision-making to optimize patient management and resource allocation.

Fusion of imaging and non-imaging data for disease trajectory prediction for coronavirus disease 2019 patients

Purpose

Our study investigates whether graph-based fusion of imaging data with non-imaging electronic health records (EHR) data can improve the prediction of the disease trajectories for patients with coronavirus disease 2019 (COVID-19) beyond the prediction performance of only imaging or non-imaging EHR data.

Approach

We present a fusion framework for fine-grained clinical outcome prediction [discharge, intensive care unit (ICU) admission, or death] that fuses imaging and non-imaging information using a similarity-based graph structure. Node features are represented by image embedding, and edges are encoded with clinical or demographic similarity.

Results

Experiments on data collected from the Emory Healthcare Network indicate that our fusion modeling scheme performs consistently better than predictive models developed using only imaging or non-imaging features, with area under the receiver operating characteristics curve of 0.76, 0.90, and 0.75 for discharge from hospital, mortality, and ICU admission, respectively. External validation was performed on data collected from the Mayo Clinic. Our scheme highlights known biases in the model prediction, such as bias against patients with alcohol abuse history and bias based on insurance status.

Conclusions

Our study signifies the importance of the fusion of multiple data modalities for the accurate prediction of clinical trajectories. The proposed graph structure can model relationships between patients based on non-imaging EHR data, and graph convolutional networks can fuse this relationship information with imaging data to effectively predict future disease trajectory more effectively than models employing only imaging or non-imaging data. Our graph-based fusion modeling frameworks can be easily extended to other prediction tasks to efficiently combine imaging data with non-imaging clinical data.

Optimizing the Implementation of Clinical Predictive Models to Minimize National Costs: Sepsis Case Study

BACKGROUND

Sepsis costs and incidence vary dramatically across diagnostic categories, warranting a customized approach for implementing predictive models.

OBJECTIVE

The aim of this study was to optimize the parameters of a sepsis prediction model within distinct patient groups to minimize the excess cost of sepsis care and analyze the potential effect of factors contributing to end-user response to sepsis alerts on overall model utility.

METHODS

We calculated the excess costs of sepsis to the Centers for Medicare and Medicaid Services (CMS) by comparing patients with and without a secondary sepsis diagnosis but with the same primary diagnosis and baseline comorbidities. We optimized the parameters of a sepsis prediction algorithm across different diagnostic categories to minimize these excess costs. At the optima, we evaluated diagnostic odds ratios and analyzed the impact of compliance factors such as noncompliance, treatment efficacy, and tolerance for false alarms on the net benefit of triggering sepsis alerts.

RESULTS

Compliance factors significantly contributed to the net benefit of triggering a sepsis alert. However, a customized deployment policy can achieve a significantly higher diagnostic odds ratio and reduced costs of sepsis care. Implementing our optimization routine with powerful predictive models could result in US $4.6 billion in excess cost savings for CMS.

CONCLUSIONS

We designed a framework for customizing sepsis alert protocols within different diagnostic categories to minimize excess costs and analyzed model performance as a function of false alarm tolerance and compliance with model recommendations. We provide a framework that CMS policymakers could use to recommend minimum adherence rates to the early recognition and appropriate care of sepsis that is sensitive to hospital department-level incidence rates and national excess costs. Customizing the implementation of clinical predictive models by accounting for various behavioral and economic factors may improve the practical benefit of predictive models.

A Scoping Review of the Use of Machine Learning in Health Economics and Outcomes Research: Part 2-Data From Nonwearables

OBJECTIVES

Despite the increasing interest in applying machine learning (ML) methods in health economics and outcomes research (HEOR), stakeholders face uncertainties in when and how ML can be used. We reviewed the recent applications of ML in HEOR.

METHODS

We searched PubMed for studies published between January 2020 and March 2021 and randomly chose 20% of the identified studies for the sake of manageability. Studies that were in HEOR and applied an ML technique were included. Studies related to wearable devices were excluded. We abstracted information on the ML applications, data types, and ML methods and analyzed it using descriptive statistics.

RESULTS

We retrieved 805 articles, of which 161 (20%) were randomly chosen. Ninety-two of the random sample met the eligibility criteria. We found that ML was primarily used for predicting future events (86%) rather than current events (14%). The most common response variables were clinical events or disease incidence (42%) and treatment outcomes (22%). ML was less used to predict economic outcomes such as health resource utilization (16%) or costs (3%). Although electronic medical records (35%) were frequently used for model development, claims data were used less frequently (9%). Tree-based methods (eg, random forests and boosting) were the most commonly used ML methods (31%).

CONCLUSIONS

The use of ML techniques in HEOR is growing rapidly, but there remain opportunities to apply them to predict economic outcomes, especially using claims databases, which could inform the development of cost-effectiveness models.

COVID-19 machine learning model predicts outcomes in older patients from various European countries, between pandemic waves, and in a cohort of Asian, African, and American patients

BACKGROUND

COVID-19 remains a complex disease in terms of its trajectory and the diversity of outcomes rendering disease management and clinical resource allocation challenging. Varying symptomatology in older patients as well as limitation of clinical scoring systems have created the need for more objective and consistent methods to aid clinical decision making. In this regard, machine learning methods have been shown to enhance prognostication, while improving consistency. However, current machine learning approaches have been limited by lack of generalisation to diverse patient populations, between patients admitted at different waves and small sample sizes.

OBJECTIVES

We sought to investigate whether machine learning models, derived on routinely collected clinical data, can generalise well i) between European countries, ii) between European patients admitted at different COVID-19 waves, and iii) between geographically diverse patients, namely whether a model derived on the European patient cohort can be used to predict outcomes of patients admitted to Asian, African and American ICUs.

METHODS

We compare Logistic Regression, Feed Forward Neural Network and XGBoost algorithms to analyse data from 3,933 older patients with a confirmed COVID-19 diagnosis in predicting three outcomes, namely: ICU mortality, 30-day mortality and patients at low risk of deterioration. The patients were admitted to ICUs located in 37 countries, between January 11, 2020, and April 27, 2021.

RESULTS

The XGBoost model derived on the European cohort and externally validated in cohorts of Asian, African, and American patients, achieved AUC of 0.89 (95% CI 0.89-0.89) in predicting ICU mortality, AUC of 0.86 (95% CI 0.86-0.86) for 30-day mortality prediction and AUC of 0.86 (95% CI 0.86-0.86) in predicting low-risk patients. Similar AUC performance was achieved also when predicting outcomes between European countries and between pandemic waves, while the models showed high calibration quality. Furthermore, saliency analysis showed that FiO2 values of up to 40% do not appear to increase the predicted risk of ICU and 30-day mortality, while PaO2 values of 75 mmHg or lower are associated with a sharp increase in the predicted risk of ICU and 30-day mortality. Lastly, increase in SOFA scores also increase the predicted risk, but only up to a value of 8. Beyond these scores the predicted risk remains consistently high.

CONCLUSION

The models captured both the dynamic course of the disease as well as similarities and differences between the diverse patient cohorts, enabling prediction of disease severity, identification of low-risk patients and potentially supporting effective planning of essential clinical resources.

TRIAL REGISTRATION NUMBER

NCT04321265.

Graph-based Fusion Modeling and Explanation for Disease Trajectory Prediction

We propose a relational graph to incorporate clinical similarity between patients while building personalized clinical event predictors with a focus on hospitalized COVID-19 patients. Our graph formation process fuses heterogeneous data, i.e., chest X-rays as node features and non-imaging EHR for edge formation. While node represents a snap-shot in time for a single patient, weighted edge structure encodes complex clinical patterns among patients. While age and gender have been used in the past for patient graph formation, our method incorporates complex clinical history while avoiding manual feature selection. The model learns from the patient's own data as well as patterns among clinically-similar patients. Our visualization study investigates the effects of 'neighborhood' of a node on its predictiveness and showcases the model's tendency to focus on edge-connected patients with highly suggestive clinical features common with the node. The proposed model generalizes well by allowing edge formation process to adapt to an external cohort.

Disease-course adapting machine learning prognostication models in critically ill elderly COVID-19 patients: a multi-centre cohort study with external validation

Background

The COVID-19 pandemic caused by SARS-CoV-2 is challenging health care systems globally. The disease disproportionately affects the elderly population, both in terms of disease severity and mortality risk.

Objective

The aim of this study was to evaluate machine learning–based prognostication models for critically ill elderly COVID-19 patients, which dynamically incorporated multifaceted clinical information on evolution of the disease.

Methods

This multicenter cohort study (COVIP study) obtained patient data from 151 intensive care units (ICUs) from 26 countries. Different models based on the Sequential Organ Failure Assessment (SOFA) score, logistic regression (LR), random forest (RF), and extreme gradient boosting (XGB) were derived as baseline models that included admission variables only. We subsequently included clinical events and time-to-event as additional variables to derive the final models using the same algorithms and compared their performance with that of the baseline group. Furthermore, we derived baseline and final models on a European patient cohort, which were externally validated on a non-European cohort that included Asian, African, and US patients.

Results

In total, 1432 elderly (≥70 years old) COVID-19–positive patients admitted to an ICU were included for analysis. Of these, 809 (56.49%) patients survived up to 30 days after admission. The average length of stay was 21.6 (SD 18.2) days. Final models that incorporated clinical events and time-to-event information provided superior performance (area under the receiver operating characteristic curve of 0.81; 95% CI 0.804-0.811), with respect to both the baseline models that used admission variables only and conventional ICU prediction models (SOFA score, P<.001). The average precision increased from 0.65 (95% CI 0.650-0.655) to 0.77 (95% CI 0.759-0.770).

Conclusions

Integrating important clinical events and time-to-event information led to a superior accuracy of 30-day mortality prediction compared with models based on the admission information and conventional ICU prediction models. This study shows that machine-learning models provide additional information and may support complex decision-making in critically ill elderly COVID-19 patients.

Trial Registration

ClinicalTrials.gov NCT04321265; https://clinicaltrials.gov/ct2/show/NCT04321265

Wearable sensor derived decompensation index for continuous remote monitoring of COVID-19 diagnosed patients

The COVID-19 pandemic has accelerated the adoption of innovative healthcare methods, including remote patient monitoring. In the setting of limited healthcare resources, outpatient management of individuals newly diagnosed with COVID-19 was commonly implemented, some taking advantage of various personal health technologies, but only rarely using a multi-parameter chest-patch for continuous monitoring. Here we describe the development and validation of a COVID-19 decompensation index (CDI) model based on chest patch-derived continuous sensor data to predict COVID-19 hospitalizations in outpatient-managed COVID-19 positive individuals, achieving an overall AUC of the ROC Curve of 0.84 on 308 event negative participants, and 22 event positive participants, out of an overall study cohort of 400 participants. We retrospectively compare the performance of CDI to standard of care modalities, finding that the machine learning model outperforms the standard of care modalities in terms of both numbers of events identified and with a lower false alarm rate. While only a pilot phase study, the CDI represents a promising application of machine learning within a continuous remote patient monitoring system.

Role of SpO2/FiO2 Ratio and ROX Index in Predicting Early Invasive Mechanical Ventilation in COVID-19. A Pragmatic, Retrospective, Multi-Center Study

The ability of COVID-19 to compromise the respiratory system has generated a substantial proportion of critically ill patients in need of invasive mechanical ventilation (IMV). The objective of this paper was to analyze the prognostic ability of the pulse oximetry saturation/fraction of inspired oxygen ratio (SpO2/FiO2) and the ratio of SpO2/FiO2 to the respiratory rate-ROX index-as predictors of IMV in an emergency department in confirmed COVID-19 patients. A multicenter, retrospective cohort study was carried out in four provinces of Spain between March and November 2020. The discriminative power of the predictive variable was assessed through a prediction model trained using a derivation sub-cohort and evaluated by the area under the curve (AUC) of the receiver operating characteristic (ROC) on the validation sub-cohort. A total of 2040 patients were included in the study. The IMV rate was 10.1%, with an in-hospital mortality rate of 35.3%. The performance of the SpO2/FiO2 ratio was better than the ROX index-AUC = 0.801 (95% CI 0.746-0.855) and AUC = 0.725 (95% CI 0.652-0.798), respectively. In fact, a direct comparison between AUCs resulted in significant differences (p = 0.001). SpO2 to FiO2 ratio is a simple and promising non-invasive tool for predicting risk of IMV in patients infected with COVID-19, and it is realizable in emergency departments.

Patient-specific COVID-19 resource utilization prediction using fusion AI model

The strain on healthcare resources brought forth by the recent COVID-19 pandemic has highlighted the need for efficient resource planning and allocation through the prediction of future consumption. Machine learning can predict resource utilization such as the need for hospitalization based on past medical data stored in electronic medical records (EMR). We conducted this study on 3194 patients (46% male with mean age 56.7 (±16.8), 56% African American, 7% Hispanic) flagged as COVID-19 positive cases in 12 centers under Emory Healthcare network from February 2020 to September 2020, to assess whether a COVID-19 positive patient’s need for hospitalization can be predicted at the time of RT-PCR test using the EMR data prior to the test. Five main modalities of EMR, i.e., demographics, medication, past medical procedures, comorbidities, and laboratory results, were used as features for predictive modeling, both individually and fused together using late, middle, and early fusion. Models were evaluated in terms of precision, recall, F1-score (within 95% confidence interval). The early fusion model is the most effective predictor with 84% overall F1-score [CI 82.1–86.1]. The predictive performance of the model drops by 6 % when using recent clinical data while omitting the long-term medical history. Feature importance analysis indicates that history of cardiovascular disease, emergency room visits in the past year prior to testing, and demographic factors are predictive of the disease trajectory. We conclude that fusion modeling using medical history and current treatment data can forecast the need for hospitalization for patients infected with COVID-19 at the time of the RT-PCR test.

Machine Learning Models to Predict 30-Day Mortality in Mechanically Ventilated Patients

Previous scoring models, such as the Acute Physiologic Assessment and Chronic Health Evaluation II (APACHE II) score, do not adequately predict the mortality of patients receiving mechanical ventilation in the intensive care unit. Therefore, this study aimed to apply machine learning algorithms to improve the prediction accuracy for 30-day mortality of mechanically ventilated patients. The data of 16,940 mechanically ventilated patients were divided into the training-validation (83%, n = 13,988) and test (17%, n = 2952) sets. Machine learning algorithms including balanced random forest, light gradient boosting machine, extreme gradient boost, multilayer perceptron, and logistic regression were used. We compared the area under the receiver operating characteristic curves (AUCs) of machine learning algorithms with those of the APACHE II and ProVent score results. The extreme gradient boost model showed the highest AUC (0.79 (0.77–0.80)) for the 30-day mortality prediction, followed by the balanced random forest model (0.78 (0.76–0.80)). The AUCs of these machine learning models as achieved by APACHE II and ProVent scores were higher than 0.67 (0.65–0.69), and 0.69 (0.67–0.71)), respectively. The most important variables in developing each machine learning model were APACHE II score, Charlson comorbidity index, and norepinephrine. The machine learning models have a higher AUC than conventional scoring systems, and can thus better predict the 30-day mortality of mechanically ventilated patients.

Artificial intelligence in clinical care amidst COVID-19 pandemic: A systematic review

The worldwide health crisis caused by the SARS-Cov-2 virus has resulted in>3 million deaths so far. Improving early screening, diagnosis and prognosis of the disease are critical steps in assisting healthcare professionals to save lives during this pandemic. Since WHO declared the COVID-19 outbreak as a pandemic, several studies have been conducted using Artificial Intelligence techniques to optimize these steps on clinical settings in terms of quality, accuracy and most importantly time. The objective of this study is to conduct a systematic literature review on published and preprint reports of Artificial Intelligence models developed and validated for screening, diagnosis and prognosis of the coronavirus disease 2019. We included 101 studies, published from January 1st, 2020 to December 30th, 2020, that developed AI prediction models which can be applied in the clinical setting. We identified in total 14 models for screening, 38 diagnostic models for detecting COVID-19 and 50 prognostic models for predicting ICU need, ventilator need, mortality risk, severity assessment or hospital length stay. Moreover, 43 studies were based on medical imaging and 58 studies on the use of clinical parameters, laboratory results or demographic features. Several heterogeneous predictors derived from multimodal data were identified. Analysis of these multimodal data, captured from various sources, in terms of prominence for each category of the included studies, was performed. Finally, Risk of Bias (RoB) analysis was also conducted to examine the applicability of the included studies in the clinical setting and assist healthcare providers, guideline developers, and policymakers.

Role of Machine Learning Techniques to Tackle the COVID-19 Crisis: Systematic Review

BACKGROUND

SARS-CoV-2, the novel coronavirus responsible for COVID-19, has caused havoc worldwide, with patients presenting a spectrum of complications that have pushed health care experts to explore new technological solutions and treatment plans. Artificial Intelligence (AI)-based technologies have played a substantial role in solving complex problems, and several organizations have been swift to adopt and customize these technologies in response to the challenges posed by the COVID-19 pandemic.

OBJECTIVE

The objective of this study was to conduct a systematic review of the literature on the role of AI as a comprehensive and decisive technology to fight the COVID-19 crisis in the fields of epidemiology, diagnosis, and disease progression.

METHODS

A systematic search of PubMed, Web of Science, and CINAHL databases was performed according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) guidelines to identify all potentially relevant studies published and made available online between December 1, 2019, and June 27, 2020. The search syntax was built using keywords specific to COVID-19 and AI.

RESULTS

The search strategy resulted in 419 articles published and made available online during the aforementioned period. Of these, 130 publications were selected for further analyses. These publications were classified into 3 themes based on AI applications employed to combat the COVID-19 crisis: Computational Epidemiology, Early Detection and Diagnosis, and Disease Progression. Of the 130 studies, 71 (54.6%) focused on predicting the COVID-19 outbreak, the impact of containment policies, and potential drug discoveries, which were classified under the Computational Epidemiology theme. Next, 40 of 130 (30.8%) studies that applied AI techniques to detect COVID-19 by using patients' radiological images or laboratory test results were classified under the Early Detection and Diagnosis theme. Finally, 19 of the 130 studies (14.6%) that focused on predicting disease progression, outcomes (ie, recovery and mortality), length of hospital stay, and number of days spent in the intensive care unit for patients with COVID-19 were classified under the Disease Progression theme.

CONCLUSIONS

In this systematic review, we assembled studies in the current COVID-19 literature that utilized AI-based methods to provide insights into different COVID-19 themes. Our findings highlight important variables, data types, and available COVID-19 resources that can assist in facilitating clinical and translational research.

Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal

OBJECTIVE

To review and appraise the validity and usefulness of published and preprint reports of prediction models for diagnosing coronavirus disease 2019 (covid-19) in patients with suspected infection, for prognosis of patients with covid-19, and for detecting people in the general population at increased risk of covid-19 infection or being admitted to hospital with the disease.

DESIGN

Living systematic review and critical appraisal by the COVID-PRECISE (Precise Risk Estimation to optimise covid-19 Care for Infected or Suspected patients in diverse sEttings) group.

DATA SOURCES

PubMed and Embase through Ovid, up to 1 July 2020, supplemented with arXiv, medRxiv, and bioRxiv up to 5 May 2020.

STUDY SELECTION

Studies that developed or validated a multivariable covid-19 related prediction model.

DATA EXTRACTION

At least two authors independently extracted data using the CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist; risk of bias was assessed using PROBAST (prediction model risk of bias assessment tool).

RESULTS

37 421 titles were screened, and 169 studies describing 232 prediction models were included. The review identified seven models for identifying people at risk in the general population; 118 diagnostic models for detecting covid-19 (75 were based on medical imaging, 10 to diagnose disease severity); and 107 prognostic models for predicting mortality risk, progression to severe disease, intensive care unit admission, ventilation, intubation, or length of hospital stay. The most frequent types of predictors included in the covid-19 prediction models are vital signs, age, comorbidities, and image features. Flu-like symptoms are frequently predictive in diagnostic models, while sex, C reactive protein, and lymphocyte counts are frequent prognostic factors. Reported C index estimates from the strongest form of validation available per model ranged from 0.71 to 0.99 in prediction models for the general population, from 0.65 to more than 0.99 in diagnostic models, and from 0.54 to 0.99 in prognostic models. All models were rated at high or unclear risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, high risk of model overfitting, and unclear reporting. Many models did not include a description of the target population (n=27, 12%) or care setting (n=75, 32%), and only 11 (5%) were externally validated by a calibration plot. The Jehi diagnostic model and the 4C mortality score were identified as promising models.

CONCLUSION

Prediction models for covid-19 are quickly entering the academic literature to support medical decision making at a time when they are urgently needed. This review indicates that almost all pubished prediction models are poorly reported, and at high risk of bias such that their reported predictive performance is probably optimistic. However, we have identified two (one diagnostic and one prognostic) promising models that should soon be validated in multiple cohorts, preferably through collaborative efforts and data sharing to also allow an investigation of the stability and heterogeneity in their performance across populations and settings. Details on all reviewed models are publicly available at https://www.covprecise.org/. Methodological guidance as provided in this paper should be followed because unreliable predictions could cause more harm than benefit in guiding clinical decisions. Finally, prediction model authors should adhere to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) reporting guideline.

SYSTEMATIC REVIEW REGISTRATION

Protocol https://osf.io/ehc47/, registration https://osf.io/wy245.

READERS' NOTE

This article is a living systematic review that will be updated to reflect emerging evidence. Updates may occur for up to two years from the date of original publication. This version is update 3 of the original article published on 7 April 2020 (BMJ 2020;369:m1328). Previous updates can be found as data supplements (https://www.bmj.com/content/369/bmj.m1328/related#datasupp). When citing this paper please consider adding the update number and date of access for clarity.