Development and external validation of deep learning clinical prediction models using variable-length time series data

OBJECTIVES

To compare and externally validate popular deep learning model architectures and data transformation methods for variable-length time series data in 3 clinical tasks (clinical deterioration, severe acute kidney injury [AKI], and suspected infection).

MATERIALS AND METHODS

This multicenter retrospective study included admissions at 2 medical centers that spanned 2007-2022. Distinct datasets were created for each clinical task, with 1 site used for training and the other for testing. Three feature engineering methods (normalization, standardization, and piece-wise linear encoding with decision trees [PLE-DTs]) and 3 architectures (long short-term memory/gated recurrent unit [LSTM/GRU], temporal convolutional network, and time-distributed wrapper with convolutional neural network [TDW-CNN]) were compared in each clinical task. Model discrimination was evaluated using the area under the precision-recall curve (AUPRC) and the area under the receiver operating characteristic curve (AUROC).

RESULTS

The study comprised 373 825 admissions for training and 256 128 admissions for testing. LSTM/GRU models tied with TDW-CNN models with both obtaining the highest mean AUPRC in 2 tasks, and LSTM/GRU had the highest mean AUROC across all tasks (deterioration: 0.81, AKI: 0.92, infection: 0.87). PLE-DT with LSTM/GRU achieved the highest AUPRC in all tasks.

DISCUSSION

When externally validated in 3 clinical tasks, the LSTM/GRU model architecture with PLE-DT transformed data demonstrated the highest AUPRC in all tasks. Multiple models achieved similar performance when evaluated using AUROC.

CONCLUSION

The LSTM architecture performs as well or better than some newer architectures, and PLE-DT may enhance the AUPRC in variable-length time series data for predicting clinical outcomes during external validation.

Automated stratification of trauma injury severity across multiple body regions using multi-modal, multi-class machine learning models

OBJECTIVE

The timely stratification of trauma injury severity can enhance the quality of trauma care but it requires intense manual annotation from certified trauma coders. The objective of this study is to develop machine learning models for the stratification of trauma injury severity across various body regions using clinical text and structured electronic health records (EHRs) data.

MATERIALS AND METHODS

Our study utilized clinical documents and structured EHR variables linked with the trauma registry data to create 2 machine learning models with different approaches to representing text. The first one fuses concept unique identifiers (CUIs) extracted from free text with structured EHR variables, while the second one integrates free text with structured EHR variables. Temporal validation was undertaken to ensure the models' temporal generalizability. Additionally, analyses to assess the variable importance were conducted.

RESULTS

Both models demonstrated impressive performance in categorizing leg injuries, achieving high accuracy with macro-F1 scores of over 0.8. Additionally, they showed considerable accuracy, with macro-F1 scores exceeding or near 0.7, in assessing injuries in the areas of the chest and head. We showed in our variable importance analysis that the most important features in the model have strong face validity in determining clinically relevant trauma injuries.

DISCUSSION

The CUI-based model achieves comparable performance, if not higher, compared to the free-text-based model, with reduced complexity. Furthermore, integrating structured EHR data improves performance, particularly when the text modalities are insufficiently indicative.

CONCLUSIONS

Our multi-modal, multiclass models can provide accurate stratification of trauma injury severity and clinically relevant interpretations.

Development of a Human Evaluation Framework and Correlation with Automated Metrics for Natural Language Generation of Medical Diagnoses

In the evolving landscape of clinical Natural Language Generation (NLG), assessing abstractive text quality remains challenging, as existing methods often overlook generative task complexities. This work aimed to examine the current state of automated evaluation metrics in NLG in healthcare. To have a robust and well-validated baseline with which to examine the alignment of these metrics, we created a comprehensive human evaluation framework. Employing ChatGPT-3.5-turbo generative output, we correlated human judgments with each metric. None of the metrics demonstrated high alignment; however, the SapBERT score-a Unified Medical Language System (UMLS)- showed the best results. This underscores the importance of incorporating domain-specific knowledge into evaluation efforts. Our work reveals the deficiency in quality evaluations for generated text and introduces our comprehensive human evaluation framework as a baseline. Future efforts should prioritize integrating medical knowledge databases to enhance the alignment of automated metrics, particularly focusing on refining the SapBERT score for improved assessments.

The Pediatric Data Science and Analytics Subgroup of the Pediatric Acute Lung Injury and Sepsis Investigators Network: Use of Supervised Machine Learning Applications in Pediatric Critical Care Medicine Research

OBJECTIVE

Perform a scoping review of supervised machine learning in pediatric critical care to identify published applications, methodologies, and implementation frequency to inform best practices for the development, validation, and reporting of predictive models in pediatric critical care.

DESIGN

Scoping review and expert opinion.

SETTING

We queried CINAHL Plus with Full Text (EBSCO), Cochrane Library (Wiley), Embase (Elsevier), Ovid Medline, and PubMed for articles published between 2000 and 2022 related to machine learning concepts and pediatric critical illness. Articles were excluded if the majority of patients were adults or neonates, if unsupervised machine learning was the primary methodology, or if information related to the development, validation, and/or implementation of the model was not reported. Article selection and data extraction were performed using dual review in the Covidence tool, with discrepancies resolved by consensus.

SUBJECTS

Articles reporting on the development, validation, or implementation of supervised machine learning models in the field of pediatric critical care medicine.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Of 5075 identified studies, 141 articles were included. Studies were primarily (57%) performed at a single site. The majority took place in the United States (70%). Most were retrospective observational cohort studies. More than three-quarters of the articles were published between 2018 and 2022. The most common algorithms included logistic regression and random forest. Predicted events were most commonly death, transfer to ICU, and sepsis. Only 14% of articles reported external validation, and only a single model was implemented at publication. Reporting of validation methods, performance assessments, and implementation varied widely. Follow-up with authors suggests that implementation remains uncommon after model publication.

CONCLUSIONS

Publication of supervised machine learning models to address clinical challenges in pediatric critical care medicine has increased dramatically in the last 5 years. While these approaches have the potential to benefit children with critical illness, the literature demonstrates incomplete reporting, absence of external validation, and infrequent clinical implementation.

Predicting transfers to intensive care in children using CEWT and other early warning systems

Background and Objective

The Children's Early Warning Tool (CEWT), developed in Australia, is widely used in many countries to monitor the risk of deterioration in hospitalized children. Our objective was to compare CEWT prediction performance against a version of the Bedside Pediatric Early Warning Score (Bedside PEWS), Between the Flags (BTF), and the pediatric Calculated Assessment of Risk and Triage (pCART).

Methods

We conducted a retrospective observational study of all patient admissions to the Comer Children's Hospital at the University of Chicago between 2009-2019. We compared performance for predicting the primary outcome of a direct ward-to-intensive care unit (ICU) transfer within the next 12 h using the area under the receiver operating characteristic curve (AUC). Alert rates at various score thresholds were also compared.

Results

Of 50,815 ward admissions, 1,874 (3.7%) experienced the primary outcome. Among patients in Cohort 1 (years 2009-2017, on which the machine learning-based pCART was trained), CEWT performed slightly worse than Bedside PEWS but better than BTF (CEWT AUC 0.74 vs. Bedside PEWS 0.76, P < 0.001; vs. BTF 0.66, P < 0.001), while pCART performed best for patients in Cohort 2 (years 2018-2019, pCART AUC 0.84 vs. CEWT AUC 0.79, P < 0.001; vs. BTF AUC 0.67, P < 0.001; vs. Bedside PEWS 0.80, P < 0.001). Sensitivity, specificity, and positive predictive values varied across all four tools at the examined thresholds for alerts.

Conclusion

CEWT has good discrimination for predicting which patients will likely be transferred to the ICU, while pCART performed the best.

Alcohol Withdrawal Severity Measures for Identifying Patients Requiring High-Intensity Care

OBJECTIVES

Alcohol withdrawal syndrome (AWS) may progress to require high-intensity care. Approaches to identify hospitalized patients with AWS who received higher level of care have not been previously examined. This study aimed to examine the utility of Clinical Institute Withdrawal Assessment Alcohol Revised (CIWA-Ar) for alcohol scale scores and medication doses for alcohol withdrawal management in identifying patients who received high-intensity care.

DESIGN

A multicenter observational cohort study of hospitalized adults with alcohol withdrawal.

SETTING

University of Chicago Medical Center and University of Wisconsin Hospital.

PATIENTS

Inpatient encounters between November 2008 and February 2022 with a CIWA-Ar score greater than 0 and benzodiazepine or barbiturate administered within the first 24 hours. The primary composite outcome was patients who progressed to high-intensity care (intermediate care or ICU).

INTERVENTIONS

None.

MAIN RESULTS

Among the 8742 patients included in the study, 37.5% (n = 3280) progressed to high-intensity care. The odds ratio for the composite outcome increased above 1.0 when the CIWA-Ar score was 24. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) at this threshold were 0.12 (95% CI, 0.11-0.13), 0.95 (95% CI, 0.94-0.95), 0.58 (95% CI, 0.54-0.61), and 0.64 (95% CI, 0.63-0.65), respectively. The OR increased above 1.0 at a 24-hour lorazepam milligram equivalent dose cutoff of 15 mg. The sensitivity, specificity, PPV, and NPV at this threshold were 0.16 (95% CI, 0.14-0.17), 0.96 (95% CI, 0.95-0.96), 0.68 (95% CI, 0.65-0.72), and 0.65 (95% CI, 0.64-0.66), respectively.

CONCLUSIONS

Neither CIWA-Ar scores nor medication dose cutoff points were effective measures for identifying patients with alcohol withdrawal who received high-intensity care. Research studies for examining outcomes in patients who deteriorate with AWS will require better methods for cohort identification.

Development and Validation of a Risk Score Predicting Death Without Transplant in Adult Heart Transplant Candidates

Importance

The US heart allocation system prioritizes medically urgent candidates with a high risk of dying without transplant. The current therapy-based 6-status system is susceptible to manipulation and has limited rank ordering ability.

Objective

To develop and validate a candidate risk score that incorporates current clinical, laboratory, and hemodynamic data.

Design, Setting, and Participants

A registry-based observational study of adult heart transplant candidates (aged ≥18 years) from the US heart allocation system listed between January 1, 2019, and December 31, 2022, split by center into training (70%) and test (30%) datasets. Adult candidates were listed between January 1, 2019, and December 31, 2022.

Main Outcomes and Measures

A US candidate risk score (US-CRS) model was developed by adding a predefined set of predictors to the current French Candidate Risk Score (French-CRS) model. Sensitivity analyses were performed, which included intra-aortic balloon pumps (IABP) and percutaneous ventricular assist devices (VAD) in the definition of short-term mechanical circulatory support (MCS) for the US-CRS. Performance of the US-CRS model, French-CRS model, and 6-status model in the test dataset was evaluated by time-dependent area under the receiver operating characteristic curve (AUC) for death without transplant within 6 weeks and overall survival concordance (c-index) with integrated AUC.

Results

A total of 16 905 adult heart transplant candidates were listed (mean [SD] age, 53 [13] years; 73% male; 58% White); 796 patients (4.7%) died without a transplant. The final US-CRS contained time-varying short-term MCS (ventricular assist-extracorporeal membrane oxygenation or temporary surgical VAD), the log of bilirubin, estimated glomerular filtration rate, the log of B-type natriuretic peptide, albumin, sodium, and durable left ventricular assist device. In the test dataset, the AUC for death within 6 weeks of listing for the US-CRS model was 0.79 (95% CI, 0.75-0.83), for the French-CRS model was 0.72 (95% CI, 0.67-0.76), and 6-status model was 0.68 (95% CI, 0.62-0.73). Overall c-index for the US-CRS model was 0.76 (95% CI, 0.73-0.80), for the French-CRS model was 0.69 (95% CI, 0.65-0.73), and 6-status model was 0.67 (95% CI, 0.63-0.71). Classifying IABP and percutaneous VAD as short-term MCS reduced the effect size by 54%.

Conclusions and Relevance

In this registry-based study of US heart transplant candidates, a continuous multivariable allocation score outperformed the 6-status system in rank ordering heart transplant candidates by medical urgency and may be useful for the medical urgency component of heart allocation.

Association between mortality and critical events within 48 hours of transfer to the pediatric intensive care unit

Introduction

Critical deterioration in hospitalized children, defined as ward to pediatric intensive care unit (PICU) transfer followed by mechanical ventilation (MV) or vasoactive infusion (VI) within 12 h, has been used as a primary metric to evaluate the effectiveness of clinical interventions or quality improvement initiatives. We explore the association between critical events (CEs), i.e., MV or VI events, within the first 48 h of PICU transfer from the ward or emergency department (ED) and in-hospital mortality.

Methods

We conducted a retrospective study of a cohort of PICU transfers from the ward or the ED at two tertiary-care academic hospitals. We determined the association between mortality and occurrence of CEs within 48 h of PICU transfer after adjusting for age, gender, hospital, and prior comorbidities.

Results

Experiencing a CE within 48 h of PICU transfer was associated with an increased risk of mortality [OR 12.40 (95% CI: 8.12-19.23, P < 0.05)]. The increased risk of mortality was highest in the first 12 h [OR 11.32 (95% CI: 7.51-17.15, P < 0.05)] but persisted in the 12-48 h time interval [OR 2.84 (95% CI: 1.40-5.22, P < 0.05)]. Varying levels of risk were observed when considering ED or ward transfers only, when considering different age groups, and when considering individual 12-h time intervals.

Discussion

We demonstrate that occurrence of a CE within 48 h of PICU transfer was associated with mortality after adjusting for confounders. Studies focusing on the impact of quality improvement efforts may benefit from using CEs within 48 h of PICU transfer as an additional evaluation metric, provided these events could have been influenced by the initiative.

Development and external validation of multimodal postoperative acute kidney injury risk machine learning models

Objectives

To develop and externally validate machine learning models using structured and unstructured electronic health record data to predict postoperative acute kidney injury (AKI) across inpatient settings.

Materials and Methods

Data for adult postoperative admissions to the Loyola University Medical Center (2009-2017) were used for model development and admissions to the University of Wisconsin-Madison (2009-2020) were used for validation. Structured features included demographics, vital signs, laboratory results, and nurse-documented scores. Unstructured text from clinical notes were converted into concept unique identifiers (CUIs) using the clinical Text Analysis and Knowledge Extraction System. The primary outcome was the development of Kidney Disease Improvement Global Outcomes stage 2 AKI within 7 days after leaving the operating room. We derived unimodal extreme gradient boosting machines (XGBoost) and elastic net logistic regression (GLMNET) models using structured-only data and multimodal models combining structured data with CUI features. Model comparison was performed using the receiver operating characteristic curve (AUROC), with Delong's test for statistical differences.

Results

The study cohort included 138 389 adult patient admissions (mean [SD] age 58 [16] years; 11 506 [8%] African-American; and 70 826 [51%] female) across the 2 sites. Of those, 2959 (2.1%) developed stage 2 AKI or higher. Across all data types, XGBoost outperformed GLMNET (mean AUROC 0.81 [95% confidence interval (CI), 0.80-0.82] vs 0.78 [95% CI, 0.77-0.79]). The multimodal XGBoost model incorporating CUIs parameterized as term frequency-inverse document frequency (TF-IDF) showed the highest discrimination performance (AUROC 0.82 [95% CI, 0.81-0.83]) over unimodal models (AUROC 0.79 [95% CI, 0.78-0.80]).

Discussion

A multimodality approach with structured data and TF-IDF weighting of CUIs increased model performance over structured data-only models.

Conclusion

These findings highlight the predictive power of CUIs when merged with structured data for clinical prediction models, which may improve the detection of postoperative AKI.

Creation of a data commons for substance misuse related health research through privacy-preserving patient record linkage between hospitals and state agencies

Objectives

Substance misuse is a complex and heterogeneous set of conditions associated with high mortality and regional/demographic variations. Existing data systems are siloed and have been ineffective in curtailing the substance misuse epidemic. Therefore, we aimed to build a novel informatics platform, the Substance Misuse Data Commons (SMDC), by integrating multiple data modalities to provide a unified record of information crucial to improving outcomes in substance misuse patients.

Materials and Methods

The SMDC was created by linking electronic health record (EHR) data from adult cases of substance (alcohol, opioid, nonopioid drug) misuse at the University of Wisconsin hospitals to socioeconomic and state agency data. To ensure private and secure data exchange, Privacy-Preserving Record Linkage (PPRL) and Honest Broker services were utilized. The overlap in mortality reporting among the EHR, state Vital Statistics, and a commercial national data source was assessed.

Results

The SMDC included data from 36 522 patients experiencing 62 594 healthcare encounters. Over half of patients were linked to the statewide ambulance database and prescription drug monitoring program. Chronic diseases accounted for most underlying causes of death, while drug-related overdoses constituted 8%. Our analysis of mortality revealed a 49.1% overlap across the 3 data sources. Nonoverlapping deaths were associated with poor socioeconomic indicators.

Discussion

Through PPRL, the SMDC enabled the longitudinal integration of multimodal data. Combining death data from local, state, and national sources enhanced mortality tracking and exposed disparities.

Conclusion

The SMDC provides a comprehensive resource for clinical providers and policymakers to inform interventions targeting substance misuse-related hospitalizations, overdoses, and death.

Hypotensive Events in Pediatric Patients Receiving Dexmedetomidine for MRI

PURPOSE

Dexmedetomidine, the preferred pediatric sedating agent for magnetic resonance imaging (MRI), has the side effect of hypotension. Newer recommendations for reporting adverse events in pediatric procedural sedation include using a two-pronged definition. Our aim was to describe the incidence of hypotension in patients undergoing sedated MRI and to identify demographic and clinical factors associated with hypotension, applying a two-pronged definition, where a numerical threshold/clinical criterion must be met as well as at least one clinical intervention performed.

DESIGN

An observational cohort study.

METHODS

Medical record data were extracted for outpatients less than 18 years of age sedated primarily with dexmedetomidine for MRI in a single center for over a seven-year period. Patients who received propofol as an adjunct were also included. Hypotension was defined using a two-pronged approach, as a 20% reduction in systolic blood pressure from baseline lasting ≥10 minutes, coupled with a fluid bolus. Analysis included descriptive statistics, t tests and logistic regression using discrete-time survival analysis.

FINDINGS

Of the 1,590 patient encounters, 90 (5.7%) experienced hypotension. Males were significantly more likely to have hypotension. Patients with hypotension had overall longer appointment times, including longer sedation times and recovery time. Greater blood pressure (BP) variability in the preceding 20 minutes also increased the risk of hypotension.

CONCLUSIONS

Our lower incidence of hypotension is likely related to the two-pronged intervention-based definition used, as it likely more accurately reflects clinically meaningful hypotension. To our knowledge, this is the first study using this approach with this population. Research further examining the relationship between prolonged sedation, blood pressure variability, gender, hypotension, and recovery time is needed. Understanding these relationships will help interdisciplinary teams, including nurses in pediatric procedural areas, to reduce the incidence of hypotension, potentially maximize patient safety, and optimize throughput.

Automated Matching of Patients to Clinical Trials: A Patient-Centric Natural Language Processing Approach for Pediatric Leukemia

PURPOSE

Matching patients to clinical trials is cumbersome and costly. Attempts have been made to automate the matching process; however, most have used a trial-centric approach, which focuses on a single trial. In this study, we developed a patient-centric matching tool that matches patient-specific demographic and clinical information with free-text clinical trial inclusion and exclusion criteria extracted using natural language processing to return a list of relevant clinical trials ordered by the patient's likelihood of eligibility.

MATERIALS AND METHODS

Records from pediatric leukemia clinical trials were downloaded from ClinicalTrials.gov. Regular expressions were used to discretize and extract individual trial criteria. A multilabel support vector machine (SVM) was trained to classify sentence embeddings of criteria into relevant clinical categories. Labeled criteria were parsed using regular expressions to extract numbers, comparators, and relationships. In the validation phase, a patient-trial match score was generated for each trial and returned in the form of a ranked list for each patient.

RESULTS

In total, 5,251 discretized criteria were extracted from 216 protocols. The most frequent criterion was previous chemotherapy/biologics (17%). The multilabel SVM demonstrated a pooled accuracy of 75%. The text processing pipeline was able to automatically extract 68% of eligibility criteria rules, as compared with 80% in a manual version of the tool. Automated matching was accomplished in approximately 4 seconds, as compared with several hours using manual derivation.

CONCLUSION

To our knowledge, this project represents the first open-source attempt to generate a patient-centric clinical trial matching tool. The tool demonstrated acceptable performance when compared with a manual version, and it has potential to save time and money when matching patients to trials.

Early Diagnosis of Primary Immunodeficiency Disease Using Clinical Data and Machine Learning

BACKGROUND

Primary immunodeficiency diseases (PIDD) are a group of immune-related disorders that have a current median delay of diagnosis between 6 and 9 years. Early diagnosis and treatment of PIDD has been associated with improved patient outcomes.

OBJECTIVE

To develop a machine learning model using elements within the electronic health record data that are related to prior symptomatic treatment to predict PIDD.

METHODS

We conducted a retrospective study of patients with PIDD identified using inclusion criteria of PIDD-related diagnoses, immunodeficiency-specific medications, and low immunoglobulin levels. We constructed a control group of age-, sex-, and race-matched patients with asthma. The primary outcome was the diagnosis of PIDD. We considered comorbidities, laboratory tests, medications, and radiological orders as features, all before diagnosis and indicative of symptom-related treatment. Features were presented sequentially to logistic regression, elastic net, and random forest classifiers, which were trained using a nested cross-validation approach.

RESULTS

Our cohort consisted of 6422 patients, of whom 247 (4%) were diagnosed with PIDD. Our logistic regression model with comorbidities demonstrated good discrimination between patients with PIDD and those with asthma (c-statistic: 0.62 [0.58-0.65]). Adding laboratory results, medications, and radiological orders improved discrimination (c-statistic: 0.70 vs 0.62, P < .001), sensitivity, and specificity. Extending to the advanced machine learning models did not improve performance.

CONCLUSIONS

We developed a prediction model for early diagnosis of PIDD using historical data that are related to symptomatic care, which has potential to fill an important need in reducing the time to diagnose PIDD, leading to better outcomes for immunodeficient patients.

Prediction of Intracranial Hypertension and Brain Tissue Hypoxia Utilizing High-Resolution Data from the BOOST-II Clinical Trial

The current approach to intracranial hypertension and brain tissue hypoxia is reactive, based on fixed thresholds. We used statistical machine learning on high-frequency intracranial pressure (ICP) and partial brain tissue oxygen tension (PbtO₂) data obtained from the BOOST-II trial with the goal of constructing robust quantitative models to predict ICP/PbtO₂ crises. We derived the following machine learning models: logistic regression (LR), elastic net, and random forest. We split the data set into 70–30% for training and testing and utilized a discrete-time survival analysis framework and 5-fold hyperparameter optimization strategy for all models. We compared model performances on discrimination between events and non-events of increased ICP or low PbtO₂ with the area under the receiver operating characteristic (AUROC) curve. We further analyzed clinical utility through a decision curve analysis (DCA). When considering discrimination, the number of features, and interpretability, we identified the RF model that combined the most recent ICP reading, episode number, and longitudinal trends over the preceding 30 min as the best performing for predicting ICP crisis events within the next 30 min (AUC 0.78). For PbtO₂, the LR model utilizing the most recent reading, episode number, and longitudinal trends over the preceding 30 min was the best performing (AUC, 0.84). The DCA showed clinical usefulness for wide risk of thresholds for both ICP and PbtO₂ predictions. Acceptable alerting thresholds could range from 20% to 80% depending on a patient-specific assessment of the benefit-risk ratio of a given intervention in response to the alert.

Multicenter Development and Validation of a Model for Predicting Retention in Care Among People with HIV

Predictive analytics can be used to identify people with HIV currently retained in care who are at risk for future disengagement from care, allowing for prioritization of retention interventions. We utilized machine learning methods to develop predictive models of retention in care, defined as no more than a 12 month gap between HIV care appointments in the Center for AIDS Research Network of Integrated Clinical Systems (CNICS) cohort. Data were split longitudinally into derivation and validation cohorts. We created logistic regression (LR), random forest (RF), and gradient boosted machine (XGB) models within a discrete-time survival analysis framework and compared their performance to a baseline model that included only demographics, viral suppression, and retention history. 21,267 Patients with 507,687 visits from 2007 to 2018 were included. The LR model outperformed the baseline model (AUC 0.68 [0.67-0.70] vs. 0.60 [0.59-0.62], P < 0.001). RF and XGB models had similar performance to the LR model. Top features in the LR model included retention history, age, and viral suppression.

Identifying infected patients using semi-supervised and transfer learning

Objectives

Early identification of infection improves outcomes, but developing models for early identification requires determining infection status with manual chart review, limiting sample size. Therefore, we aimed to compare semi-supervised and transfer learning algorithms with algorithms based solely on manual chart review for identifying infection in hospitalized patients.

Materials and Methods

This multicenter retrospective study of admissions to 6 hospitals included “gold-standard” labels of infection from manual chart review and “silver-standard” labels from nonchart-reviewed patients using the Sepsis-3 infection criteria based on antibiotic and culture orders. “Gold-standard” labeled admissions were randomly allocated to training (70%) and testing (30%) datasets. Using patient characteristics, vital signs, and laboratory data from the first 24 hours of admission, we derived deep learning and non-deep learning models using transfer learning and semi-supervised methods. Performance was compared in the gold-standard test set using discrimination and calibration metrics.

Results

The study comprised 432 965 admissions, of which 2724 underwent chart review. In the test set, deep learning and non-deep learning approaches had similar discrimination (area under the receiver operating characteristic curve of 0.82). Semi-supervised and transfer learning approaches did not improve discrimination over models fit using only silver- or gold-standard data. Transfer learning had the best calibration (unreliability index P value: .997, Brier score: 0.173), followed by self-learning gradient boosted machine (P value: .67, Brier score: 0.170).

Discussion

Deep learning and non-deep learning models performed similarly for identifying infection, as did models developed using Sepsis-3 and manual chart review labels.

Conclusion

In a multicenter study of almost 3000 chart-reviewed patients, semi-supervised and transfer learning models showed similar performance for model discrimination as baseline XGBoost, while transfer learning improved calibration.

Predicting neurological outcomes after in-hospital cardiac arrests for patients with Coronavirus Disease 2019

Background

Machine learning models are more accurate than standard tools for predicting neurological outcomes in patients resuscitated after cardiac arrest. However, their accuracy in patients with Coronavirus Disease 2019 (COVID-19) is unknown. Therefore, we compared their performance in a cohort of cardiac arrest patients with COVID-19.

Methods

We conducted a retrospective analysis of resuscitation survivors in the Get With The Guidelines®-Resuscitation (GWTG-R) COVID-19 registry between February 2020 and May 2021. The primary outcome was a favorable neurological outcome, indicated by a discharge Cerebral Performance Category score ≤ 2. Pre- and peri-arrest variables were used as predictors. We applied our published logistic regression, neural network, and gradient boosted machine models developed in patients without COVID-19 to the COVID-19 cohort. We also updated the neural network model using transfer learning. Performance was compared between models and the Cardiac Arrest Survival Post-Resuscitation In-Hospital (CASPRI) score.

Results

Among the 4,125 patients with COVID-19 included in the analysis, 484 (12 %) patients survived with favorable neurological outcomes. The gradient boosted machine, trained on non-COVID-19 patients was the best performing model for predicting neurological outcomes in COVID-19 patients, significantly better than the CASPRI score (c-statistic: 0.75 vs 0.67, P < 0.001). While calibration improved for the neural network with transfer learning, it did not surpass the gradient boosted machine in terms of discrimination.

Conclusion

Our gradient boosted machine model developed in non-COVID patients had high discrimination and adequate calibration in COVID-19 resuscitation survivors and may provide clinicians with important information for these patients.

PICU Survivorship: Factors Affecting Feasibility and Cohort Retention in a Long-Term Outcomes Study

Our understanding of longitudinal outcomes of Pediatric Intensive Care Unit (PICU) survivors is limited by the heterogeneity of follow-up intervals, populations, and outcomes assessed. We sought to demonstrate (1) the feasibility of longitudinal multidimensional outcome assessment and (2) methods to promote cohort retention. The objective of this presented study was to provide details of follow-up methodology in a PICU survivor cohort and not to present the outcomes at long-term follow-up for this cohort. We enrolled 152 children aged 0 to 17 years admitted to the PICU in a prospective longitudinal cohort study. We examined resource utilization, family impact of critical illness, and neurodevelopment using the PICU Outcomes Portfolio (POP) Survey which included a study-specific survey and validated tools: 1. Functional Status Scale, 2. Pediatric Evaluation of Disability Inventory Computer Adaptive Test, 3. Pediatric Quality of Life Inventory, 4. Strengths and Difficulties Questionnaire, and 5. Vanderbilt Assessment Scales for Attention Deficit-Hyperactivity Disorder. POP Survey completion rates were 89%, 78%, and 84% at 1, 3, and 6 months. Follow-up rates at 1, 2, and 3 years were 80%, 55%, and 43%. Implementing a longitudinal multidimensional outcome portfolio for PICU survivors is feasible within an urban, tertiary-care, academic hospital. Our attrition after one year demonstrates the long-term follow-up challenges in this population. Our findings inform ongoing efforts to implement core outcome sets after pediatric critical illness.

Development and External Validation of a Machine Learning Model for Prediction of Potential Transfer to the PICU

OBJECTIVES

Unrecognized clinical deterioration during illness requiring hospitalization is associated with high risk of mortality and long-term morbidity among children. Our objective was to develop and externally validate machine learning algorithms using electronic health records for identifying ICU transfer within 12 hours indicative of a child's condition.

DESIGN

Observational cohort study.

SETTING

Two urban, tertiary-care, academic hospitals (sites 1 and 2).

PATIENTS

Pediatric inpatients (age <18 yr).

INTERVENTIONS

None.

MEASUREMENT AND MAIN RESULTS

Our primary outcome was direct ward to ICU transfer. Using age, vital signs, and laboratory results, we derived logistic regression with regularization, restricted cubic spline regression, random forest, and gradient boosted machine learning models. Among 50,830 admissions at site 1 and 88,970 admissions at site 2, 1,993 (3.92%) and 2,317 (2.60%) experienced the primary outcome, respectively. Site 1 data were split longitudinally into derivation (2009-2017) and validation (2018-2019), whereas site 2 constituted the external test cohort. Across both sites, the gradient boosted machine was the most accurate model and outperformed a modified version of the Bedside Pediatric Early Warning Score that only used physiologic variables in terms of discrimination ( C -statistic site 1: 0.84 vs 0.71, p < 0.001; site 2: 0.80 vs 0.74, p < 0.001), sensitivity, specificity, and number needed to alert.

CONCLUSIONS

We developed and externally validated a novel machine learning model that identifies ICU transfers in hospitalized children more accurately than current tools. Our model enables early detection of children at risk for deterioration, thereby creating opportunities for intervention and improvement in outcomes.

A prediction model for bacteremia and transfer to intensive care in pediatric and adolescent cancer patients with febrile neutropenia

Febrile neutropenia (FN) is a common condition in children receiving chemotherapy. Our goal in this study was to develop a model for predicting blood stream infection (BSI) and transfer to intensive care (TIC) at time of presentation in pediatric cancer patients with FN. We conducted an observational cohort analysis of pediatric and adolescent cancer patients younger than 24 years admitted for fever and chemotherapy-induced neutropenia over a 7-year period. We excluded stem cell transplant recipients who developed FN after transplant and febrile non-neutropenic episodes. The primary outcome was onset of BSI, as determined by positive blood culture within 7 days of onset of FN. The secondary outcome was transfer to intensive care (TIC) within 14 days of FN onset. Predictor variables include demographics, clinical, and laboratory measures on initial presentation for FN. Data were divided into independent derivation (2009-2014) and prospective validation (2015-2016) cohorts. Prediction models were built for both outcomes using logistic regression and random forest and compared with Hakim model. Performance was assessed using area under the receiver operating characteristic curve (AUC) metrics. A total of 505 FN episodes (FNEs) were identified in 230 patients. BSI was diagnosed in 106 (21%) and TIC occurred in 56 (10.6%) episodes. The most common oncologic diagnosis with FN was acute lymphoblastic leukemia (ALL), and the highest rate of BSI was in patients with AML. Patients who had BSI had higher maximum temperature, higher rates of prior BSI and higher incidence of hypotension at time of presentation compared with patients who did not have BSI. FN patients who were transferred to the intensive care (TIC) had higher temperature and higher incidence of hypotension at presentation compared to FN patients who didn't have TIC. We compared 3 models: (1) random forest (2) logistic regression and (3) Hakim model. The areas under the curve for BSI prediction were (0.79, 0.65, and 0.64, P < 0.05) for models 1, 2, and 3, respectively. And for TIC prediction were (0.88, 0.76, and 0.65, P < 0.05) respectively. The random forest model demonstrated higher accuracy in predicting BSI and TIC and showed a negative predictive value (NPV) of 0.91 and 0.97 for BSI and TIC respectively at the best cutoff point as determined by Youden's Index. Likelihood ratios (LRs) (post-test probability) for RF model have potential utility of identifying low risk for BSI and TIC (0.24 and 0.12) and high-risk patients (3.5 and 6.8) respectively. Our prediction model has a very good diagnostic performance in clinical practices for both BSI and TIC in FN patients at the time of presentation. The model can be used to identify a group of individuals at low risk for BSI who may benefit from early discharge and reduced length of stay, also it can identify FN patients at high risk of complications who might benefit from more intensive therapies at presentation.

Abstract TMP38: Using Natural Language Processing To Investigate Diagnostic Error In Acute Stroke

Introduction: Diagnostic error occurs in approximately 10% of acute stroke (AS) presentations. The diagnostic process includes history, physical examination, and test performance and interpretation. However, critical information for diagnosis is contained in unstructured clinical notes.

Hypothesis: We hypothesized that natural language processing (NLP) can identify features in unstructured clinical notes associated with potential diagnostic error during ED “catch and release” (CR) encounters prior to AS admissions.

Methods: Using a retrospective case-control design and ICD-10 codes, we identified index emergency department (ED) admissions with a diagnosis of first-time stroke (cases) and age and sex-matched gastroenteritis (controls) who had an ED CR encounter in prior 30 days. Notes were processed using cTAKES to identify concept unique identifiers (CUI) among clinical narratives from the CR encounters. Regression analysis was utilized to determine CUI terms from the CR encounter that were associated with stroke cases compared to controls. These CUI terms were grouped by clinical experts into 3 aspects of the diagnostic process: history (e.g., risk factors, medications, symptoms), neurologic examination (e.g., mental status exam, cranial nerves, pronator drift), and tests (e.g., labs, CT, MRI).

Results: In an analytic cohort of 319 stroke cases and 319 gastroenteritis controls, a non-cerebrovascular neurologic diagnosis at the CR encounter was noted in 20.2% of cases versus 6.0% in controls (P<0.01). We identified 120 terms at the CR encounter associated with stroke (OR >2.0 and p<0.05). Grouped by themes, tests accounted for 50 (41.7%), examination for 37 (30.1%), and history for 33 (27.5%) terms. Terms related to neurologic examination had the highest median OR (median OR 6.7, IQR 2.7-11.5) followed by history (median OR 3.8, IQR 3.2-4.9) and tests (median OR 3.5, IQR 2.8-4.6).

Conclusions: Neurologic presentations to the ED preceded 20% of stroke cases suggesting some of these may represent missed diagnoses for minor stroke and TIA. NLP may be a useful surveillance approach to identify neurologic symptoms, deficits, and tests present at CR encounters and trigger interventions to reduce diagnostic error prior to stroke.

Multiple Organ Dysfunction Interactions in Critically Ill Children

INTRODUCTION

Multiple organ dysfunction (MOD) is a common pathway to morbidity and death in critically ill children. Defining organ dysfunction is challenging, as we lack a complete understanding of the complex pathobiology. Current pediatric organ dysfunction criteria assign the same diagnostic value-the same "weight"- to each organ system. While each organ dysfunction in isolation contributes to the outcome, there are likely complex interactions between multiple failing organs that are not simply additive.

OBJECTIVE

Determine whether certain combinations of organ system dysfunctions have a significant interaction associated with higher risk of morbidity or mortality in critically ill children.

METHODS

We conducted a retrospective observational cohort study of critically ill children at two large academic medical centers from 2010 and 2018. Patients were included in the study if they had at least two organ dysfunctions by day 3 of PICU admission based on the Pediatric Organ Dysfunction Information Update Mandate (PODIUM) criteria. Mortality was described as absolute number of deaths and mortality rate. Combinations of two pediatric organ dysfunctions were analyzed with interaction terms as independent variables and mortality or persistent MOD as the dependent variable in logistic regression models.

RESULTS

Overall, 7,897 patients met inclusion criteria and 446 patients (5.6%) died. The organ dysfunction interactions that were significantly associated with the highest absolute number of deaths were cardiovascular + endocrinologic, cardiovascular + neurologic, and cardiovascular + respiratory. Additionally, the interactions associated with the highest mortality rates were liver + cardiovascular, respiratory + hematologic, and respiratory + renal. Among patients with persistent MOD, the most common organ dysfunctions with significant interaction terms were neurologic + respiratory, hematologic + immunologic, and endocrinologic + respiratory. Further analysis using classification and regression trees (CART) demonstrated that the absence of respiratory and liver dysfunction was associated with the lowest likelihood of mortality.

IMPLICATIONS AND FUTURE DIRECTIONS

Certain combinations of organ dysfunctions are associated with a higher risk of persistent MOD or death. Notably, the three most common organ dysfunction interactions were associated with 75% of the mortality in our cohort. Critically ill children with MOD presenting with these combinations of organ dysfunctions warrant further study.

Applications of Artificial Intelligence in Pediatric Oncology: A Systematic Review

PURPOSE

There is a need for an improved understanding of clinical and biologic risk factors in pediatric cancer to improve patient outcomes. Machine learning (ML) represents the application of computational inference from advanced statistical methods that can be applied to increasing amount of data available for study in pediatric oncology. The goal of this systematic review was to systematically characterize the state of ML in pediatric oncology and highlight advances and opportunities in the field.

METHODS

We conducted a systematic review of the Embase, Scopus, and MEDLINE databases for applications of ML in pediatric oncology. Query results from all three databases were aggregated and duplicate studies were removed.

RESULTS

A total of 42 unique articles that examined the applications of ML in pediatric oncology met inclusion criteria for review. We identified 20 studies of CNS tumors, 13 of solid tumors, and nine of leukemia. ML tasks included classification, prediction of treatment response, and dose optimization with a variety of methods being used including neural network, k-nearest neighbor, random forest, naive Bayes, and support vector machines. Strengths of the identified studies included matching or outperforming physician comparators via automated analysis and predicting therapeutic response. Common limitations included significant heterogeneity in reporting standards, clinical applicability, small sample sizes, and missing external validation cohorts.

CONCLUSION

We identified areas where ML can enhance clinical care in ways that may not otherwise be achievable. Although ML promises enormous potential in improving diagnostics, decision making, and monitoring for children with cancer, the field remains in early stages and future work will be aided by standards and guidelines to ensure rigorous methodologic design and maximizing clinical utility.

Predicting Response to Chemotherapy in Patients With Newly Diagnosed High-Risk Neuroblastoma: A Report From the International Neuroblastoma Risk Group

PURPOSE

Metaiodobenzylguanidine (MIBG) scans are a radionucleotide imaging modality that undergo Curie scoring to semiquantitatively assess neuroblastoma burden, which can be used as a marker of therapy response. We hypothesized that a convolutional neural network (CNN) could be developed that uses diagnostic MIBG scans to predict response to induction chemotherapy.

METHODS

We analyzed MIBG scans housed in the International Neuroblastoma Risk Group Data Commons from patients enrolled in the Children's Oncology Group high-risk neuroblastoma study ANBL12P1. The primary outcome was response to upfront chemotherapy, defined as a Curie score ≤ 2 after four cycles of induction chemotherapy. We derived and validated a CNN using two-dimensional whole-body MIBG scans from diagnosis and evaluated model performance using area under the receiver operating characteristic curve (AUC). We also developed a clinical classification model to predict response on the basis of age, stage, and MYCN amplification.

RESULTS

Among 103 patients with high-risk neuroblastoma included in the final cohort, 67 (65%) were responders. Performance in predicting response to upfront chemotherapy was equivalent using the CNN and the clinical model. Class-activation heatmaps verified that the CNN used areas of disease within the MIBG scans to make predictions. Furthermore, integrating predictions using a geometric mean approach improved detection of responders to upfront chemotherapy (geometric mean AUC 0.73 v CNN AUC 0.63, P < .05; v clinical model AUC 0.65, P < .05).

CONCLUSION

We demonstrate feasibility in using machine learning of diagnostic MIBG scans to predict response to induction chemotherapy for patients with high-risk neuroblastoma. We highlight improvements when clinical risk factors are also integrated, laying the foundation for using a multimodal approach to guiding treatment decisions for patients with high-risk neuroblastoma.

Increase in Epinephrine Administration for Food-Induced Anaphylaxis in Pediatric Emergency Departments from 2007 to 2015

BACKGROUND

Epinephrine is underused in the treatment of anaphylaxis, despite being the first-line treatment, which reflects the challenges in diagnosing anaphylaxis and understanding the appropriate therapy.

OBJECTIVE

To describe trends in epinephrine administration for patients visiting the pediatric emergency department (ED) with food-induced anaphylaxis (FIA) from 2007 to 2015.

METHODS

This retrospective cohort study included children 0 to 17 years of age with FIA from 46 children's hospitals in the United States between 2007 and 2015. Multivariable regression was used to identify factors associated with epinephrine administration.

RESULTS

A total of 15,318 cases of FIA cases were seen in the pediatric EDs from 2007 to 2015. Among these ED visits, 7,600 (49.6%) had at least 1 dose of epinephrine administered in the ED. Administration of epinephrine for anaphylaxis in the pediatric ED increased by 4% each year (odds ratio [OR] 1.04; 95% CI 1.03-1.05; P < .001). Sensitivity analysis by census region demonstrated that hospitals in the Northeast and the West were associated with an increase in epinephrine administration per year (Northeast OR 1.18, 95% CI 1.13-1.22, P < .001; West OR 1.14, 95% CI 1.10-1.18, P < .001).

CONCLUSIONS

Epinephrine administration for FIA in the pediatric ED has increased over time, reflecting the need for continued advocacy for the optimal management of FIA. Further research is warranted to identify optimal strategies for proper recognition and early administration of epinephrine for anaphylaxis.

Multicenter Validation of the Neonatal Sequential Organ Failure Assessment Score for Prognosis in the Neonatal Intensive Care Unit

Infants in the neonatal intensive care unit are at risk of life-threatening organ dysfunction, but few objective tools with utility exist. In a multicenter cohort of 20 152 infants, we show the neonatal sequential organ failure assessment score had good-to-excellent discrimination of mortality across centers, birth weights, and time points after admission.

Use of Statins Among Patients Taking Levothyroxine: an Observational Drug Utilization Study Across Sites

CONTEXT

Treatment with levothyroxine (LT4) that normalize serum thyrotropin (TSH) is expected to restore lipid metabolism.

OBJECTIVE

To assess statin utilization in LT4-treated patients through an observational drug utilization study.

METHODS

Three sites were involved: (1) 10 723 outpatients placed on LT4 during 2006-2019 identified from the Clinical Research Data Warehouse of the University of Chicago; (2) ~1.4 million LT4 prescriptions prepared by primary care physicians during January-December 2018, identified from the IQVIA™ database of medical prescriptions in Brazil; (30 ~5.4 million patient interviews during 2009-2019, including ~0.32 million patients on LT4, identified from the Fleury Group database in Brazil.

RESULTS

On site 1, initiation of therapy with LT4 increased the frequency of statin utilization (19.1% vs 24.6%), which occurred ~1.5 years later (median 76 weeks) and, among those patients that were on statins, increased intensity of treatment by 33%, despite normalization of serum TSH levels; on site 2, after matching for sex and age, the frequency of statins prescription was higher for those patients using LT4: females, 2.1 vs 3.4% (odds ratio [OR] 1.656 [1.639-1.673]); males, 3.1 vs 4.4% (OR 1.435 [1.409-1.462]); and, on site 3, after matching for sex and age, the frequency of statin utilization was higher in those patients using LT4: females, 10 vs 18% (OR 2.02 [2.00-2.04]); males, 15 vs 25% (OR 1.92 [1.88-1.96]); all P values were <.0001.

CONCLUSION

Prescription and utilization of statins were higher in patients taking LT4. The reasons for this association should be addressed in future studies.

Improving Prehospital Stroke Diagnosis Using Natural Language Processing of Paramedic Reports

[Figure: see text].

Predicting response to chemotherapy in neuroblastoma using deep learning: A report from the International Neuroblastoma Risk Group

10039

Background: Metaiodobenzylguanidine (MIBG) scans are a radionucleotide imaging modality used to evaluate neuroblastoma stage at diagnosis and also determine disease response following therapy. Curie scoring is used to semi-quantitatively assess disease burden from an MIBG scan on a scale from none (0) to widespread throughout the body (30). While a Curie score ≤2 after six cycles of induction chemotherapy has been shown to be prognostic of outcome, there is no established correlation between diagnostic Curie score and outcome. Deep learning models, such as convolutional neural networks (CNN), have been shown to learn generalizable patterns within images for successful classification of metastases and detection of multiple adult cancers. We hypothesized a CNN could be developed to predict response to induction chemotherapy, a proxy for outcome, using diagnostic MIBG scans. Methods: DICOM MIBG scans and associated clinical data from a Children’s Oncology Group (COG) pilot study for children diagnosed with high-risk neuroblastoma (ANBL12P1; NCT1798004) were deidentified and linked to clinical data by the Pediatric Cancer Data Commons and obtained from the International Neuroblastoma Risk Group Data Commons. Patients were defined as having a poor response to induction chemotherapy if their Curie score after four cycles of induction chemotherapy was ≥2. An independent external validation cohort was comprised of 29 images from 26 high-risk patients treated at the University of Chicago with clinically-annotated diagnostic and post-cycle six induction DICOM MIBG scans. The CNN was trained using 2D whole body MIBG scans obtained at diagnosis. We developed the CNN using a transfer learning approach using the Xception architecture as the base layer. Hyperparameter optimization was performed using an 80%-20% train-validation strategy. Model performance was evaluated using area under the receiver operating characteristic curve (AUROC). Results: Among 146 patients with high-risk neuroblastoma enrolled on ANBL12P1, 104 had available diagnostic and end-induction MIBG scans. There were no differences in clinical or biological characteristics between included and excluded patients. The base model CNN was able to predict which patients had a poor response to induction chemotherapy with an AUROC of 0.72 in the validation set from the ANBL12P1 cohort. Additionally, the CNN was able to predict patient response to therapy with an AUROC of 0.64 in an independent external dataset from University of Chicago. Conclusions: Our study suggests it is feasible to apply machine learning of diagnostic MIBG scans to predict response to chemotherapy for high-risk neuroblastoma patients. Given these promising results, further work to improve AUROC and performance within larger datasets is ongoing.

Abstract P270: Simulating the Effects of Door-In-Door-Out Interventions

Introduction: Acute stroke (AS) is a highly time sensitive treatment condition affecting approximately 800,000 people/year in the US. Most AS patients receive care at a primary stroke center (PSC), but some require more advanced treatments, and rely on a timely transfer to a comprehensive stroke center (CSC) where such treatments can be given. Stroke teams at 2 Chicago area PSCs and 4 CSCs, collectively, developed solutions (Graph) targeting both reported and perceived failures/delays/weakness in the current PSC door-in-door-out (DIDO) process for transferring patients to a CSC. The study simulates the potential impact of the solutions on DIDO.

Methods: Current state (baseline) times were calculated from time stamps in the electronic health record (EHR) (e.g., door to CT), estimated by the stroke teams (e.g., hand-off time) or retrieved (e.g., DIDO, door to stroke activation) from a prospectively maintained REDCap data registry (2/2018-1/2020). Proportions (e.g., % with ischemic stroke, % transferred) were estimated from hospital data. Changes in times after implementation of a solution were obtained from peer reviewed literature, when available, or by consensus expert opinion. Simio (version 11.197.19514) was used to simulate the current and future states with implementation of the solutions, with 500 replications, to estimate changes in DIDO.

Results: Implementation of all solutions would achieve a decrease in DIDO of 33 minutes (19%) from current state. The largest driver of this change was direct to CT/CTA protocol implementation (21 minutes) followed by using a handoff tool for paramedics prior to transfer (13 minutes).

Conclusion: The proposed solutions can achieve nearly a 20% reduction in DIDO times. The “Direct to CT/CTA Protocol” solution is the major driver of the improvement. Data simulation is helpful by assessing the potential impact of many solutions and the relative impact of each solution to inform implementation decisions.

Characteristics of Critically Ill Adults With Sacrococcygeal Unavoidable Hospital-Acquired Pressure Injuries: A Retrospective, Matched, Case-Control Study

PURPOSE

To identify characteristics of critically ill adults with sacrococcygeal, unavoidable hospital-acquired pressure injuries (uHAPIs).

DESIGN

Retrospective, matched, case-control design.

SUBJECTS/SETTING

Patients admitted to adult intensive care units (ICUs) at an urban academic medical center from January 2014 through July 2016.

METHODS

Thirty-four patients without uHAPI were matched to 34 patients with sacrococcygeal uHAPI. Time points of interest included admission to the ICU, the week preceding the definitive assessment date, and hospital discharge status. Variables of interest included length of stay, any diagnosis of sepsis, severity of illness, degree of organ dysfunction/failure, supportive therapies in use (eg, mechanical ventilation), and pressure injury risk (Braden Scale score).

RESULTS

All 34 sacrococcygeal pressure injuries were classified as uHAPI using the pressure injury prevention inventory instrument. No statistically significant differences were noted between patients for severity of illness, degree of organ dysfunction/failure, or pressure injury risk at ICU admission. At 1 day prior to the definitive assessment date and at discharge, patients with uHAPI had significantly higher mean Sequential Organ Failure Assessment (SOFA) scores (greater organ dysfunction/failure) and lower mean Braden Scale scores (greater pressure injury risk) than patients without uHAPI. Patients with uHAPI had significantly longer lengths of stay, more supportive therapies in use, were more often diagnosed with sepsis, and were more likely to die during hospitalization.

CONCLUSION

Sacrococcygeal uHAPI development was associated with progressive multiorgan dysfunction/failure, greater use of supportive therapies, sepsis diagnosis, and mortality. Additional research investigating the role of multiorgan dysfunction/failure and sepsis on uHAPI development is warranted.

A Vital Sign-Based Model to Predict Clinical Deterioration in Hospitalized Children

OBJECTIVES

Clinical deterioration in hospitalized children is associated with increased risk of mortality and morbidity. A prediction model capable of accurate and early identification of pediatric patients at risk of deterioration can facilitate timely assessment and intervention, potentially improving survival and long-term outcomes. The objective of this study was to develop a model utilizing vital signs from electronic health record data for predicting clinical deterioration in pediatric ward patients.

DESIGN

Observational cohort study.

SETTING

An urban, tertiary-care medical center.

PATIENTS

Patients less than 18 years admitted to the general ward during years 2009-2018.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

The primary outcome of clinical deterioration was defined as a direct ward-to-ICU transfer. A discrete-time logistic regression model utilizing six vital signs along with patient characteristics was developed to predict ICU transfers several hours in advance. Among 31,899 pediatric admissions, 1,375 (3.7%) experienced the outcome. Data were split into independent derivation (yr 2009-2014) and prospective validation (yr 2015-2018) cohorts. In the prospective validation cohort, the vital sign model significantly outperformed a modified version of the Bedside Pediatric Early Warning System score in predicting ICU transfers 12 hours prior to the event (C-statistic 0.78 vs 0.72; p < 0.01).

CONCLUSIONS

We developed a model utilizing six commonly used vital signs to predict risk of deterioration in hospitalized children. Our model demonstrated greater accuracy in predicting ICU transfers than the modified Bedside Pediatric Early Warning System. Our model may promote opportunities for timelier intervention and risk mitigation, thereby decreasing preventable death and improving long-term health.

Predictors of pressure injury development in critically ill adults: A retrospective cohort study

OBJECTIVE

The purpose of this research was to identify predictors of pressure injury, using data from the electronic health records of critically ill adults.

METHODOLOGY

A retrospective cohort study was conducted using logistic regression models to examine risk factors adjusted for age, gender, race/ethnicity and length of stay.

SETTING

The study cohort included 1587 adults in intensive care units within an urban academic medical centre.

MAIN OUTCOME MEASURES

The presence or absence of a hospital-acquired pressure injury was determined during monthly skin integrity prevalence surveys. All pressure injuries were independently confirmed by two Certified Wound Care Nurses.

RESULTS

Eighty-one (5.1%) of the 1587 cohort patients developed pressure injuries. After adjusting for confounders, the clinical variables associated with pressure injury development included mean arterial pressure <60 mmHg and lowest Total Braden score up to two weeks prior to the date of HAPI development or date of prevalence survey for the comparison group.

CONCLUSIONS

This study provides a more comprehensive understanding about pressure injury risk in critically ill adults, identifying extrinsic and intrinsic factors associated with pressure injury development. Prospective multisite studies are needed to further examine these potential contributors to pressure injury development within the context of adherence to prevention interventions.

Abnormal vital signs after laparoscopic colorectal surgery: More common than you think

BACKGROUND

Anastomotic leak is a feared complication. The presence of abnormal vital signs is often cited as an important overlooked predictive clue in retrospective settings once the diagnosis of leak has already been established. We aimed to determine the prevalence of abnormal vital signs following colorectal resection and assess its predictive value.

METHODS

We retrospectively studied patients undergoing colorectal resection. The performance of vital signs in predicting anastomotic leak was assessed using discrete-time survival analysis and receiver operator characteristic curve.

RESULTS

1662 patients (841 laparoscopic, 821 open) were included. Clinical anastomotic leak was diagnosed in 50 patients (3.1%). 96.8% of patients of the entire cohort had at least one abnormal vital sign during their postoperative course. No individual vital sign was a strong predictor of anastomotic leak in either laparoscopic or open cohorts.

CONCLUSION

Vital sign abnormalities are extremely common following open and laparoscopic colorectal surgery and alone are poor predictors of anastomotic leak.

Combining patient visual timelines with deep learning to predict mortality

BACKGROUND

Deep learning algorithms have achieved human-equivalent performance in image recognition. However, the majority of clinical data within electronic health records is inherently in a non-image format. Therefore, creating visual representations of clinical data could facilitate using cutting-edge deep learning models for predicting outcomes such as in-hospital mortality, while enabling clinician interpretability. The objective of this study was to develop a framework that first transforms longitudinal patient data into visual timelines and then utilizes deep learning to predict in-hospital mortality.

METHODS AND FINDINGS

All adult consecutive patient admissions from 2008-2016 at a tertiary care center were included in this retrospective study. Two-dimensional visual representations for each patient were created with clinical variables on one dimension and time on the other. Predictors included vital signs, laboratory results, medications, interventions, nurse examinations, and diagnostic tests collected over the first 48 hours of the hospital stay. These visual timelines were utilized by a convolutional neural network with a recurrent layer model to predict in-hospital mortality. Seventy percent of the cohort was used for model derivation and 30% for independent validation. Of 115,825 hospital admissions, 2,926 (2.5%) suffered in-hospital mortality. Our model predicted in-hospital mortality significantly better than the Modified Early Warning Score (area under the receiver operating characteristic curve [AUC]: 0.91 vs. 0.76, P < 0.001) and the Sequential Organ Failure Assessment score (AUC: 0.91 vs. 0.57, P < 0.001) in the independent validation set. Class-activation heatmaps were utilized to highlight areas of the picture that were most important for making the prediction, thereby providing clinicians with insight into each individual patient's prediction.

CONCLUSIONS

We converted longitudinal patient data into visual timelines and applied a deep neural network for predicting in-hospital mortality more accurately than current standard clinical models, while allowing for interpretation. Our framework holds promise for predicting several important outcomes in clinical medicine.

SAT-266 A Retrospective Study Investigating the Association Between ADHD and Glycemic Control in Children and Adolescents with Type 1 Diabetes Mellitus

Background: Attention deficit hyperactivity disorder (ADHD) is a chronic neuropsychiatric disorder of childhood that often persists into adulthood. Common symptoms of ADHD include difficulty in sustaining attention, failure to give attention to detail, forgetfulness and avoiding tasks requiring mental effort, as well as lower executive functioning. Management of diabetes involves complex multistep tasks, requiring good cognitive and executive functioning. We hypothesized that patients with Type 1 diabetes mellitus (T1DM) who have a concurrent ADHD diagnosis, may encounter difficulty in managing their diabetes on a daily basis thereby leading to poor glycemic control. Objective: We compared average HbA1C values and number of diabetic ketoacidosis (DKA) episodes in patients between ages 6-25 years with T1DM with ADHD (case cohort) and T1DM without ADHD (control group). Methods: The study was designed as a retrospective review of data obtained from the electronic health record in our medical system. Cohort identification was performed through a combination of ICD 9/10 codes, problem list search and medication use. Results were grouped by age. Results: In the age range between 6-18 years, there were 334 controls and 25 cases. Between ages 18-25 years, there were 443 controls and 30 cases. There was an increased number of boys with ADHD in the age group 6-18 years (p=0.0035). There was no significant difference between the average HbA1C over a 2 year period between the two groups in either of the age ranges (p=0.6 and 0.8 respectively). There was no difference in the number of DKA episodes in either group. We did find a significant difference in the type of insurance coverage between the groups. The population with T1DM and ADHD had higher rates of commercial versus public insurance as compared to the control group with T1DM and no diagnosed ADHD in both age groups (p=0.004). Conclusions: The main limitation of the study is its retrospective nature and the possibility of undiagnosed ADHD symptoms in the control group. Diagnosis of ADHD in the Medicaid population with T1DM may be lower due to infrequent clinic visits, long waiting time for specialists or lack of resources and knowledge. Future studies could clarify whether screening for symptoms of inattention and school/work performance in this particular population may identify patients with ADHD who could benefit from therapeutic or medical intervention for the ADHD that could improve T1DM outcome. Funding: Pediatrics Research Grant, Center for Research Informatics.

Association Between Room Location and Adverse Outcomes in Hospitalized Patients

OBJECTIVE

To investigate whether a patient's proximity to the nurse's station or ward entrance at time of admission was associated with increased risk of adverse outcomes.

METHOD

We conducted a retrospective cohort study of consecutive adult inpatients to 13 medical-surgical wards at an academic hospital from 2009 to 2013. Proximity of admission room to the nurse's station and to the ward entrance was measured using Euclidean distances. Outcomes of interest include development of critical illness (defined as cardiac arrests or transfer to an intensive care unit), inhospital mortality, and increase in length of stay (LOS).

RESULTS

Of the 83,635 admissions, 4,129 developed critical illness and 1,316 died. The median LOS was 3 days. After adjusting for admission severity of illness, ward, shift, and year, we found no relationship between proximity at admission to nurse's station our outcomes. However, patients admitted to end of the ward had higher risk of developing critical illness (odds ratio [ OR] = 1.15, 95% confidence interval [CI] = [1.08, 1.23]), mortality ( OR = 1.16, 95% CI [1.03, 1.33]), and a higher LOS (13-hr increase, 95% CI [10, 15] hours) compared to patients admitted closer to the ward entrance. Similar results were observed in sensitivity analyses adjusting for isolation room patients and considering patients without room transfers in the first 48 hr.

CONCLUSIONS

Our study suggests that being away from the nurse's station did not increase the risk of these adverse events in ward patients, but being farther from the ward entrance was associated with increase in risk of adverse outcomes. Patient safety can be improved by recognizing this additional risk factor.

Using photoplethysmography data to estimate heart rate variability and its association with organ dysfunction in pediatric oncology patients

Pediatric oncology patients are at high risk of developing clinical deterioration and organ dysfunction during their illness. Heart rate variability (HRV) measured using electrocardiography waveforms is associated with increased organ dysfunction and clinical deterioration in adult and pediatric patients in the intensive care unit (ICU). Here, we explore the feasibility of using photoplethysmography (PPG)-derived integer pulse rate variability (PRVi) to estimate HRV and determine its association with organ dysfunction in pediatric oncology patients in the ward and pediatric ICU. The advantage of using PPG sensor data over electrocardiography is its higher availability in most healthcare settings and in wearable technology. In a cohort of 38 patients, reduced median daily PRVi was significantly associated with increase in two pediatric organ dysfunction scores after adjusting for confounders (p < 0.001). PRVi shows promise as a real-time physiologic marker of clinical deterioration using highly-available PPG data, but further research is warranted.

Whole-Proteome Analysis of Human Craniosynostotic Tissue Suggests a Link between Inflammatory Signaling and Osteoclast Activation in Human Cranial Suture Patency

BACKGROUND

The pathophysiology of nonsyndromic craniosynostosis remains poorly understood. The authors seek to understand the cause of this condition with a specific focus on how osteoclasts may contribute to craniosynostosis. Here, the authors characterize proteins differentially expressed in patent and fused cranial sutures by comparing their respective proteomes.

METHODS

Fused and patent suture samples were obtained from craniosynostotic patients undergoing surgery at a single academic medical center. Extracted protein from samples was interrogated using mass spectrometry. Differential protein expression was determined using maximum likelihood-based G-test with a q-value cutoffs of 0.5 after correction for multiple hypothesis testing. Immunolocalization of lead protein candidates was performed to validate proteomic findings. In addition, quantitative polymerase chain reaction analysis of corresponding gene expression of proteins of interest was performed.

RESULTS

Proteins differentially expressed in patent versus fused sutures included collagen 6A1 (Col6A1), fibromodulin, periostin, aggrecan, adipocyte enhancer-binding protein 1, and osteomodulin (OMD). Maximum likelihood-based G-test suggested that Col6A1, fibromodulin, and adipocyte enhancer-binding protein 1 are highly expressed in patent sutures compared with fused sutures, whereas OMD is up-regulated in fused sutures compared with patent sutures. These results were corroborated by immunohistochemistry. Quantitative polymerase chain reaction data point to an inverse relationship in proteins of interest to RNA transcript levels, in prematurely fused and patent sutures that potentially describes a feedback loop mechanism.

CONCLUSIONS

Proteome analysis validated by immunohistochemistry may provide insight into the mechanism of cranial suture patency and disease from an osteoclast perspective. The authors results suggest a role of inflammatory mediators in nonsyndromic craniosynostosis. Col6A1 may aid in the regulation of suture patency, and OMD may be involved in premature fusion. Additional validation studies are required.

Plasma Biomarkers of Inflammation Reflect Seizures and Hemorrhagic Activity of Cerebral Cavernous Malformations

The clinical course of cerebral cavernous malformations (CCMs) is highly variable. Based on recent discoveries implicating angiogenic and inflammatory mechanisms, we hypothesized that serum biomarkers might reflect chronic or acute disease activity. This single-site prospective observational cohort study included 85 CCM patients, in whom 24 a priori chosen plasma biomarkers were quantified and analyzed in relation to established clinical and imaging parameters of disease categorization and severity. We subsequently validated the positive correlations in longitudinal follow-up of 49 subjects. Plasma levels of matrix metalloproteinase-2 and intercellular adhesion molecule 1 were significantly higher (P = 0.02 and P = 0.04, respectively, FDR corrected), and matrix metalloproteinase-9 was lower (P = 0.04, FDR corrected) in patients with seizure activity at any time in the past. Vascular endothelial growth factor and endoglin (both P = 0.04, FDR corrected) plasma levels were lower in patients who had suffered a symptomatic bleed in the prior 3 months. The hierarchical clustering analysis revealed a cluster of four plasma inflammatory cytokines (interleukin 2, interferon gamma, tumor necrosis factor alpha, and interleukin 1 beta) separating patients into what we designated "high" and "low" inflammatory states. The "high" inflammatory state was associated with seizure activity (P = 0.02) and more than one hemorrhagic event during a patient's lifetime (P = 0.04) and with a higher rate of new hemorrhage, lesion growth, or new lesion formation (P < 0.05) during prospective follow-up. Peripheral plasma biomarkers reflect seizure and recent hemorrhagic activity in CCM patients. In addition, four clustered inflammatory biomarkers correlate with cumulative disease aggressiveness and predict future clinical activity.

Abstract 124: Plasma Biomarkers of inflammation Reflect Seizures and Hemorrhagic Activity of Cerebral Cavernous Malformations

Introduction: The clinical course cerebral of cavernous malformations (CCMs) is highly variable, with a limited number of recent studies querying factors associated with disease severity. We hereby explore a panel of peripheral plasma biomarkers implied with inflammation and angiogenesis in relation to CCM clinical activity.

Methods: Blood samples of 85 CCM patients (49 with solitary/sporadic lesions and 36 with multifocal/familial CCMs) were collected at the time of the clinical visit, concurrently with advanced MRI sequences. Twenty a priori chosen plasma biomarkers were quantified and analyzed in relation to established parameters of disease categorization and severity, including genotype, lesion burden, age at symptomatic presentation, CCM-related seizures and the number and timing of prior symptomatic hemorrhages. We first tested classic univariate correlations of each biomarker with disease features, including an FDR correction, and we then applied a multivariate hierarchical clustering approach. We further correlated the peripheral plasma biomarkers with measures of lesional permeability and iron deposition using previously validated MRI protocols.

Results: MMP2 and ICAM1 levels were significantly higher (p=0.02 and p=0.04 respectively) in patients with seizure activity while MMP9 was lower (p=0.04). VEGF and endoglin/CD105 (p=0.04 for both) plasma levels were both lower in patients who had suffered a symptomatic bleed in the prior 3 months. The hierarchical clustering analysis revealed a cluster of 4 plasma inflammatory cytokines (TNFα, IL1β, IL2 and IFNγ) separating patients into high and low inflammatory states. The high inflammatory state was associated with more CCM hemorrhagic events during a patient’s lifetime (p=0.04) but not recent bleeding. CCM lesion iron concentrations were inversely correlated with IL-10 (r=-0.61, p=0.02), CCL2/MCP1 (r=-0.60, p=0.02) and ROBO4 (r=-0.53, p=0.05) in CCM lesions that recently bled.

Conclusion: Peripheral plasma biomarkers reflect seizure and recent hemorrhagic activity from CCM. And clusters of pro-inflammatory biomarkers correlate with cumulative chronic disease aggressiveness. Other biomarkers may reflect the clearance of lesional iron after recent hemorrhage.

Automated Glycan Sequencing from Tandem Mass Spectra of N-Linked Glycopeptides

Mass spectrometry has become a routine experimental tool for proteomic biomarker analysis of human blood samples, partly due to the large availability of informatics tools. As one of the most common protein post-translational modifications (PTMs) in mammals, protein glycosylation has been observed to alter in multiple human diseases and thus may potentially be candidate markers of disease progression. While mass spectrometry instrumentation has seen advancements in capabilities, discovering glycosylation-related markers using existing software is currently not straightforward. Complete characterization of protein glycosylation requires the identification of intact glycopeptides in samples, including identification of the modification site as well as the structure of the attached glycans. In this paper, we present GlycoSeq, an open-source software tool that implements a heuristic iterated glycan sequencing algorithm coupled with prior knowledge for automated elucidation of the glycan structure within a glycopeptide from its collision-induced dissociation tandem mass spectrum. GlycoSeq employs rules of glycosidic linkage as defined by glycan synthetic pathways to eliminate improbable glycan structures and build reasonable glycan trees. We tested the tool on two sets of tandem mass spectra of N-linked glycopeptides cell lines acquired from breast cancer patients. After employing enzymatic specificity within the N-linked glycan synthetic pathway, the sequencing results of GlycoSeq were highly consistent with the manually curated glycan structures. Hence, GlycoSeq is ready to be used for the characterization of glycan structures in glycopeptides from MS/MS analysis. GlycoSeq is released as open source software at https://github.com/chpaul/GlycoSeq/ .

Synergistic Myeloma Cell Death via Novel Intracellular Activation of Caspase-10-Dependent Apoptosis by Carfilzomib and Selinexor

Exportin1 (XPO1; also known as chromosome maintenance region 1, or CRM1) controls nucleo-cytoplasmic transport of most tumor suppressors and is overexpressed in many cancers, including multiple myeloma, functionally impairing tumor suppressive function via target mislocalization. Selective inhibitor of nuclear export (SINE) compounds block XPO1-mediated nuclear escape by disrupting cargo protein binding, leading to retention of tumor suppressors, induction of cancer cell death, and sensitization to other drugs. Combined treatment with the clinical stage SINE compound selinexor and the irreversible proteasome inhibitor (PI) carfilzomib induced synergistic cell death of myeloma cell lines and primary plasma cells derived from relapsing/refractory myeloma patients and completely impaired the growth of myeloma cell line-derived tumors in mice. Investigating the details of SINE/PI-induced cell death revealed (i) reduced Bcl-2 expression and cleavage and inactivation of Akt, two prosurvival regulators of apoptosis and autophagy; (ii) intracellular membrane-associated aggregation of active caspases, which depended on caspase-10 protease activity; and (iii) novel association of caspase-10 and autophagy-associated proteins p62 and LC3 II, which may prime activation of the caspase cascade. Overall, our findings provide novel mechanistic rationale behind the potent cell death induced by combining selinexor with carfilzomib and support their use in the treatment of relapsed/refractory myeloma and potentially other cancers.

N-linked glycan profiling in neuroblastoma cell lines

Although MYCN amplification has been associated with aggressive neuroblastoma, the molecular mechanisms that differentiate low-risk, MYCN-nonamplified neuroblastoma from high-risk, MYCN-amplified disease are largely unknown. Genomic and proteomic studies have been limited in discerning differences in signaling pathways that account for this heterogeneity. N-Linked glycosylation is a common protein modification resulting from the attachment of sugars to protein residues and is important in cell signaling and immune response. Aberrant N-linked glycosylation has been routinely linked to various cancers. In particular, glycomic markers have often proven to be useful in distinguishing cancers from precancerous conditions. Here, we perform a systematic comparison of N-linked glycomic variation between MYCN-nonamplified SY5Y and MYCN-amplified NLF cell lines with the aim of identifying changes in sugar abundance linked to high-risk neuroblastoma. Through a combination of liquid chromatography-mass spectrometry and bioinformatics analysis, we identified 16 glycans that show a statistically significant change in abundance between NLF and SY5Y samples. Closer examination revealed the preference for larger (in terms of total monosaccharide count) and more sialylated glycan structures in the MYCN-amplified samples in comparison to smaller, nonsialylated glycans that are more dominant in the MYCN-nonamplified samples. These results offer clues for deriving marker candidates for accurate neuroblastoma risk diagnosis.

Glycoproteomics: identifying the glycosylation of prostate specific antigen at normal and high isoelectric points by LC-MS/MS

Prostate specific antigen (PSA) is currently used as a biomarker to diagnose prostate cancer. PSA testing has been widely used to detect and screen prostate cancer. However, in the diagnostic gray zone, the PSA test does not clearly distinguish between benign prostate hypertrophy and prostate cancer due to their overlap. To develop more specific and sensitive candidate biomarkers for prostate cancer, an in-depth understanding of the biochemical characteristics of PSA (such as glycosylation) is needed. PSA has a single glycosylation site at Asn69, with glycans constituting approximately 8% of the protein by weight. Here, we report the comprehensive identification and quantitation of N-glycans from two PSA isoforms using LC–MS/MS. There were 56 N-glycans associated with PSA, whereas 57 N-glycans were observed in the case of the PSA-high isoelectric point (pI) isoform (PSAH). Three sulfated/phosphorylated glycopeptides were detected, the identification of which was supported by tandem MS data. One of these sulfated/phosphorylated N-glycans, HexNAc5Hex4dHex1s/p1 was identified in both PSA and PSAH at relative intensities of 0.52 and 0.28%, respectively. Quantitatively, the variations were monitored between these two isoforms. Because we were one of the laboratories participating in the 2012 ABRF Glycoprotein Research Group (gPRG) study, those results were compared to that presented in this study. Our qualitative and quantitative results summarized here were comparable to those that were summarized in the interlaboratory study.