Dynamic Mortality Risk Predictions for Children in ICUs: Development and Validation of Machine Learning Models

Development and validation of machine learning-based prediction model for central venous access device-related thrombosis in children

BACKGROUND

Identifying independent risk factors and implementing high-quality assessment tools for early detection of patients at high risk of central venous access device (CVAD)-related thrombosis (CRT) plays a critical role in delivering timely preventive interventions and reducing the incidence of CRT. Approaches for identifying the risk of CRT in children have not been well-researched.

OBJECTIVE

To identify the critical risk factors for CRT in children and to construct machine learning-based prediction models tailored to this group, providing a theoretical basis and technical support for the prediction and prevention of CRT in these patients.

STUDY DESIGN

Retrospective data of pediatric patients receiving CVAD catheterization from January 1, 2018 to June 31, 2023 in Tongji Hospital were collected and divided into a training set and an internal validation set in a ratio of 7:3. Relevant data from July 1, 2023 to July 1, 2024 were prospectively collected for external validation of the model. LASSO regression was applied to determine CRT independent risk factors. Subsequently, four prediction models were constructed using logistic regression (LR), random forest, artificial neural network, and eXtreme Gradient Boosting.

RESULTS

A total of 1445 children were included in this study and the overall incidence of CRT was 17.4 %. The LASSO regression screened out 11 critical variables, including history of thrombosis, leukemia, number of catheters, history of catheterization, chemotherapy, parenteral nutrition, mechanical prophylaxis, dialysis, hypertonic liquid, anticoagulants, and post-catheterization D-dimer. The LR model outperformed the other models in both internal and external validation and was considered the best model for this study, which was transformed into a nomogram.

CONCLUSIONS

This study identified 11 independent risk factors for CRT in children. The prediction model developed using LR algorithm demonstrated excellent clinical applicability and may provide valuable support for early prediction of CRT.

The Challenges and Opportunities in Pediatric Medical Device Innovation: Monitoring Devices

BACKGROUND

Medical device innovation and development for pediatric care lags behind that of adults. With higher technical risks, challenges in accumulating data, smaller market sizes, and limited return on investment, there is less incentive for pediatric device development. Consequently, translation of medical devices specifically designed to improve pediatric care is limited. However, a changing landscape and novel programs may support an expansion of pediatric device development.

METHODS

A keyword-based search was conducted in PubMed; market databases and Food and Drug Administration guidance documents were also used in this review. A total of 18,017 articles underwent a title and abstract review, and 190 articles were reviewed by the expert authors.

RESULTS

Collectively, challenges with evidence generation and business and technical forces have disincentivized innovators from pursuing pediatric device innovation. Innovative programs such as the System of Hospitals for Innovation in Pediatrics-Medical Devices and the existing Pediatric Device Consortia, if strategically leveraged, may foster a robust national pediatric device innovation ecosystem.

CONCLUSIONS

Whereas there is a higher opportunity cost for pediatric device development, leveraging the programs and incentives available in a "pediatric-first" approach may provide effective paths to larger total addressable markets for innovators.

The 2024 Pediatric Sepsis Challenge: Predicting In-Hospital Mortality in Children With Suspected Sepsis in Uganda

The aim of this "Technical Note" is to inform the pediatric critical care data research community about the "2024 Pediatric Sepsis Data Challenge." This competition aims to facilitate the development of open-source algorithms to predict in-hospital mortality in Ugandan children with sepsis. The challenge is to first develop an algorithm using a synthetic training dataset, which will then be scored according to standard diagnostic testing criteria, and then be evaluated against a nonsynthetic test dataset. The datasets originate from admissions to six hospitals in Uganda (2017-2020) and include 3837 children, 6 to 60 months old, who were confirmed or suspected to have a diagnosis of sepsis. The synthetic dataset was created from a random subset of the original data. The test validation dataset closely resembles the synthetic dataset. The challenge should generate an optimal model for predicting in-hospital mortality. Following external validation, this model could be used to improve the outcomes for children with proven or suspected sepsis in low- and middle-income settings.

Machine learning models to evaluate mortality in pediatric patients with pneumonia in the intensive care unit

OBJECTIVES

This study aimed to predict mortality in children with pneumonia who were admitted to the intensive care unit (ICU) to aid decision-making.

STUDY DESIGN

Retrospective cohort study conducted at a single tertiary hospital.

PATIENTS

This study included children who were admitted to the pediatric ICU at the National Taiwan University Hospital between 2010 and 2019 due to pneumonia.

METHODOLOGY

Two prediction models were developed using tree-structured machine learning algorithms. The primary outcomes were ICU mortality and 24-h ICU mortality. A total of 33 features, including demographics, underlying diseases, vital signs, and laboratory data, were collected from the electronic health records. The machine learning models were constructed using the development data set, and performance matrices were computed using the holdout test data set.

RESULTS

A total of 1231 ICU admissions of children with pneumonia were included in the final cohort. The area under the receiver operating characteristic curves (AUROCs) of the ICU mortality model and 24-h ICU mortality models was 0.80 (95% confidence interval [CI], 0.69-0.91) and 0.92 (95% CI, 0.86-0.92), respectively. Based on feature importance, the model developed in this study tended to predict increased mortality for the subsequent 24 h if a reduction in the blood pressure, peripheral capillary oxygen saturation (SpO2), or higher partial pressure of carbon dioxide (PCO2) were observed.

CONCLUSIONS

This study demonstrated that the machine learning models for predicting ICU mortality and 24-h ICU mortality in children with pneumonia have the potential to support decision-making, especially in resource-limited settings.

Delay Between Actual Occurrence of Patient Vital Sign and the Nominal Appearance in the Electronic Health Record: Single-Center, Retrospective Study of PICU Data, 2014-2018

OBJECTIVES

Patient vital sign data charted in the electronic health record (EHR) are used for time-sensitive decisions, yet little is known about when these data become nominally available compared with when the vital sign was actually measured. The objective of this study was to determine the magnitude of any delay between when a vital sign was actually measured in a patient and when it nominally appears in the EHR.

DESIGN

We performed a single-center retrospective cohort study.

SETTING

Tertiary academic children's hospital.

PATIENTS

A total of 5,458 patients were admitted to a PICU from January 2014 to December 2018.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We analyzed entry and display times of all vital signs entered in the EHR. The primary outcome measurement was time between vital sign occurrence and nominal timing of the vital sign in the EHR. An additional outcome measurement was the frequency of batch charting. A total of 9,818,901 vital sign recordings occurred during the study period. Across the entire cohort the median (interquartile range [IQR]) difference between time of occurrence and nominal time in the EHR was in hours:minutes:seconds, 00:41:58 (IQR 00:13:42-01:44:10). Lag in the first 24 hours of PICU admission was 00:47:34 (IQR 00:15:23-02:19:00), lag in the last 24 hours was 00:38:49 (IQR 00:13:09-01:29:22; p < 0.001). There were 1,892,143 occurrences of batch charting.

CONCLUSIONS

This retrospective study shows a lag between vital sign occurrence and its appearance in the EHR, as well as a frequent practice of batch charting. The magnitude of the delay-median ~40 minutes-suggests that vital signs available in the EHR for clinical review and incorporation into clinical alerts may be outdated by the time they are available.

Editor's Choice Articles for September

The September 2023 issue and this year has already proven to be important for improving our understanding of pediatric acute respiratory distress syndrome (PARDS); Pediatric Critical Care Medicine (PCCM) has published 16 articles so far. Therefore, my three Editor's Choice articles this month highlight yet more PCCM material about PARDS by covering the use of noninvasive ventilation (NIV), the trajectory in cytokine profile during illness, and a new look at lung mechanics. The PCCM Connections for Readers give us the opportunity to focus on some clinical biomarkers of severity and mortality risk during critical illness.

Clinical Instability Is a Sign of Severity of Illness: A Cohort Study

OBJECTIVES

Test the hypothesis that within patient clinical instability measured by deterioration and improvement in mortality risk over 3-, 6-, 9-, and 12-hour time intervals is indicative of increasing severity of illness.

DESIGN

Analysis of electronic health data from January 1, 2018, to February 29, 2020.

SETTING

PICU and cardiac ICU at an academic children's hospital.

PATIENTS

All PICU patients. Data included descriptive information, outcome, and independent variables used in the Criticality Index-Mortality.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

There were 8,399 admissions with 312 deaths (3.7%). Mortality risk determined every three hours using the Criticality Index-Mortality, a machine learning algorithm calibrated to this hospital. Since the sample sizes were sufficiently large to expect statical differences, we also used two measures of effect size, the proportion of time deaths had greater instability than survivors, and the rank-biserial correlation, to assess the magnitude of the effect and complement our hypothesis tests. Within patient changes were compared for survivors and deaths. All comparisons of survivors versus deaths were less than 0.001. For all time intervals, two measures of effect size indicated that the differences between deaths and survivors were not clinically important. However, the within-patient maximum risk increase (clinical deterioration) and maximum risk decrease (clinical improvement) were both substantially greater in deaths than survivors for all time intervals. For deaths, the maximum risk increase ranged from 11.1% to 16.1% and the maximum decrease ranged from -7.3% to -10.0%, while the median maximum increases and decreases for survivors were all less than ± 0.1%. Both measures of effect size indicated moderate to high clinical importance. The within-patient volatility was greater than 4.5-fold greater in deaths than survivors during the first ICU day, plateauing at ICU days 4-5 at 2.5 greater volatility.

CONCLUSIONS

Episodic clinical instability measured with mortality risk is a reliable sign of increasing severity of illness. Mortality risk changes during four time intervals demonstrated deaths have greater maximum and within-patient clinical instability than survivors. This observation confirms the clinical teaching that clinical instability is a sign of severity of illness.

Signatures of illness in children requiring unplanned intubation in the pediatric intensive care unit: A retrospective cohort machine-learning study

Acute respiratory failure requiring the initiation of invasive mechanical ventilation remains commonplace in the pediatric intensive care unit (PICU). Early recognition of patients at risk for respiratory failure may provide clinicians with the opportunity to intervene and potentially improve outcomes. Through the development of a random forest model to identify patients at risk for requiring unplanned intubation, we tested the hypothesis that subtle signatures of illness are present in physiological and biochemical time series of PICU patients in the early stages of respiratory decompensation. We included 116 unplanned intubation events as recorded in the National Emergency Airway Registry for Children in 92 PICU admissions over a 29-month period at our institution. We observed that children have a physiologic signature of illness preceding unplanned intubation in the PICU. Generally, it comprises younger age, and abnormalities in electrolyte, hematologic and vital sign parameters. Additionally, given the heterogeneity of the PICU patient population, we found differences in the presentation among the major patient groups - medical, cardiac surgical, and non-cardiac surgical. At four hours prior to the event, our random forest model demonstrated an area under the receiver operating characteristic curve of 0.766 (0.738 for medical, 0.755 for cardiac surgical, and 0.797 for non-cardiac surgical patients). The multivariable statistical models that captured the physiological and biochemical dynamics leading up to the event of urgent unplanned intubation in a PICU can be repurposed for bedside risk prediction.

Identification and Optimization of Contributing Factors for Precocious Puberty by Machine/Deep Learning Methods in Chinese Girls

BACKGROUND AND OBJECTIVES

As the worldwide secular trends are toward earlier puberty, identification of contributing factors for precocious puberty is critical. We aimed to identify and optimize contributing factors responsible for onset of precocious puberty via machine learning/deep learning algorithms in girls.

METHODS

A cross-sectional study was performed among girls aged 6-16 years from 26 schools in Beijing based on a cluster sampling method. Information was gleaned online via questionnaires. Machine/deep learning algorithms were performed using Python language (v3.7.6) on PyCharm platform.

RESULTS

Of 11308 students enrolled, there are 5527 girls, and 408 of them had experienced precocious puberty. Training 13 machine learning algorithms revealed that gradient boosting machine (GBM) performed best in predicting precocious puberty. By comparison, six top factors including maternal age at menarche, paternal body mass index (BMI), waist-to-height ratio, maternal BMI, screen time, and physical activity were sufficient in prediction performance, with accuracy of 0.9530, precision of 0.9818, and area under the receiver operating characteristic curve (AUROC) of 0.7861. The performance of the top six factors was further validated by deep learning sequential model, with accuracy reaching 92.9%.

CONCLUSIONS

We identified six important factors from both parents and girls that can help predict the onset of precocious puberty among Chinese girls.

The criticality Index-mortality: A dynamic machine learning prediction algorithm for mortality prediction in children cared for in an ICU

BACKGROUND

The Criticality Index-Mortality uses physiology, therapy, and intensity of care to compute mortality risk for pediatric ICU patients. If the frequency of mortality risk computations were increased to every 3 h with model performance that could improve the assessment of severity of illness, it could be utilized to monitor patients for significant mortality risk change.

OBJECTIVES

To assess the performance of a dynamic method of updating mortality risk every 3 h using the Criticality Index-Mortality methodology and identify variables that are significant contributors to mortality risk predictions.

POPULATION

There were 8,399 pediatric ICU admissions with 312 (3.7%) deaths from January 1, 2018 to February 29, 2020. We randomly selected 75% of patients for training, 13% for validation, and 12% for testing.

MODEL

A neural network was trained to predict hospital survival or death during or following an ICU admission. Variables included age, gender, laboratory tests, vital signs, medications categories, and mechanical ventilation variables. The neural network was calibrated to mortality risk using nonparametric logistic regression.

RESULTS

Discrimination assessed across all time periods found an AUROC of 0.851 (0.841-0.862) and an AUPRC was 0.443 (0.417-0.467). When assessed for performance every 3 h, the AUROCs had a minimum value of 0.778 (0.689-0.867) and a maximum value of 0.885 (0.841,0.862); the AUPRCs had a minimum value 0.148 (0.058-0.328) and a maximum value of 0.499 (0.229-0.769). The calibration plot had an intercept of 0.011, a slope of 0.956, and the R² was 0.814. Comparison of observed vs. expected proportion of deaths revealed that 95.8% of the 543 risk intervals were not statistically significantly different. Construct validity assessed by death and survivor risk trajectories analyzed by mortality risk quartiles and 7 high and low risk diseases confirmed a priori clinical expectations about the trajectories of death and survivors.

CONCLUSIONS

The Criticality Index-Mortality computing mortality risk every 3 h for pediatric ICU patients has model performance that could enhance the clinical assessment of severity of illness. The overall Criticality Index-Mortality framework was effectively applied to develop an institutionally specific, and clinically relevant model for dynamic risk assessment of pediatric ICU patients.