Incorporating repeated measurements into prediction models in the critical care setting: a framework, systematic review and meta-analysis

Construction and validation of a clinical prediction model for sepsis using peripheral perfusion index to predict in-hospital and 28-day mortality risk

Sepsis is a clinical syndrome caused by infection, leading to organ dysfunction due to a dysregulated host response. In recent years, its high mortality rate has made it a significant cause of death and disability worldwide. The pathophysiological process of sepsis is related to the body's dysregulated response to infection, with microcirculatory changes serving as early warning signals that guide clinical treatment. The Peripheral Perfusion Index (PI), as an indicator of peripheral microcirculation, can effectively evaluate patient prognosis. This study aims to develop two new prediction models using PI and other common clinical indicators to assess the mortality risk of sepsis patients during hospitalization and within 28 days post-ICU admission. This retrospective study analyzed data from sepsis patients treated in the Intensive Care Unit of Peking Union Medical College Hospital between December 2019 and June 2023, ultimately including 645 patients. LASSO regression and logistic regression analyses were used to select predictive factors from 35 clinical indicators, and two clinical prediction models were constructed to predict in-hospital mortality and 28-day mortality. The models' performance was then evaluated using ROC curve, calibration curve, and decision curve analyses. The two prediction models performed excellently in distinguishing patient mortality risk. The AUC for the in-hospital mortality prediction model was 0.82 in the training set and 0.73 in the validation set; for the 28-day mortality prediction model, the AUC was 0.79 in the training set and 0.73 in the validation set. The calibration curves closely aligned with the ideal line, indicating consistency between predicted and actual outcomes. Decision curve analysis also demonstrated high net benefits for the clinical utility of both models. The study shows that these two prediction models not only perform excellently statistically but also hold high practical value in clinical applications. The models can help physicians accurately assess the mortality risk of sepsis patients, providing a scientific basis for personalized treatment.

Predictive Accuracy of Infant Clinical Sign Algorithms for Mortality in Young Infants Aged 0 to 59 Days: A Systematic Review

CONTEXT

Clinical sign algorithms are a key strategy to identify young infants at risk of mortality.

OBJECTIVE

Synthesize the evidence on the accuracy of clinical sign algorithms to predict all-cause mortality in young infants 0-59 days.

DATA SOURCES

MEDLINE, Embase, CINAHL, Global Index Medicus, and Cochrane CENTRAL Registry of Trials.

STUDY SELECTION

Studies evaluating the accuracy of infant clinical sign algorithms to predict mortality.

DATA EXTRACTION

We used Cochrane methods for study screening, data extraction, and risk of bias assessment. We determined certainty of evidence using Grading of Recommendations Assessment Development and Evaluation.

RESULTS

We included 11 studies examining 26 algorithms. Three studies from non-hospital/community settings examined sign-based checklists (n = 13). Eight hospital-based studies validated regression models (n = 13), which were administered as weighted scores (n = 8), regression formulas (n = 4), and a nomogram (n = 1). One checklist from India had a sensitivity of 98% (95% CI: 88%-100%) and specificity of 94% (93%-95%) for predicting sepsis-related deaths. However, external validation in Bangladesh showed very low sensitivity of 3% (0%-10%) with specificity of 99% (99%-99%) for all-cause mortality (ages 0-9 days). For hospital-based prediction models, area under the curve (AUC) ranged from 0.76-0.93 (n = 13). The Score for Essential Neonatal Symptoms and Signs had an AUC of 0.89 (0.84-0.93) in the derivation cohort for mortality, and external validation showed an AUC of 0.83 (0.83-0.84).

LIMITATIONS

Heterogeneity of algorithms and lack of external validation limited the evidence.

CONCLUSIONS

Clinical sign algorithms may help identify at-risk young infants, particularly in hospital settings; however, overall certainty of evidence is low with limited external validation.

Clinical prediction models in children that use repeated measurements with time-varying covariates: a scoping review

BACKGROUND

Emerging evidence suggests that clinical prediction models that use repeated (time-varying) measurements within each patient may have higher predictive accuracy than models that use patient information from a single measurement.

OBJECTIVE

To determine the breadth of the published literature reporting the development of clinical prediction models in children that use time-varying predictors.

DATA SOURCES

MEDLINE, EMBASE and Cochrane databases.

ELIGIBILITY CRITERIA

We included studies reporting the development of a multivariable clinical prediction model in children, with or without validation, to predict a repeatedly measured binary or time-to-event outcome and utilizing at least one repeatedly measured predictor.

SYNTHESIS METHODS

We categorized included studies by the method used to model time-varying predictors.

RESULTS

Of 99 clinical prediction model studies that had a repeated measurements data structure, only 27 (27%) used methods that incorporated the repeated measurements as time-varying predictors in a single model. Among these 27 time-varying prediction model studies, we grouped model types into nine categories: time-dependent Cox regression, generalized estimating equations, random effects model, landmark model, joint model, neural network, K-nearest neighbor, support vector machine and tree-based algorithms. Where there was comparison of time-varying models to single measurement models, using time-varying predictors improved predictive accuracy.

CONCLUSIONS

Various methods have been used to develop time-varying prediction models in children, but there is a paucity of pediatric time-varying models in the literature. Incorporating time-varying covariates in pediatric prediction models may improve predictive accuracy. Future research in pediatric prediction model development should further investigate whether incorporation of time-varying covariates improves predictive accuracy.

Young Infants Clinical Signs Study 8-sign Algorithm for Identification of Sick Infants Adapted for Routine Home Visits: A Systematic Review and Critical Appraisal of its Measurement Properties

Objective. The 8-sign algorithm adapted from the Young Infants Clinical Signs Study (YICSS) is widely used to identify sick infants during home visits (YICSS-home algorithm). We aimed to critically appraise the development and evidence of measurement properties, including sensibility, reliability, and validity, of the YICSS-home algorithm. Methods. Relevant studies were identified through a systematic literature search. Results. The YICSS-home algorithm has good sensibility. The algorithm demonstrated at least moderate inter-rater reliability and sensitivity ranging from 69% to 80%. However, the algorithm was developed among sick infants brought for care to a health facility and not initially developed for use by community health workers (CHWs) during home visits. Some important risk factors were omitted at item generation. Inter-CHW reliability and construct validity have not been estimated. Conclusion. Future research should build on the strengths of the YICSS-home algorithm and address its limitations to develop a new algorithm with improved predictive accuracy.

Improving cardiovascular risk prediction through machine learning modelling of irregularly repeated electronic health records

Aims

Existing electronic health records (EHRs) often consist of abundant but irregular longitudinal measurements of risk factors. In this study, we aim to leverage such data to improve the risk prediction of atherosclerotic cardiovascular disease (ASCVD) by applying machine learning (ML) algorithms, which can allow automatic screening of the population.

Methods and results

A total of 215 744 Chinese adults aged between 40 and 79 without a history of cardiovascular disease were included (6081 cases) from an EHR-based longitudinal cohort study. To allow interpretability of the model, the predictors of demographic characteristics, medication treatment, and repeatedly measured records of lipids, glycaemia, obesity, blood pressure, and renal function were used. The primary outcome was ASCVD, defined as non-fatal acute myocardial infarction, coronary heart disease death, or fatal and non-fatal stroke. The eXtreme Gradient boosting (XGBoost) algorithm and Least Absolute Shrinkage and Selection Operator (LASSO) regression models were derived to predict the 5-year ASCVD risk. In the validation set, compared with the refitted Chinese guideline-recommended Cox model (i.e. the China-PAR), the XGBoost model had a significantly higher C-statistic of 0.792, (the differences in the C-statistics: 0.011, 0.006-0.017, P < 0.001), with similar results reported for LASSO regression (the differences in the C-statistics: 0.008, 0.005-0.011, P < 0.001). The XGBoost model demonstrated the best calibration performance (men: Dx = 0.598, P = 0.75; women: Dx = 1.867, P = 0.08). Moreover, the risk distribution of the ML algorithms differed from that of the conventional model. The net reclassification improvement rates of XGBoost and LASSO over the Cox model were 3.9% (1.4-6.4%) and 2.8% (0.7-4.9%), respectively.

Conclusion

Machine learning algorithms with irregular, repeated real-world data could improve cardiovascular risk prediction. They demonstrated significantly better performance for reclassification to identify the high-risk population correctly.

Study protocol for "In-vehicle sensors to detect changes in cognition of older drivers"

BACKGROUND

Driving is a complex behavior that may be affected by early changes in the cognition of older individuals. Early changes in driving behavior may include driving more slowly, making fewer and shorter trips, and errors related to inadequate anticipation of situations. Sensor systems installed in older drivers' vehicles may detect these changes and may generate early warnings of possible changes in cognition.

METHOD

A naturalistic longitudinal design is employed to obtain continuous information on driving behavior that will be compared with the results of extensive cognitive testing conducted every 3 months for 3 years. A driver facing camera, forward facing camera, and telematics unit are installed in the vehicle and data downloaded every 3 months when the cognitive tests are administered.

RESULTS

Data processing and analysis will proceed through a series of steps including data normalization, adding information on external factors (weather, traffic conditions), and identifying critical features (variables). Traditional prediction modeling results will be compared with Recurring Neural Network (RNN) approach to produce Driver Behavior Indices (DBIs), and algorithms to classify drivers within age, gender, ethnic group membership, and other potential group characteristics.

CONCLUSION

It is well established that individuals with progressive dementias are eventually unable to drive safely, yet many remain unaware of their cognitive decrements. Current screening and evaluation services can test only a small number of individuals with cognitive concerns, missing many who need to know if they require treatment. Given the increasing number of sensors being installed in passenger vehicles and pick-up trucks and their increasing acceptability, reconfigured in-vehicle sensing systems could provide widespread, low-cost early warnings of cognitive decline to the large number of older drivers on the road in the U.S. The proposed testing and evaluation of a readily and rapidly available, unobtrusive in-vehicle sensing system could provide the first step toward future widespread, low-cost early warnings of cognitive change for this large number of older drivers in the U.S. and elsewhere.

Cefepime Daily Exposure and the Associated Impact on the Change in Sequential Organ Failure Assessment Scores and Vasopressors Requirement in Critically Ill Patients Using Repeated-Measures Mixed-Effect Modeling

IMPORTANCE

Sepsis and septic shock are major healthcare problems that need early and appropriate management.

OBJECTIVES

To evaluate the association of daily cefepime pharmacokinetic/pharmacodynamic (PK/PD) parameters with change in Sequential Organ Failure Assessment (SOFA) score and vasopressors requirement.

DESIGN SETTING AND PARTICIPANTS

This is a retrospective study. Adult ICU patients who received cefepime for Gram-negative pneumonia or bloodstream infection (BSI) and had cefepime concentrations measured were included. Daily cefepime exposure was generated and PK/PD parameters calculated for patients. Repeated-measures mixed-effect modeling was used to evaluate the impact of PK/PD on the outcomes.

MAIN OUTCOMES AND MEASURES

Change in daily SOFA score and vasopressors requirement.

RESULTS

A total of 394 and 207 patients were included in the SOFA and vasopressors analyses, respectively. The mean (±sd) age was 55 years (19) and weight 81 kg (29). For the change in SOFA score, daily SOFA score, mechanical ventilation, renal replacement therapy, and number of vasopressors were included. In the vasopressors analysis, daily SOFA score, day of therapy, and hydrocortisone dose were significant covariates in the final model. Achieving cefepime concentrations above the minimum inhibitory concentration (MIC) (T>MIC) for 100% of the dosing interval was associated with 0.006 µg/kg/min decrease in norepinephrine-equivalent dose. Cefepime PK/PD did not have an impact on the daily change in SOFA score.

CONCLUSIONS AND RELEVANCE

Achieving 100% T>MIC was associated with negligible decrease in vasopressors requirement in ICU patients with Gram-negative pneumonia and BSI. There was no impact on the change in SOFA score.

Vital signs-based deterioration prediction model assumptions can lead to losses in prediction performance

OBJECTIVE

Vital signs-based models are complicated by repeated measures per patient and frequently missing data. This paper investigated the impacts of common vital signs modeling assumptions during clinical deterioration prediction model development.

STUDY DESIGN AND SETTING

Electronic medical record (EMR) data from five Australian hospitals (1 January 2019-31 December 2020) were used. Summary statistics for each observation's prior vital signs were created. Missing data patterns were investigated using boosted decision trees, then imputed with common methods. Two example models predicting in-hospital mortality were developed, as follows: logistic regression and eXtreme Gradient Boosting. Model discrimination and calibration were assessed using the C-statistic and nonparametric calibration plots.

RESULTS

The data contained 5,620,641 observations from 342,149 admissions. Missing vitals were associated with observation frequency, vital sign variability, and patient consciousness. Summary statistics improved discrimination slightly for logistic regression and markedly for eXtreme Gradient Boosting. Imputation method led to notable differences in model discrimination and calibration. Model calibration was generally poor.

CONCLUSION

Summary statistics and imputation methods can improve model discrimination and reduce bias during model development, but it is questionable whether these differences are clinically significant. Researchers should consider why data are missing during model development and how this may impact clinical utility.

Multivariate longitudinal data for survival analysis of cardiovascular event prediction in young adults: insights from a comparative explainable study

BACKGROUND

Multivariate longitudinal data are under-utilized for survival analysis compared to cross-sectional data (CS - data collected once across cohort). Particularly in cardiovascular risk prediction, despite available methods of longitudinal data analysis, the value of longitudinal information has not been established in terms of improved predictive accuracy and clinical applicability.

METHODS

We investigated the value of longitudinal data over and above the use of cross-sectional data via 6 distinct modeling strategies from statistics, machine learning, and deep learning that incorporate repeated measures for survival analysis of the time-to-cardiovascular event in the Coronary Artery Risk Development in Young Adults (CARDIA) cohort. We then examined and compared the use of model-specific interpretability methods (Random Survival Forest Variable Importance) and model-agnostic methods (SHapley Additive exPlanation (SHAP) and Temporal Importance Model Explanation (TIME)) in cardiovascular risk prediction using the top-performing models.

RESULTS

In a cohort of 3539 participants, longitudinal information from 35 variables that were repeatedly collected in 6 exam visits over 15 years improved subsequent long-term (17 years after) risk prediction by up to 8.3% in C-index compared to using baseline data (0.78 vs. 0.72), and up to approximately 4% compared to using the last observed CS data (0.75). Time-varying AUC was also higher in models using longitudinal data (0.86-0.87 at 5 years, 0.79-0.81 at 10 years) than using baseline or last observed CS data (0.80-0.86 at 5 years, 0.73-0.77 at 10 years). Comparative model interpretability analysis revealed the impact of longitudinal variables on model prediction on both the individual and global scales among different modeling strategies, as well as identifying the best time windows and best timing within that window for event prediction. The best strategy to incorporate longitudinal data for accuracy was time series massive feature extraction, and the easiest interpretable strategy was trajectory clustering.

CONCLUSION

Our analysis demonstrates the added value of longitudinal data in predictive accuracy and epidemiological utility in cardiovascular risk survival analysis in young adults via a unified, scalable framework that compares model performance and explainability. The framework can be extended to a larger number of variables and other longitudinal modeling methods.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT00005130, Registration Date: 26/05/2000.

Modern Learning from Big Data in Critical Care: Primum Non Nocere

Large and complex data sets are increasingly available for research in critical care. To analyze these data, researchers use techniques commonly referred to as statistical learning or machine learning (ML). The latter is known for large successes in the field of diagnostics, for example, by identification of radiological anomalies. In other research areas, such as clustering and prediction studies, there is more discussion regarding the benefit and efficiency of ML techniques compared with statistical learning. In this viewpoint, we aim to explain commonly used statistical learning and ML techniques and provide guidance for responsible use in the case of clustering and prediction questions in critical care. Clustering studies have been increasingly popular in critical care research, aiming to inform how patients can be characterized, classified, or treated differently. An important challenge for clustering studies is to ensure and assess generalizability. This limits the application of findings in these studies toward individual patients. In the case of predictive questions, there is much discussion as to what algorithm should be used to most accurately predict outcome. Aspects that determine usefulness of ML, compared with statistical techniques, include the volume of the data, the dimensionality of the preferred model, and the extent of missing data. There are areas in which modern ML methods may be preferred. However, efforts should be made to implement statistical frameworks (e.g., for dealing with missing data or measurement error, both omnipresent in clinical data) in ML methods. To conclude, there are important opportunities but also pitfalls to consider when performing clustering or predictive studies with ML techniques. We advocate careful valuation of new data-driven findings. More interaction is needed between the engineer mindset of experts in ML methods, the insight in bias of epidemiologists, and the probabilistic thinking of statisticians to extract as much information and knowledge from data as possible, while avoiding harm.

Harnessing repeated measurements of predictor variables for clinical risk prediction: a review of existing methods

BACKGROUND

Clinical prediction models (CPMs) predict the risk of health outcomes for individual patients. The majority of existing CPMs only harness cross-sectional patient information. Incorporating repeated measurements, such as those stored in electronic health records, into CPMs may provide an opportunity to enhance their performance. However, the number and complexity of methodological approaches available could make it difficult for researchers to explore this opportunity. Our objective was to review the literature and summarise existing approaches for harnessing repeated measurements of predictor variables in CPMs, primarily to make this field more accessible for applied researchers.

METHODS

MEDLINE, Embase and Web of Science were searched for articles reporting the development of a multivariable CPM for individual-level prediction of future binary or time-to-event outcomes and modelling repeated measurements of at least one predictor. Information was extracted on the following: the methodology used, its specific aim, reported advantages and limitations, and software available to apply the method.

RESULTS

The search revealed 217 relevant articles. Seven methodological frameworks were identified: time-dependent covariate modelling, generalised estimating equations, landmark analysis, two-stage modelling, joint-modelling, trajectory classification and machine learning. Each of these frameworks satisfies at least one of three aims: to better represent the predictor-outcome relationship over time, to infer a covariate value at a pre-specified time and to account for the effect of covariate change.

CONCLUSIONS

The applicability of identified methods depends on the motivation for including longitudinal information and the method's compatibility with the clinical context and available patient data, for both model development and risk estimation in practice.