A general framework for inference on algorithm-agnostic variable importance

Non-Invasive Ventilation Failure in Pediatric ICU: A Machine Learning Driven Prediction

Background/Objectives: Non-invasive ventilation (NIV) has emerged as a possible first-step treatment to avoid invasive intubation in pediatric intensive care units (PICUs) due to its advantages in reducing intubation-associated risks. However, the timely identification of NIV failure is crucial to prevent adverse outcomes. This study aims to identify predictors of first-attempt NIV failure in PICU patients by testing various machine learning techniques and comparing their predictive abilities. Methods: Data were sourced from the TIPNet registry, which comprised patients admitted to 23 Italian Paediatric Intensive Care Units (PICUs). We selected patients between January 2010 and January 2024 who received non-invasive ventilation (NIV) as their initial approach to respiratory support. The study aimed to develop a predictive model for NIV failure, selecting the best Machine Learning technique, including Generalized Linear Models, Random Forest, Extreme Gradient Boosting, and Neural Networks. Additionally, an ensemble approach was implemented. Model performances were measured using sensitivity, specificity, AUROC, and predictive values. Moreover, the model calibration was evaluated. Results: Out of 43,794 records, 1861 admissions met the inclusion criteria, with 678 complete cases and 97 NIV failures. The RF model demonstrated the highest AUROC and sensitivity equal to 0.83 (0.64, 0.94). Base excess, weight, age, systolic blood pressure, and fraction of inspired oxygen were identified as the most predictive features. A check for model calibration ensured the model's reliability in predicting NIV failure probabilities. Conclusions: This study identified highly sensitive models for predicting NIV failure in PICU patients, with RF as a robust option.

Importance of variables from different time frames for predicting self-harm using health system data

OBJECTIVE

Self-harm risk prediction models developed using health system data (electronic health records and insurance claims information) often use patient information from up to several years prior to the index visit when the prediction is made. Measurements from some time periods may not be available for all patients. Using the framework of algorithm-agnostic variable importance, we study the predictive potential of variables corresponding to different time horizons prior to the index visit and demonstrate the application of variable importance techniques in the biomedical informatics setting.

MATERIALS AND METHODS

We use variable importance to quantify the potential of recent (up to three months before the index visit) and distant (more than one year before the index visit) patient mental health information for predicting self-harm risk using data from seven health systems. We quantify importance as the decrease in predictiveness when the variable set of interest is excluded from the prediction task. We define predictiveness using discriminative metrics: area under the receiver operating characteristic curve (AUC), sensitivity, and positive predictive value.

RESULTS

Mental health predictors corresponding to the three months prior to the index visit show strong signal of importance; in one setting, excluding these variables decreased AUC from 0.85 to 0.77. Predictors corresponding to more distant information were less important.

DISCUSSION

Predictors from the months immediately preceding the index visit are highly important. Implementation of self-harm prediction models may be challenging in settings where recent data are not completely available (e.g., due to lags in insurance claims processing) at the time a prediction is made.

CONCLUSION

Clinically derived variables from different time frames exhibit varying levels of importance for predicting self-harm. Variable importance analyses can inform whether and how to implement risk prediction models into clinical practice given real-world data limitations. These analyses be applied more broadly in biomedical informatics research to provide insight into general clinical risk prediction tasks.

Out of (the) bag-encoding categorical predictors impacts out-of-bag samples

Performance of random forest classification models is often assessed and interpreted using out-of-bag (OOB) samples. Observations which are OOB when a tree is trained may serve as a test set for that tree and predictions from the OOB observations used to calculate OOB error and variable importance measures (VIM). OOB errors are popular because they are fast to compute and, for large samples, are a good estimate of the true prediction error. In this study, we investigate how target-based vs. target-agnostic encoding of categorical predictor variables for random forest can bias performance measures based on OOB samples. We show that, when categorical variables are encoded using a target-based encoding method, and when the encoding takes place prior to bagging, the OOB sample can underestimate the true misclassification rate, and overestimate variable importance. We recommend using a separate test data set when evaluating variable importance and/or predictive performance of tree based methods that utilise a target-based encoding method.

Assessing variable importance in survival analysis using machine learning

Given a collection of features available for inclusion in a predictive model, it may be of interest to quantify the relative importance of a subset of features for the prediction task at hand. For example, in HIV vaccine trials, participant baseline characteristics are used to predict the probability of HIV acquisition over the intended follow-up period, and investigators may wish to understand how much certain types of predictors, such as behavioural factors, contribute to overall predictiveness. Time-to-event outcomes such as time to HIV acquisition are often subject to right censoring, and existing methods for assessing variable importance are typically not intended to be used in this setting. We describe a broad class of algorithm-agnostic variable importance measures for prediction in the context of survival data. We propose a nonparametric efficient estimation procedure that incorporates flexible learning of nuisance parameters, yields asymptotically valid inference and enjoys double robustness. We assess the performance of our proposed procedure via numerical simulations and analyse data from the HVTN 702 vaccine trial to inform enrolment strategies for future HIV vaccine trials.

Flexible variable selection in the presence of missing data

In many applications, it is of interest to identify a parsimonious set of features, or panel, from multiple candidates that achieves a desired level of performance in predicting a response. This task is often complicated in practice by missing data arising from the sampling design or other random mechanisms. Most recent work on variable selection in missing data contexts relies in some part on a finite-dimensional statistical model, e.g., a generalized or penalized linear model. In cases where this model is misspecified, the selected variables may not all be truly scientifically relevant and can result in panels with suboptimal classification performance. To address this limitation, we propose a nonparametric variable selection algorithm combined with multiple imputation to develop flexible panels in the presence of missing-at-random data. We outline strategies based on the proposed algorithm that achieve control of commonly used error rates. Through simulations, we show that our proposal has good operating characteristics and results in panels with higher classification and variable selection performance compared to several existing penalized regression approaches in cases where a generalized linear model is misspecified. Finally, we use the proposed method to develop biomarker panels for separating pancreatic cysts with differing malignancy potential in a setting where complicated missingness in the biomarkers arose due to limited specimen volumes.

Algorithm-agnostic significance testing in supervised learning with multimodal data

MOTIVATION

Valid statistical inference is crucial for decision-making but difficult to obtain in supervised learning with multimodal data, e.g. combinations of clinical features, genomic data, and medical images. Multimodal data often warrants the use of black-box algorithms, for instance, random forests or neural networks, which impede the use of traditional variable significance tests.

RESULTS

We address this problem by proposing the use of COvariance MEasure Tests (COMETs), which are calibrated and powerful tests that can be combined with any sufficiently predictive supervised learning algorithm. We apply COMETs to several high-dimensional, multimodal data sets to illustrate (i) variable significance testing for finding relevant mutations modulating drug-activity, (ii) modality selection for predicting survival in liver cancer patients with multiomics data, and (iii) modality selection with clinical features and medical imaging data. In all applications, COMETs yield results consistent with domain knowledge without requiring data-driven pre-processing, which may invalidate type I error control. These novel applications with high-dimensional multimodal data corroborate prior results on the power and robustness of COMETs for significance testing.

AVAILABILITY AND IMPLEMENTATION

COMETs are implemented in the cometsR package available on CRAN and pycometsPython library available on GitHub. Source code for reproducing all results is available at https://github.com/LucasKook/comets. All data sets used in this work are openly available.

Importance of variables from different time frames for predicting self-harm using health system data

Objective

Self-harm risk prediction models developed using health system data (electronic health records and insurance claims information) often use patient information from up to several years prior to the index visit when the prediction is made. Measurements from some time periods may not be available for all patients. Using the framework of algorithm-agnostic variable importance, we study the predictive potential of variables corresponding to different time horizons prior to the index visit and demonstrate the application of variable importance techniques in the biomedical informatics setting.

Materials and Methods

We use variable importance to quantify the potential of recent (up to three months before the index visit) and distant (more than one year before the index visit) patient mental health information for predicting self-harm risk using data from seven health systems. We quantify importance as the decrease in predictiveness when the variable set of interest is excluded from the prediction task. We define predictiveness using discriminative metrics: area under the receiver operating characteristic curve (AUC), sensitivity, and positive predictive value.

Results

Mental health predictors corresponding to the three months prior to the index visit show strong signal of importance; in one setting, excluding these variables decreased AUC from 0.85 to 0.77. Predictors corresponding to more distant information were less important.

Discussion

Predictors from the months immediately preceding the index visit are highly important. Implementation of self-harm prediction models may be challenging in settings where recent data are not completely available (e.g., due to lags in insurance claims processing) at the time a prediction is made.

Conclusion

Clinically derived variables from different time frames exhibit varying levels of importance for predicting self-harm. Variable importance analyses can inform whether and how to implement risk prediction models into clinical practice given real-world data limitations. These analyses be applied more broadly in biomedical informatics research to provide insight into general clinical risk prediction tasks.

SARS-CoV-2 Viral Load in the Nasopharynx at Time of First Infection Among Unvaccinated Individuals: A Secondary Cross-Protocol Analysis of 4 Randomized Trials

Importance

SARS-CoV-2 viral load (VL) in the nasopharynx is difficult to quantify and standardize across settings, but it may inform transmission potential and disease severity.

Objective

To characterize VL at COVID-19 diagnosis among previously uninfected and unvaccinated individuals by evaluating the association of demographic and clinical characteristics, viral variant, and trial with VL, as well as the ability of VL to predict severe disease.

Design, Setting, and Participants

This secondary cross-protocol analysis used individual-level data from placebo recipients from 4 harmonized, phase 3 COVID-19 vaccine efficacy trials sponsored by Moderna, AstraZeneca, Janssen, and Novavax. Participants were SARS-CoV-2 negative at baseline and acquired COVID-19 during the blinded phase of the trials. The setting included the US, Brazil, South Africa, Colombia, Argentina, Peru, Chile, and Mexico; start dates were July 27, 2020, to December 27, 2020; data cutoff dates were March 26, 2021, to July 30, 2021. Statistical analysis was performed from November 2022 to June 2023.

Main Outcomes and Measures

Linear regression was used to assess the association of demographic and clinical characteristics, viral variant, and trial with polymerase chain reaction-measured log10 VL in nasal and/or nasopharyngeal swabs taken at the time of COVID-19 diagnosis.

Results

Among 1667 participants studied (886 [53.1%] male; 995 [59.7%] enrolled in the US; mean [SD] age, 46.7 [14.7] years; 204 [12.2%] aged 65 years or older; 196 [11.8%] American Indian or Alaska Native, 150 [9%] Black or African American, 1112 [66.7%] White; 762 [45.7%] Hispanic or Latino), median (IQR) log10 VL at diagnosis was 6.18 (4.66-7.12) log10 copies/mL. Participant characteristics and viral variant explained only 5.9% of the variability in VL. The independent factor with the highest observed differences was trial: Janssen participants had 0.54 log10 copies/mL lower mean VL vs Moderna participants (95% CI, 0.20 to 0.87 log10 copies/mL lower). In the Janssen study, which captured the largest number of COVID-19 events and variants and used the most intensive post-COVID surveillance, neither VL at diagnosis nor averaged over days 1 to 28 post diagnosis was associated with COVID-19 severity.

Conclusions and Relevance

In this study of placebo recipients from 4 randomized phase 3 trials, high variability was observed in SARS-CoV-2 VL at the time of COVID-19 diagnosis, and only a fraction was explained by individual participant characteristics or viral variant. These results suggest challenges for future studies of interventions seeking to influence VL and elevates the importance of standardized methods for specimen collection and viral load quantitation.

Quantifying how single dose Ad26.COV2.S vaccine efficacy depends on Spike sequence features

In the ENSEMBLE randomized, placebo-controlled phase 3 trial (NCT04505722), estimated single-dose Ad26.COV2.S vaccine efficacy (VE) was 56% against moderate to severe-critical COVID-19. SARS-CoV-2 Spike sequences were determined from 484 vaccine and 1,067 placebo recipients who acquired COVID-19. In this set of prespecified analyses, we show that in Latin America, VE was significantly lower against Lambda vs. Reference and against Lambda vs. non-Lambda [family-wise error rate (FWER) p < 0.05]. VE differed by residue match vs. mismatch to the vaccine-insert at 16 amino acid positions (4 FWER p < 0.05; 12 q-value ≤ 0.20); significantly decreased with physicochemical-weighted Hamming distance to the vaccine-strain sequence for Spike, receptor-binding domain, N-terminal domain, and S1 (FWER p < 0.001); differed (FWER ≤ 0.05) by distance to the vaccine strain measured by 9 antibody-epitope escape scores and 4 NTD neutralization-impacting features; and decreased (p = 0.011) with neutralization resistance level to vaccinee sera. VE against severe-critical COVID-19 was stable across most sequence features but lower against the most distant viruses.

Prevention efficacy of the broadly neutralizing antibody VRC01 depends on HIV-1 envelope sequence features

In the Antibody Mediated Prevention (AMP) trials (HVTN 704/HPTN 085 and HVTN 703/HPTN 081), prevention efficacy (PE) of the monoclonal broadly neutralizing antibody (bnAb) VRC01 (vs. placebo) against HIV-1 acquisition diagnosis varied according to the HIV-1 Envelope (Env) neutralization sensitivity to VRC01, as measured by 80% inhibitory concentration (IC80). Here, we performed a genotypic sieve analysis, a complementary approach to gaining insight into correlates of protection that assesses how PE varies with HIV-1 sequence features. We analyzed HIV-1 Env amino acid (AA) sequences from the earliest available HIV-1 RNA-positive plasma samples from AMP participants diagnosed with HIV-1 and identified Env sequence features that associated with PE. The strongest Env AA sequence correlate in both trials was VRC01 epitope distance that quantifies the divergence of the VRC01 epitope in an acquired HIV-1 isolate from the VRC01 epitope of reference HIV-1 strains that were most sensitive to VRC01-mediated neutralization. In HVTN 704/HPTN 085, the Env sequence-based predicted probability that VRC01 IC80 against the acquired isolate exceeded 1 µg/mL also significantly associated with PE. In HVTN 703/HPTN 081, a physicochemical-weighted Hamming distance across 50 VRC01 binding-associated Env AA positions of the acquired isolate from the most VRC01-sensitive HIV-1 strain significantly associated with PE. These results suggest that incorporating mutation scoring by BLOSUM62 and weighting by the strength of interactions at AA positions in the epitope:VRC01 interface can optimize performance of an Env sequence-based biomarker of VRC01 prevention efficacy. Future work could determine whether these results extend to other bnAbs and bnAb combinations.

Predicting neutralization susceptibility to combination HIV-1 monoclonal broadly neutralizing antibody regimens

Combination monoclonal broadly neutralizing antibodies (bnAbs) are currently being developed for preventing HIV-1 infection. Recent work has focused on predicting in vitro neutralization potency of both individual bnAbs and combination regimens against HIV-1 pseudoviruses using Env sequence features. To predict in vitro combination regimen neutralization potency against a given HIV-1 pseudovirus, previous approaches have applied mathematical models to combine individual-bnAb neutralization and have predicted this combined neutralization value; we call this the combine-then-predict (CP) approach. However, prediction performance for some individual bnAbs has exceeded that for the combination, leading to another possibility: combining the individual-bnAb predicted values and using these to predict combination regimen neutralization; we call this the predict-then-combine (PC) approach. We explore both approaches in both simulated data and data from the Los Alamos National Laboratory's Compile, Neutralize, and Tally NAb Panels repository. The CP approach is superior to the PC approach when the neutralization outcome of interest is binary (e.g., neutralization susceptibility, defined as inhibitory concentration < 1 μg/mL. For continuous outcomes, the CP approach performs at least as well as the PC approach, and is superior to the PC approach when the individual-bnAb prediction algorithms have poor performance. This knowledge may be used when building prediction models for novel antibody combinations in the absence of in vitro neutralization data for the antibody combination; this, in turn, will aid in the evaluation and down-selection of these antibody combinations into prevention efficacy trials.

Efficient Estimation under Data Fusion

We aim to make inferences about a smooth, finite-dimensional parameter by fusing data from multiple sources together. Previous works have studied the estimation of a variety of parameters in similar data fusion settings, including in the estimation of the average treatment effect and average reward under a policy, with the majority of them merging one historical data source with covariates, actions, and rewards and one data source of the same covariates. In this work, we consider the general case where one or more data sources align with each part of the distribution of the target population, for example, the conditional distribution of the reward given actions and covariates. We describe potential gains in efficiency that can arise from fusing these data sources together in a single analysis, which we characterize by a reduction in the semiparametric efficiency bound. We also provide a general means to construct estimators that achieve these bounds. In numerical simulations, we illustrate marked improvements in efficiency from using our proposed estimators rather than their natural alternatives. Finally, we illustrate the magnitude of efficiency gains that can be realized in vaccine immunogenicity studies by fusing data from two HIV vaccine trials.

Application of the SLAPNAP statistical learning tool to broadly neutralizing antibody HIV prevention research

Combination monoclonal broadly neutralizing antibody (bnAb) regimens are in clinical development for HIV prevention, necessitating additional knowledge of bnAb neutralization potency/breadth against circulating viruses. Williamson et al. (2021) described a software tool, Super LeArner Prediction of NAb Panels (SLAPNAP), with application to any HIV bnAb regimen with sufficient neutralization data against a set of viruses in the Los Alamos National Laboratory's Compile, Neutralize, and Tally Nab Panels repository. SLAPNAP produces a proteomic antibody resistance (PAR) score for Env sequences based on predicted neutralization resistance and estimates variable importance of Env amino acid features. We apply SLAPNAP to compare HIV bnAb regimens undergoing clinical testing, finding improved power for downstream sieve analyses and increased precision for comparing neutralization potency/breadth of bnAb regimens due to the inclusion of PAR scores of Env sequences with much larger sample sizes available than for neutralization outcomes. SLAPNAP substantially improves bnAb regimen characterization, ranking, and down-selection.

Use and misuse of random forest variable importance metrics in medicine: demonstrations through incident stroke prediction

BACKGROUND

Machine learning tools such as random forests provide important opportunities for modeling large, complex modern data generated in medicine. Unfortunately, when it comes to understanding why machine learning models are predictive, applied research continues to rely on 'out of bag' (OOB) variable importance metrics (VIMPs) that are known to have considerable shortcomings within the statistics community. After explaining the limitations of OOB VIMPs - including bias towards correlated features and limited interpretability - we describe a modern approach called 'knockoff VIMPs' and explain its advantages.

METHODS

We first evaluate current VIMP practices through an in-depth literature review of 50 recent random forest manuscripts. Next, we recommend organized and interpretable strategies for analysis with knockoff VIMPs, including computing them for groups of features and considering multiple model performance metrics. To demonstrate methods, we develop a random forest to predict 5-year incident stroke in the Sleep Heart Health Study and compare results based on OOB and knockoff VIMPs.

RESULTS

Nearly all papers in the literature review contained substantial limitations in their use of VIMPs. In our demonstration, using OOB VIMPs for individual variables suggested two highly correlated lung function variables (forced expiratory volume, forced vital capacity) as the best predictors of incident stroke, followed by age and height. Using an organized analytic approach that considered knockoff VIMPs of both groups of features and individual features, the largest contributions to model sensitivity were medications (especially cardiovascular) and measured medical risk factors, while the largest contributions to model specificity were age, diastolic blood pressure, self-reported medical risk factors, polysomnography features, and pack-years of smoking. Thus, we reach very different conclusions about stroke risk factors using OOB VIMPs versus knockoff VIMPs.

CONCLUSIONS

The near-ubiquitous reliance on OOB VIMPs may provide misleading results for researchers who use such methods to guide their research. Given the rapid pace of scientific inquiry using machine learning, it is essential to bring modern knockoff VIMPs that are interpretable and unbiased into widespread applied practice to steer researchers using random forest machine learning toward more meaningful results.

Immune correlates analysis of the PREVENT-19 COVID-19 vaccine efficacy clinical trial

In the PREVENT-19 phase 3 trial of the NVX-CoV2373 vaccine (NCT04611802), anti-spike binding IgG concentration (spike IgG), anti-RBD binding IgG concentration (RBD IgG), and pseudovirus 50% neutralizing antibody titer (nAb ID50) measured two weeks post-dose two are assessed as correlates of risk and as correlates of protection against COVID-19. Analyses are conducted in the U.S. cohort of baseline SARS-CoV-2 negative per-protocol participants using a case-cohort design that measures the markers from all 12 vaccine recipient breakthrough COVID-19 cases starting 7 days post antibody measurement and from 639 vaccine recipient non-cases. All markers are inversely associated with COVID-19 risk and directly associated with vaccine efficacy. In vaccine recipients with nAb ID50 titers of 50, 100, and 7230 international units (IU50)/ml, vaccine efficacy estimates are 75.7% (49.8%, 93.2%), 81.7% (66.3%, 93.2%), and 96.8% (88.3%, 99.3%). The results support potential cross-vaccine platform applications of these markers for guiding decisions about vaccine approval and use.

The applications of machine learning in HIV neutralizing antibodies research-A systematic review

Machine learning algorithms play an essential role in bioinformatics and allow exploring the vast and noisy biological data in unrivaled ways. This paper is a systematic review of the applications of machine learning in the study of HIV neutralizing antibodies. This significant and vast research domain can pave the way to novel treatments and to a vaccine. We selected the relevant papers by investigating the available literature from the Web of Science and PubMed databases in the last decade. The computational methods are applied in neutralization potency prediction, neutralization span prediction against multiple viral strains, antibody-virus binding sites detection, enhanced antibodies design, and the study of the antibody-induced immune response. These methods are viewed from multiple angles spanning data processing, model description, feature selection, evaluation, and sometimes paper comparisons. The algorithms are diverse and include supervised, unsupervised, and generative types. Both classical machine learning and modern deep learning were taken into account. The review ends with our ideas regarding future research directions and challenges.

Testing a global null hypothesis using ensemble machine learning methods

Testing a global null hypothesis that there are no significant predictors for a binary outcome of interest among a large set of biomarker measurements is an important task in biomedical studies. We seek to improve the power of such testing methods by leveraging ensemble machine learning methods. Ensemble machine learning methods such as random forest, bagging, and adaptive boosting model the relationship between the outcome and the predictor nonparametrically, while stacking combines the strength of multiple learners. We demonstrate the power of the proposed testing methods through Monte Carlo studies and show the use of the methods by applying them to the immunologic biomarkers dataset from the RV144 HIV vaccine efficacy trial.

MDA for random forests: inconsistency, and a practical solution via the Sobol-MDA

Variable importance measures are the main tools used to analyse the black-box mechanisms of random forests. Although the mean decrease accuracy is widely accepted as the most efficient variable importance measure for random forests, little is known about its statistical properties. In fact, the definition of mean decrease accuracy varies across the main random forest software. In this article, our objective is to rigorously analyse the behaviour of the main mean decrease accuracy implementations. Consequently, we mathematically formalize the various implemented MDA algorithms, and then establish their limits when the sample size increases. This asymptotic analysis reveals that these mean decrease accuracy versions differ as importance measures, since they converge towards different quantities. More importantly, we break down these limits into three components: the first two terms are related to Sobol indices, which are well-defined measures of a covariate contribution to the response variance, widely used in the sensitivity analysis field, as opposed to the third term, whose value increases with dependence within covariates. Thus, we theoretically demonstrate that the mean decrease accuracy does not target the right quantity to detect influential covariates in a dependent setting, a fact that has already been noticed experimentally. To address this issue, we define a new importance measure for random forests, the Sobol-mean decrease accuracy, which fixes the flaws of the original mean decrease accuracy, and consistently estimates the accuracy decrease of the forest retrained without a given covariate, but with an efficient computational cost. The Sobol-mean decrease accuracy empirically outperforms its competitors on both simulated and real data for variable selection. An open source implementation in R and C ++ is available online.

Can auxiliary indicators improve COVID-19 forecasting and hotspot prediction?

Short-term forecasts of traditional streams from public health reporting (such as cases, hospitalizations, and deaths) are a key input to public health decision-making during a pandemic. Since early 2020, our research group has worked with data partners to collect, curate, and make publicly available numerous real-time COVID-19 indicators, providing multiple views of pandemic activity in the United States. This paper studies the utility of five such indicators-derived from deidentified medical insurance claims, self-reported symptoms from online surveys, and COVID-related Google search activity-from a forecasting perspective. For each indicator, we ask whether its inclusion in an autoregressive (AR) model leads to improved predictive accuracy relative to the same model excluding it. Such an AR model, without external features, is already competitive with many top COVID-19 forecasting models in use today. Our analysis reveals that 1) inclusion of each of these five indicators improves on the overall predictive accuracy of the AR model; 2) predictive gains are in general most pronounced during times in which COVID cases are trending in "flat" or "down" directions; and 3) one indicator, based on Google searches, seems to be particularly helpful during "up" trends.