Measuring the performance of prediction models to personalize treatment choice

Developing a multivariable prediction model to support personalized selection among five major empirically-supported treatments for adult depression. Study protocol of a systematic review and individual participant data network meta-analysis

BACKGROUND

Various treatments are recommended as first-line options in practice guidelines for depression, but it is unclear which is most efficacious for a given person. Accurate individualized predictions of relative treatment effects are needed to optimize treatment recommendations for depression and reduce this disorder's vast personal and societal costs.

AIMS

We describe the protocol for a systematic review and individual participant data (IPD) network meta-analysis (NMA) to inform personalized treatment selection among five major empirically-supported depression treatments.

METHOD

We will use the METASPY database to identify randomized clinical trials that compare two or more of five treatments for adult depression: antidepressant medication, cognitive therapy, behavioral activation, interpersonal psychotherapy, and psychodynamic therapy. We will request IPD from identified studies. We will conduct an IPD-NMA and develop a multivariable prediction model that estimates individualized relative treatment effects from demographic, clinical, and psychological participant characteristics. Depressive symptom level at treatment completion will constitute the primary outcome. We will evaluate this model using a range of measures for discrimination and calibration, and examine its potential generalizability using internal-external cross-validation.

CONCLUSIONS

We describe a state-of-the-art method to predict personalized treatment effects based on IPD from multiple trials. The resulting prediction model will need prospective evaluation in mental health care for its potential to inform shared decision-making. This study will result in a unique database of IPD from randomized clinical trials around the world covering five widely used depression treatments, available for future research.

Measuring the Performance of Survival Models to Personalize Treatment Choices

Various statistical and machine learning algorithms can be used to predict treatment effects at the patient level using data from randomized clinical trials (RCTs). Such predictions can facilitate individualized treatment decisions. Recently, a range of methods and metrics were developed for assessing the accuracy of such predictions. Here, we extend these methods, focusing on the case of survival (time-to-event) outcomes. We start by providing alternative definitions of the participant-level treatment benefit; subsequently, we summarize existing and propose new measures for assessing the performance of models estimating participant-level treatment benefits. We explore metrics assessing discrimination and calibration for benefit and decision accuracy. These measures can be used to assess the performance of statistical as well as machine learning models and can be useful during model development (i.e., for model selection or for internal validation) or when testing a model in new settings (i.e., in an external validation). We illustrate methods using simulated data and real data from the OPERAM trial, an RCT in multimorbid older people, which randomized participants to either standard care or a pharmacotherapy optimization intervention. We provide R codes for implementing all models and measures.

Predicting benefit from adjuvant therapy with corticosteroids in community-acquired pneumonia: a data-driven analysis of randomised trials

BACKGROUND

Despite several randomised controlled trials (RCTs) on the use of adjuvant treatment with corticosteroids in patients with community-acquired pneumonia (CAP), the effect of this intervention on mortality remains controversial. We aimed to evaluate heterogeneity of treatment effect (HTE) of adjuvant treatment with corticosteroids on 30-day mortality in patients with CAP.

METHODS

In this individual patient data meta-analysis, we included RCTs published before July 1, 2024, comparing adjuvant treatment with corticosteroids versus placebo in patients hospitalised with CAP. The primary endpoint was 30-day all-cause mortality, collected across all trials, and analyses followed the intention-to-treat principle. We analysed HTE using risk and effect modelling. For risk modelling, patients were classified as having less severe or severe CAP based on the pneumonia severity index (PSI), comparing PSI class I-III versus class IV-V. For effect modelling, we trained a corticosteroid-effect model on six trials and externally validated it using data from two trials, received after model preregistration. This model classified patients into two groups: no predicted benefit and predicted benefit from adjuvant treatment with corticosteroids. The literature search was registered on PROSPERO, CRD42022380746.

FINDINGS

We included eight RCTs with 3224 patients. Across all eight trials, 246 (7·6%) patients died within 30 days (106 [6·6%] of 1618 in the corticosteroid group vs 140 [8·7%] of 1606 in the placebo group; odds ratio [OR] 0·72 [95% CI 0·56-0·94], p=0·017). The corticosteroid-effect model, which selected C-reactive protein (CRP), showed significant HTE during external validation in the two most recent trials. In these trials, 154 (11·4%) of 1355 patients died within 30 days (88 [13·1%] of 671 in the placebo group vs 66 [9·6%] of 684 in the corticosteroid group; OR 0·71 [95% CI 0·50-0·99], p=0·044). Among patients predicted to have no benefit (CRP ≤204 mg/L, n=725), no significant effect was observed (OR 0·98 [95% CI 0·63-1·50]), whereas for those with predicted benefit (CRP >204 mg/L, n=630), 39 (13·0%) of 301 patients died in the placebo group compared with 20 (6·1%) of 329 in the corticosteroid group (0·43 [0·25-0·76], pinteraction=0·026). No significant HTE was found between less severe CAP (PSI class I-III, n=229) and severe CAP (PSI class IV-V, n=1126). Corticosteroid therapy significantly increased hyperglycaemia risk (44 [12·8%] of 344 in the placebo group vs 84 [24·8%] of 339 in the corticosteroid group; OR 2·50 [95% CI 1·63-3·83], p<0·0001) and hospital re-admission risk (30 [3·7%] of 814 in the placebo group vs 57 [7·0%] of 819 in the corticosteroid group; 1·95 [1·24-3·07], p=0·0038).

INTERPRETATION

Overall, adjuvant therapy with corticosteroids significantly reduces 30-day mortality in patients hospitalised with CAP. The treatment effect varied significantly among subgroups based on CRP concentrations, with a substantial mortality reduction observed only in patients with high baseline CRP.

FUNDING

None.

Efficacy of psychotherapy versus pharmacotherapy, or their combination, in chronic depression: study protocol for a systematic review and network meta-analysis using aggregated and individual patient data

INTRODUCTION

Chronic depression represents a common and highly disabling disorder. Several randomised controlled trials (RCTs) investigated the effectiveness of psychological, pharmacological and combined treatments for chronic depression. This is the first overarching systematic review and network meta-analysis (NMA) based on aggregated and individual patient data comparing the efficacy and acceptability of various treatment options for all subtypes of chronic depression. Furthermore, individual demographic and clinical characteristics that predict or moderate therapy outcomes will be investigated.

METHODS AND ANALYSIS

A systematic literature search of the Cochrane Library, MEDLINE via Ovid, PsycINFO, Web of Science and metapsy databases will be conducted from database inception without language restrictions to include all available samples from RCTs that investigated the efficacy of psychotherapy versus pharmacotherapy, or their combination in adult inpatients or outpatients with a primary diagnosis of chronic depression. Exclusively internet-based treatment studies will be excluded. The main outcome is depression severity measured on a continuous observer-rated scale for depression at 6 months post-treatment (range 3-12 months). Two reviewers will independently screen and select eligible studies based on the predefined inclusion and exclusion criteria. Risk of bias will be assessed using version 2 of the Cochrane risk-of-bias tool for randomised trials (RoB 2). Individual patient data (IPD) will be requested and incorporated in the network when provided, as it is the gold standard of evidence. For studies which do not provide IPD, aggregate data (AD) will be extracted and incorporated in lieu of IPD for the NMA, strengthening the evidence base and leveraging all existing evidence regardless of data availability restrictions. An NMA comparing psychotherapies and a network meta-regression estimating individualised treatment effects of psychotherapy will be implemented assuming a Bayesian framework. All models will be fitted in R with calls to JAGS. Empirical informative prior distributions will be used for model parameters where available, and non-informative priors will be used in cases where empirical priors are not available.

ETHICS AND DISSEMINATION

This IPD-NMA requires no ethical approval. All results will be disseminated as peer-reviewed publication in a leading journal in this field and presented at (inter)national scientific conferences.

PROSPERO REGISTRATION NUMBER

CRD42024526755.

Machine learning approaches to evaluate heterogeneous treatment effects in randomized controlled trials: a scoping review

BACKGROUND AND OBJECTIVES

Estimating heterogeneous treatment effects (HTEs) in randomized controlled trials (RCTs) has received substantial attention recently. This has led to the development of several statistical and machine learning (ML) algorithms to assess HTEs through identifying individualized treatment effects. However, a comprehensive review of these algorithms is lacking. We thus aimed to catalog and outline currently available statistical and ML methods for identifying HTEs via effect modeling using clinical RCT data and summarize how they have been applied in practice.

STUDY DESIGN AND SETTING

We performed a scoping review using prespecified search terms in MEDLINE and Embase, aiming to identify studies that assessed HTEs using advanced statistical and ML methods in RCT data published from 2010 to 2022.

RESULTS

Among a total of 32 studies identified in the review, 17 studies applied existing algorithms to RCT data, and 15 extended existing algorithms or proposed new algorithms. Applied algorithms included penalized regression, causal forest, Bayesian causal forest, and other metalearner frameworks. Of these methods, causal forest was the most frequently used (7 studies) followed by Bayesian causal forest (4 studies). Most applications were in cardiology (6 studies), followed by psychiatry (4 studies). We provide example R codes in simulated data to illustrate how to implement these algorithms.

CONCLUSION

This review identified and outlined various algorithms currently used to identify HTEs and individualized treatment effects in RCT data. Given the increasing availability of new algorithms, analysts should carefully select them after examining model performance and considering how the models will be used in practice.

Evaluating individualized treatment effect predictions: A model-based perspective on discrimination and calibration assessment

In recent years, there has been a growing interest in the prediction of individualized treatment effects. While there is a rapidly growing literature on the development of such models, there is little literature on the evaluation of their performance. In this paper, we aim to facilitate the validation of prediction models for individualized treatment effects. The estimands of interest are defined based on the potential outcomes framework, which facilitates a comparison of existing and novel measures. In particular, we examine existing measures of discrimination for benefit (variations of the c-for-benefit), and propose model-based extensions to the treatment effect setting for discrimination and calibration metrics that have a strong basis in outcome risk prediction. The main focus is on randomized trial data with binary endpoints and on models that provide individualized treatment effect predictions and potential outcome predictions. We use simulated data to provide insight into the characteristics of the examined discrimination and calibration statistics under consideration, and further illustrate all methods in a trial of acute ischemic stroke treatment. The results show that the proposed model-based statistics had the best characteristics in terms of bias and accuracy. While resampling methods adjusted for the optimism of performance estimates in the development data, they had a high variance across replications that limited their accuracy. Therefore, individualized treatment effect models are best validated in independent data. To aid implementation, a software implementation of the proposed methods was made available in R.

Validation of predicted individual treatment effects in out of sample respondents

Personalized medicine promises the ability to improve patient outcomes by tailoring treatment recommendations to the likelihood that any given patient will respond well to a given treatment. It is important that predictions of treatment response be validated and replicated in independent data to support their use in clinical practice. In this paper, we propose and test an approach for validating predictions of individual treatment effects with continuous outcomes across samples that uses matching in a test (validation) sample to match individuals in the treatment and control arms based on their predicted treatment response and their predicted response under control. To examine the proposed validation approach, we conducted simulations where test data is generated from either an identical, similar, or unrelated process to the training data. We also examined the impact of nuisance variables. To demonstrate the use of this validation procedure in the context of predicting individual treatment effects in the treatment of alcohol use disorder, we apply our validation procedure using data from a clinical trial of combined behavioral and pharmacotherapy treatments. We find that the validation algorithm accurately confirms validation and lack of validation, and also provides insights into cases where test data were generated under similar, but not identical conditions. We also show that the presence of nuisance variables detrimentally impacts algorithm performance, which can be partially reduced though the use of variable selection methods. An advantage of the approach is that it can be widely applied to different predictive methods.

Uncovering individualised treatment effects for educational trials

Large-scale Randomised Controlled Trials (RCTs) are widely regarded as "the gold standard" for testing the causal effects of school-based interventions. RCTs typically present the statistical significance of the average treatment effect (ATE), which captures the effect an intervention has had on average for a given population. However, key decisions in child health and education are often about individuals who may be very different from those averages. One way to identify heterogeneous treatment effects across different individuals, not captured by the ATE, is to conduct subgroup analyses. For example, free school meal (FSM) pupils as required for projects funded by the Education Endowment Foundation (EEF) in England. These subgroup analyses, as we demonstrate in 48 EEF-funded RCTs involving over 200,000 students, are usually not standardised across studies and offer flexible degrees of freedom to researchers, potentially leading to mixed, if not misleading, results. Here, we develop and deploy an alternative to ATE and subgroup analysis, a machine-learning and regression-based framework to predict individualised treatment effects (ITEs). ITEs could show where an intervention worked, for which individuals, and to what extent. Our findings have implications for decision-makers in fields like education, healthcare, law, and clinical practices concerning children and adolescents.

Development and Validation of PRE-SARC (PREdiction of SARCopenia Risk in Community Older Adults) Sarcopenia Prediction Model

OBJECTIVE

Reliable identification of high-risk older adults who are likely to develop sarcopenia is essential to implement targeted preventive measures and follow-up. However, no sarcopenia prediction model is currently available for community use. Our objective was to develop and validate a risk prediction model for calculating the 1-year absolute risk of developing sarcopenia in an aging population.

METHODS

One prospective population-based cohort of non-sarcopenic individuals aged 60 years or older were used for the development of a sarcopenia risk prediction model and model validation. Sarcopenia was defined according to the 2019 Asian Working Group for Sarcopenia consensus. Stepwise logistic regression was used to identify risk factors for sarcopenia incidence within a 1-year follow-up. Model performance was evaluated using the area under the receiver operating characteristics curve (AUROC) and calibration plot, respectively.

RESULTS

The development cohort included 1042 older adults, among whom 87 participants developed sarcopenia during a 1-year follow-up. The PRE-SARC (PREdiction of SARCopenia Risk in community older adults) model can accurately predict the 1-year risk of sarcopenia by using 7 easily accessible community-based predictors. The PRE-SARC model performed well in predicting sarcopenia, with an AUROC of 87% (95% CI, 0.83-0.90) and good calibration. Internal validation showed minimal optimism, with an adjusted AUROC of 0.85. The prediction score was categorized into 4 risk groups: low (0%-10%), moderate (>10%-20%), high (>20%-40%), and very high (>40%). The PRE-SARC model has been incorporated into an online risk calculator, which is freely accessible for daily clinical applications (https://sarcopeniariskprediction.shinyapps.io/dynnomapp/).

CONCLUSIONS

In community-dwelling individuals, the PRE-SARC model can accurately predict 1-year sarcopenia incidence. This model serves as a readily available and free accessible tool to identify older adults at high risk of sarcopenia, thereby facilitating personalized early preventive approaches and optimizing the utilization of health care resources.

Combining randomized and non-randomized data to predict heterogeneous effects of competing treatments

Some patients benefit from a treatment while others may do so less or do not benefit at all. We have previously developed a two-stage network meta-regression prediction model that synthesized randomized trials and evaluates how treatment effects vary across patient characteristics. In this article, we extended this model to combine different sources of types in different formats: aggregate data (AD) and individual participant data (IPD) from randomized and non-randomized evidence. In the first stage, a prognostic model is developed to predict the baseline risk of the outcome using a large cohort study. In the second stage, we recalibrated this prognostic model to improve our predictions for patients enrolled in randomized trials. In the third stage, we used the baseline risk as effect modifier in a network meta-regression model combining AD, IPD randomized clinical trial to estimate heterogeneous treatment effects. We illustrated the approach in the re-analysis of a network of studies comparing three drugs for relapsing-remitting multiple sclerosis. Several patient characteristics influence the baseline risk of relapse, which in turn modifies the effect of the drugs. The proposed model makes personalized predictions for health outcomes under several treatment options and encompasses all relevant randomized and non-randomized evidence.

Framework for personalized prediction of treatment response in relapsing-remitting multiple sclerosis: a replication study in independent data

BACKGROUND

Individualizing and optimizing treatment of relapsing-remitting multiple sclerosis patients is a challenging problem, which would benefit from a clinically valid decision support. Stühler et al. presented black box models for this aim which were developed and internally evaluated in a German registry but lacked external validation.

METHODS

In patients from the French OFSEP registry, we independently built and validated models predicting being free of relapse and free of confirmed disability progression (CDP), following the methodological roadmap and predictors reported by Stühler. Hierarchical Bayesian models were fit to predict the outcomes under 6 disease-modifying treatments given the individual disease course up to the moment of treatment change. Data was temporally split on 2017, and models were developed in patients treated earlier (n = 5517). Calibration curves, discrimination, mean squared error (MSE) and relative percentage of root MSE (RMSE%) were assessed by external validation of models in more-recent patients (n = 3768). Non-Bayesian fixed-effects GLMs were also applied and their outcomes were compared to these of the Bayesian ones. For both, we modelled the number of on-therapy relapses with a negative binomial distribution, and CDP occurrence with a binomial distribution.

RESULTS

The performance of our temporally-validated relapse model (MSE: 0.326, C-Index: 0.639) is potentially superior to that of Stühler's (MSE: 0.784, C-index: 0.608). Calibration plots revealed miscalibration. Our CDP model (MSE: 0.072, C-Index: 0.777) was also better than its counterpart (MSE: 0.131, C-index: 0.554). Results from non-Bayesian fixed-effects GLM models were similar to the Bayesian ones.

CONCLUSIONS

The relapse and CDP models rebuilt and externally validated in independent data could compare and strengthen the credibility of the Stühler models. Their model-building strategy was replicable.

Prediction Under Interventions: Evaluation of Counterfactual Performance Using Longitudinal Observational Data

Predictions under interventions are estimates of what a person's risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision-making. However, evaluating the predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance do not apply when using observational data, because prediction under interventions involves obtaining predictions of the outcome under conditions that are different from those that are observed for a subset of individuals in the validation dataset. This work describes methods for evaluating counterfactual performance of predictions under interventions for time-to-event outcomes. This means we aim to assess how well predictions would match the validation data if all individuals had followed the treatment strategy under which predictions are made. We focus on counterfactual performance evaluation using longitudinal observational data, and under treatment strategies that involve sustaining a particular treatment regime over time. We introduce an estimation approach using artificial censoring and inverse probability weighting that involves creating a validation dataset mimicking the treatment strategy under which predictions are made. We extend measures of calibration, discrimination (c-index and cumulative/dynamic AUCt) and overall prediction error (Brier score) to allow assessment of counterfactual performance. The methods are evaluated using a simulation study, including scenarios in which the methods should detect poor performance. Applying our methods in the context of liver transplantation shows that our procedure allows quantification of the performance of predictions supporting crucial decisions on organ allocation.

Predicting Individual Treatment Effects: Challenges and Opportunities for Machine Learning and Artificial Intelligence

Personalized medicine seeks to identify the right treatment for the right patient at the right time. Predicting the treatment effect for an individual patient has the potential to transform treatment of patients and drastically improve patients outcomes. In this work, we illustrate the potential for ML and AI methods to yield useful predictions of individual treatment effects. Using the predicted individual treatment effects (PITE) framework which uses baseline covariates (features) to predict whether a treatment is expected to yield benefit for a given patient compared to an alternative intervention we provide an illustration of the potential of such approaches and provide a detailed discussion of opportunities for further research and open challenges when seeking to predict individual treatment effects.

Development and validation of a prognostic score to identify the optimal candidate for preemptive TIPS in patients with cirrhosis and acute variceal bleeding

BACKGROUND AND AIM

Baveno VII workshop recommends the use of preemptive TIPS (p-TIPS) in patients with cirrhosis and acute variceal bleeding (AVB) at high- risk of treatment failure. However, the criteria defining "high-risk" have low clinical accessibility or include subjective variables. We aimed to develop and externally validate a model for better identification of p-TIPS candidates.

APPROACH AND RESULTS

The derivation cohort included 1554 patients with cirrhosis and AVB who were treated with endoscopy plus drug (n = 1264) or p-TIPS (n = 290) from 12 hospitals in China between 2010 and 2017. We first used competing risk regression to develop a score for predicting 6-week and 1-year mortality in patients treated with endoscopy plus drugs, which included age, albumin, bilirubin, international normalized ratio, white blood cell, creatinine, and sodium. The score was internally validated with the bootstrap method, which showed good discrimination (6 wk/1 y concordance-index: 0.766/0.740) and calibration, and outperformed other currently available models. In the second stage, the developed score was combined with treatment and their interaction term to predicate the treatment effect of p-TIPS (mortality risk difference between treatment groups) in the whole derivation cohort. The estimated treatment effect of p-TIPS varied substantially among patients. The prediction model had good discriminative ability (6 wk/1 y c -for-benefit: 0.696/0.665) and was well calibrated. These results were confirmed in the validation dataset of 445 patients with cirrhosis with AVB from 6 hospitals in China between 2017 and 2019 (6-wk/1-y c-for-benefit: 0.675/0.672).

CONCLUSIONS

We developed and validated a clinical prediction model that can help to identify individuals who will benefit from p-TIPS, which may guide clinical decision-making.

Comparison of causal forest and regression-based approaches to evaluate treatment effect heterogeneity: an application for type 2 diabetes precision medicine

OBJECTIVE

Precision medicine requires reliable identification of variation in patient-level outcomes with different available treatments, often termed treatment effect heterogeneity. We aimed to evaluate the comparative utility of individualized treatment selection strategies based on predicted individual-level treatment effects from a causal forest machine learning algorithm and a penalized regression model.

METHODS

Cohort study characterizing individual-level glucose-lowering response (6 month reduction in HbA1c) in people with type 2 diabetes initiating SGLT2-inhibitor or DPP4-inhibitor therapy. Model development set comprised 1,428 participants in the CANTATA-D and CANTATA-D2 randomised clinical trials of SGLT2-inhibitors versus DPP4-inhibitors. For external validation, calibration of observed versus predicted differences in HbA1c in patient strata defined by size of predicted HbA1c benefit was evaluated in 18,741 patients in UK primary care (Clinical Practice Research Datalink).

RESULTS

Heterogeneity in treatment effects was detected in clinical trial participants with both approaches (proportion predicted to have a benefit on SGLT2-inhibitor therapy over DPP4-inhibitor therapy: causal forest: 98.6%; penalized regression: 81.7%). In validation, calibration was good with penalized regression but sub-optimal with causal forest. A strata with an HbA1c benefit > 10 mmol/mol with SGLT2-inhibitors (3.7% of patients, observed benefit 11.0 mmol/mol [95%CI 8.0-14.0]) was identified using penalized regression but not causal forest, and a much larger strata with an HbA1c benefit 5-10 mmol with SGLT2-inhibitors was identified with penalized regression (regression: 20.9% of patients, observed benefit 7.8 mmol/mol (95%CI 6.7-8.9); causal forest 11.6%, observed benefit 8.7 mmol/mol (95%CI 7.4-10.1).

CONCLUSIONS

Consistent with recent results for outcome prediction with clinical data, when evaluating treatment effect heterogeneity researchers should not rely on causal forest or other similar machine learning algorithms alone, and must compare outputs with standard regression, which in this evaluation was superior.