A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint

The potential benefit of statin prescription based on prediction of treatment responsiveness in older individuals: an application to the PROSPER randomized controlled trial

AIMS

Clinical guidelines often recommend treating individuals based on their cardiovascular risk. We revisit this paradigm and quantify the efficacy of three treatment strategies: (i) overall prescription, i.e. treatment to all individuals sharing the eligibility criteria of a trial; (ii) risk-stratified prescription, i.e. treatment only to those at an elevated outcome risk; and (iii) prescription based on predicted treatment responsiveness.

METHODS AND RESULTS

We reanalysed the PROSPER randomized controlled trial, which included individuals aged 70-82 years with a history of, or risk factors for, vascular diseases. We conducted the derivation and internal-external validation of a model predicting treatment responsiveness. We compared with placebo (n = 2913): (i) pravastatin (n = 2891); (ii) pravastatin in the presence of previous vascular diseases and placebo in the absence thereof (n = 2925); and (iii) pravastatin in the presence of a favourable prediction of treatment response and placebo in the absence thereof (n = 2890). We found an absolute difference in primary outcome events composed of coronary death, non-fatal myocardial infarction, and fatal or non-fatal stroke, per 10 000 person-years equal to: -78 events (95% CI, -144 to -12) when prescribing pravastatin to all participants; -66 events (95% CI, -114 to -18) when treating only individuals with an elevated vascular risk; and -103 events (95% CI, -162 to -44) when restricting pravastatin to individuals with a favourable prediction of treatment response.

CONCLUSION

Pravastatin prescription based on predicted responsiveness may have an encouraging potential for cardiovascular prevention. Further external validation of our results and clinical experiments are needed.

TRIAL REGISTRATION

ISRCTN40976937.

Machine Learning-Driven Analysis of Individualized Treatment Effects Comparing Buprenorphine and Naltrexone in Opioid Use Disorder Relapse Prevention

OBJECTIVE

A trial comparing extended-release naltrexone and sublingual buprenorphine-naloxone demonstrated higher relapse rates in individuals randomized to extended-release naltrexone. The effectiveness of treatment might vary based on patient characteristics. We hypothesized that causal machine learning would identify individualized treatment effects for each medication.

METHODS

This is a secondary analysis of a multicenter randomized trial that compared the effectiveness of extended-release naltrexone versus buprenorphine-naloxone for preventing relapse of opioid misuse. Three machine learning models were derived using all trial participants with 50% randomly selected for training (n = 285) and the remaining 50% for validation. Individualized treatment effect was measured by the Qini value and c-for-benefit, with the absence of relapse denoting treatment success. Patients were grouped into quartiles by predicted individualized treatment effect to examine differences in characteristics and the observed treatment effects.

RESULTS

The best-performing model had a Qini value of 4.45 (95% confidence interval, 1.02-7.83) and a c-for-benefit of 0.63 (95% confidence interval, 0.53-0.68). The quartile most likely to benefit from buprenorphine-naloxone had a 35% absolute benefit from this treatment, and at study entry, they had a high median opioid withdrawal score (P < 0.001), used cocaine on more days over the prior 30 days than other quartiles (P < 0.001), and had highest proportions with alcohol and cocaine use disorder (P ≤ 0.02). Quartile 4 individuals were predicted to be most likely to benefit from extended-release naltrexone, with the greatest proportion having heroin drug preference (P = 0.02) and all experiencing homelessness (P < 0.001).

CONCLUSIONS

Causal machine learning identified differing individualized treatment effects between medications based on characteristics associated with preventing relapse.

Bayesian Multivariate Logistic Regression for Superiority and Inferiority Decision-Making under Observable Treatment Heterogeneity

The effects of treatments may differ between persons with different characteristics. Addressing such treatment heterogeneity is crucial to investigate whether patients with specific characteristics are likely to benefit from a new treatment. The current paper presents a novel Bayesian method for superiority decision-making in the context of randomized controlled trials with multivariate binary responses and heterogeneous treatment effects. The framework is based on three elements: a) Bayesian multivariate logistic regression analysis with a Pólya-Gamma expansion; b) a transformation procedure to transfer obtained regression coefficients to a more intuitive multivariate probability scale (i.e., success probabilities and the differences between them); and c) a compatible decision procedure for treatment comparison with prespecified decision error rates. Procedures for a priori sample size estimation under a non-informative prior distribution are included. A numerical evaluation demonstrated that decisions based on a priori sample size estimation resulted in anticipated error rates among the trial population as well as subpopulations. Further, average and conditional treatment effect parameters could be estimated unbiasedly when the sample was large enough. Illustration with the International Stroke Trial dataset revealed a trend toward heterogeneous effects among stroke patients: Something that would have remained undetected when analyses were limited to average treatment effects.

Covariate Balancing Methods for Randomized Controlled Trials Are Not Adversarially Robust

The first step toward investigating the effectiveness of a treatment via a randomized trial is to split the population into control and treatment groups then compare the average response of the treatment group receiving the treatment to the control group receiving the placebo. To ensure that the difference between the two groups is caused only by the treatment, it is crucial that the control and the treatment groups have similar statistics. Indeed, the validity and reliability of a trial are determined by the similarity of two groups' statistics. Covariate balancing methods increase the similarity between the distributions of the two groups' covariates. However, often in practice, there are not enough samples to accurately estimate the groups' covariate distributions. In this article, we empirically show that covariate balancing with the standardized means difference (SMD) covariate balancing measure, as well as Pocock and Simon's sequential treatment assignment method, are susceptible to worst case treatment assignments. Worst case treatment assignments are those admitted by the covariate balance measure, but result in highest possible ATE estimation errors. We developed an adversarial attack to find adversarial treatment assignment for any given trial. Then, we provide an index to measure how close the given trial is to the worst case. To this end, we provide an optimization-based algorithm, namely adversarial treatment assignment in treatment effect trials (ATASTREET), to find the adversarial treatment assignments.

Combining randomized and non-randomized data to predict heterogeneous effects of competing treatments

Some patients benefit from a treatment while others may do so less or do not benefit at all. We have previously developed a two-stage network meta-regression prediction model that synthesized randomized trials and evaluates how treatment effects vary across patient characteristics. In this article, we extended this model to combine different sources of types in different formats: aggregate data (AD) and individual participant data (IPD) from randomized and non-randomized evidence. In the first stage, a prognostic model is developed to predict the baseline risk of the outcome using a large cohort study. In the second stage, we recalibrated this prognostic model to improve our predictions for patients enrolled in randomized trials. In the third stage, we used the baseline risk as effect modifier in a network meta-regression model combining AD, IPD randomized clinical trial to estimate heterogeneous treatment effects. We illustrated the approach in the re-analysis of a network of studies comparing three drugs for relapsing-remitting multiple sclerosis. Several patient characteristics influence the baseline risk of relapse, which in turn modifies the effect of the drugs. The proposed model makes personalized predictions for health outcomes under several treatment options and encompasses all relevant randomized and non-randomized evidence.

Delayed versus primary closure to minimize risk of surgical-site infection for complicated appendicitis: A secondary analysis of a randomized trial using counterfactual prediction modeling

OBJECTIVE

To evaluate the risk of surgical site infection (SSI) following complicated appendectomy in individual patients receiving delayed primary closure (DPC) versus primary closure (PC) after adjustment for individual risk factors.

DESIGN

Secondary analysis of randomized controlled trial (RCT) with prediction model.

SETTING

Referral centers across Thailand.

PARTICIPANTS

Adult patients who underwent appendectomy via a lower-right-quadrant abdominal incision due to complicated appendicitis.

METHODS

A secondary analysis of a published RCT was performed applying a counterfactual prediction model considering interventions (PC vs DPC) and other significant predictors. A multivariable logistic regression was applied, and a likelihood-ratio test was used to select significant predictors to retain in a final model. Factual versus counterfactual SSI risks for individual patients along with individual treatment effect (iTE) were estimated.

RESULTS

In total, 546 patients (271 PC vs 275 DPC) were included in the analysis. The individualized prediction model consisted of allocated intervention, diabetes, type of complicated appendicitis, fecal contamination, and incision length. The iTE varied between 0.4% and 7% for PC compared to DPC; ∼38.1% of patients would have ≥2.1% lower SSI risk following PC compared to DPC. The greatest risk reduction was identified in diabetes with ruptured appendicitis, fecal contamination, and incision length of 10 cm, where SSI risks were 47.1% and 54.1% for PC and DPC, respectively.

CONCLUSIONS

In this secondary analysis, we found that most patients benefited from early PC versus DPC. Findings may be used to inform SSI prevention strategies for patients with complicated appendicitis.

Beyond traditional training: Integrating data from semi-immersive VR dual-task intervention in Parkinsonian Syndromes. A study protocol

Completing cognitive and motor tasks simultaneously requires a high level of cognitive control in terms of executive processes and attentional abilities. Most of the daily activities require a dual-task performance. While walking, for example, it may be necessary to adapt gait to obstacles of the environment or simply participate in a conversation; all these activities involve more than one ability at the same time. This parallel performance may be critical in the cognitive or motor load, especially for patients with neurological diseases such as Parkinsonian Syndromes. Patients are often characterized by a crucial impairment in performing both tasks concurrently, showing a decrease in attention skills and executive functions, thus leading to increased negative outcomes. In this scenario, the accurate assessment of the components involved in dual-task performance is crucial, and providing an early specific training program appears to be essential. The objective of this protocol is to assess cognitive and motor components involved in dual-task performance and create a training program based on ecological activities focusing on executive and motor functions. Thus, we will employ Virtual Reality to provide semi-immersive, multisensory, ecological, standardized, and realistic experiences for rehabilitative purposes in patients with Parkinsonian Syndromes, considering its high prevalence in aging and the incidence of motor and cognitive dysfunctions in this population. Moreover, we propose to integrate the great amount of different data provided by dual-task and Virtual Reality system, using machine learning techniques. These integrations may increase the treatment's reliability in terms of better prognostic indexes and individualized training.

Prognosis and prediction of antibiotic benefit in adults with clinically diagnosed acute rhinosinusitis: an individual participant data meta-analysis

BACKGROUND

A previous individual participant data meta-analysis (IPD-MA) of antibiotics for adults with clinically diagnosed acute rhinosinusitis (ARS) showed a marginal overall effect of antibiotics, but was unable to identify patients that are most likely to benefit from antibiotics when applying conventional (i.e. univariable or one-variable-at-a-time) subgroup analysis. We updated the systematic review and investigated whether multivariable prediction of patient-level prognosis and antibiotic treatment effect may lead to more tailored treatment assignment in adults presenting to primary care with ARS.

METHODS

An IPD-MA of nine double-blind placebo-controlled trials of antibiotic treatment (n=2539) was conducted, with the probability of being cured at 8-15 days as the primary outcome. A logistic mixed effects model was developed to predict the probability of being cured based on demographic characteristics, signs and symptoms, and antibiotic treatment assignment. Predictive performance was quantified based on internal-external cross-validation in terms of calibration and discrimination performance, overall model fit, and the accuracy of individual predictions.

RESULTS

Results indicate that the prognosis with respect to risk of cure could not be reliably predicted (c-statistic 0.58 and Brier score 0.24). Similarly, patient-level treatment effect predictions did not reliably distinguish between those that did and did not benefit from antibiotics (c-for-benefit 0.50).

CONCLUSIONS

In conclusion, multivariable prediction based on patient demographics and common signs and symptoms did not reliably predict the patient-level probability of cure and antibiotic effect in this IPD-MA. Therefore, these characteristics cannot be expected to reliably distinguish those that do and do not benefit from antibiotics in adults presenting to primary care with ARS.

Comparison of causal forest and regression-based approaches to evaluate treatment effect heterogeneity: an application for type 2 diabetes precision medicine

OBJECTIVE

Precision medicine requires reliable identification of variation in patient-level outcomes with different available treatments, often termed treatment effect heterogeneity. We aimed to evaluate the comparative utility of individualized treatment selection strategies based on predicted individual-level treatment effects from a causal forest machine learning algorithm and a penalized regression model.

METHODS

Cohort study characterizing individual-level glucose-lowering response (6 month reduction in HbA1c) in people with type 2 diabetes initiating SGLT2-inhibitor or DPP4-inhibitor therapy. Model development set comprised 1,428 participants in the CANTATA-D and CANTATA-D2 randomised clinical trials of SGLT2-inhibitors versus DPP4-inhibitors. For external validation, calibration of observed versus predicted differences in HbA1c in patient strata defined by size of predicted HbA1c benefit was evaluated in 18,741 patients in UK primary care (Clinical Practice Research Datalink).

RESULTS

Heterogeneity in treatment effects was detected in clinical trial participants with both approaches (proportion predicted to have a benefit on SGLT2-inhibitor therapy over DPP4-inhibitor therapy: causal forest: 98.6%; penalized regression: 81.7%). In validation, calibration was good with penalized regression but sub-optimal with causal forest. A strata with an HbA1c benefit > 10 mmol/mol with SGLT2-inhibitors (3.7% of patients, observed benefit 11.0 mmol/mol [95%CI 8.0-14.0]) was identified using penalized regression but not causal forest, and a much larger strata with an HbA1c benefit 5-10 mmol with SGLT2-inhibitors was identified with penalized regression (regression: 20.9% of patients, observed benefit 7.8 mmol/mol (95%CI 6.7-8.9); causal forest 11.6%, observed benefit 8.7 mmol/mol (95%CI 7.4-10.1).

CONCLUSIONS

Consistent with recent results for outcome prediction with clinical data, when evaluating treatment effect heterogeneity researchers should not rely on causal forest or other similar machine learning algorithms alone, and must compare outputs with standard regression, which in this evaluation was superior.

Conditional Treatment Effect Analysis of Two Infusion Rates for Fluid Challenges in Critically Ill Patients: A Secondary Analysis of Balanced Solution versus Saline in Intensive Care Study (BaSICS) Trial

Rationale: Optimal infusion rate for fluid challenges in critically ill patients is unknown. A large clinical trial comparing two different infusion rates yielded neutral results. Conditional average treatment effect (CATE) assessment may aid in tailoring therapy. Objectives: To estimate CATE in patients enrolled in the BaSICS trial and to assess the effects of receiving CATE model-recommended treatment in terms of hospital mortality. Methods: Post hoc analysis of the BaSICS trial assessing the effect of two infusion rates for the fluid challenge (fast, 999 ml/h, control group; vs. slow, 333 ml/h, intervention group) on hospital mortality. CATE was estimated as the difference in outcome for treatment arms in counterfactuals obtained from a Bayesian model trained in the first half of the trial adjusted for predictors hypothesized to interact with the intervention. The model recommended slow or fast infusion or made no recommendation in the second half. A threshold greater than 0.90 probability of benefit was considered. Results: A total of 10,465 patients were analyzed. The model was trained in 5,230 patients and tested in 5,235 patients. A recommendation could be made in the test set in 19% of patients (14% were recommended the control group and 5% the treatment group); for 81% of patients, no recommendation could be made. Slow infusion was more frequently recommended in cases of planned admissions in younger patients; fast infusion was recommended for older patients with sepsis. Slow infusion rate in the subgroup of patients in the test set in which slow infusion was recommended by the model was associated with an odds ratio of 0.58 (95% credible interval of 0.32-0.90; 0.99 posterior probability of benefit) for hospital mortality. Fast infusion in the subgroup in which the model recommended fast infusion was associated with an odds ratio of 0.72 (credible intervals from 0.54 to 0.91; probability of benefit >0.99). Conclusions: Estimation of CATEs from counterfactual probabilities in data from BaSICS provided additional information on trial data. Agreement between treatment recommendation and actual treatment was associated with lower hospital mortality. Clinical trial registered with clinicaltrials.gov (NCT02875873).

DR-VIDAL - Doubly Robust Variational Information-theoretic Deep Adversarial Learning for Counterfactual Prediction and Treatment Effect Estimation on Real World Data

Determining causal effects of interventions onto outcomes from real-world, observational (non-randomized) data, e.g., treatment repurposing using electronic health records, is challenging due to underlying bias. Causal deep learning has improved over traditional techniques for estimating individualized treatment effects (ITE). We present the Doubly Robust Variational Information-theoretic Deep Adversarial Learning (DR-VIDAL), a novel generative framework that combines two joint models of treatment and outcome, ensuring an unbiased ITE estimation even when one of the two is misspecified. DR-VIDAL integrates: (i) a variational autoencoder (VAE) to factorize confounders into latent variables according to causal assumptions; (ii) an information-theoretic generative adversarial network (Info-GAN) to generate counterfactuals; (iii) a doubly robust block incorporating treatment propensities for outcome predictions. On synthetic and real-world datasets (Infant Health and Development Program, Twin Birth Registry, and National Supported Work Program), DR-VIDAL achieves better performance than other non-generative and generative methods. In conclusion, DR-VIDAL uniquely fuses causal assumptions, VAE, Info-GAN, and doubly robustness into a comprehensive, per- formant framework. Code is available at: https://github.com/Shantanu48114860/DR-VIDAL-AMIA-22 under MIT license.

Estimating individualized treatment effects from randomized controlled trials: a simulation study to compare risk-based approaches

BACKGROUND

Baseline outcome risk can be an important determinant of absolute treatment benefit and has been used in guidelines for "personalizing" medical decisions. We compared easily applicable risk-based methods for optimal prediction of individualized treatment effects.

METHODS

We simulated RCT data using diverse assumptions for the average treatment effect, a baseline prognostic index of risk, the shape of its interaction with treatment (none, linear, quadratic or non-monotonic), and the magnitude of treatment-related harms (none or constant independent of the prognostic index). We predicted absolute benefit using: models with a constant relative treatment effect; stratification in quarters of the prognostic index; models including a linear interaction of treatment with the prognostic index; models including an interaction of treatment with a restricted cubic spline transformation of the prognostic index; an adaptive approach using Akaike's Information Criterion. We evaluated predictive performance using root mean squared error and measures of discrimination and calibration for benefit.

RESULTS

The linear-interaction model displayed optimal or close-to-optimal performance across many simulation scenarios with moderate sample size (N = 4,250; ~ 785 events). The restricted cubic splines model was optimal for strong non-linear deviations from a constant treatment effect, particularly when sample size was larger (N = 17,000). The adaptive approach also required larger sample sizes. These findings were illustrated in the GUSTO-I trial.

CONCLUSIONS

An interaction between baseline risk and treatment assignment should be considered to improve treatment effect predictions.

Predicting psychotic relapse following randomised discontinuation of paliperidone in individuals with schizophrenia or schizoaffective disorder: an individual participant data analysis

BACKGROUND

Predicting relapse for individuals with psychotic disorders is not well established, especially after discontinuation of antipsychotic treatment. We aimed to identify general prognostic factors of relapse for all participants (irrespective of treatment continuation or discontinuation) and specific predictors of relapse for treatment discontinuation, using machine learning.

METHODS

For this individual participant data analysis, we searched the Yale University Open Data Access Project's database for placebo-controlled, randomised antipsychotic discontinuation trials with participants with schizophrenia or schizoaffective disorder (aged ≥18 years). We included studies in which participants were treated with any antipsychotic study drug and randomly assigned to continue the same antipsychotic drug or to discontinue it and receive placebo. We assessed 36 prespecified baseline variables at randomisation to predict time to relapse, using univariate and multivariate proportional hazard regression models (including multivariate treatment group by variable interactions) with machine learning to categorise the variables as general prognostic factors of relapse, specific predictors of relapse, or both.

FINDINGS

We identified 414 trials, of which five trials with 700 participants (304 [43%] women and 396 [57%] men) were eligible for the continuation group and 692 participants (292 [42%] women and 400 [58%] men) were eligible for the discontinuation group (median age 37 [IQR 28-47] years for continuation group and 38 [28-47] years for discontinuation group). Out of the 36 baseline variables, general prognostic factors of increased risk of relapse for all participants were drug-positive urine; paranoid, disorganised, and undifferentiated types of schizophrenia (lower risk for schizoaffective disorder); psychiatric and neurological adverse events; higher severity of akathisia (ie, difficulty or inability to sit still); antipsychotic discontinuation; lower social performance; younger age; lower glomerular filtration rate; benzodiazepine comedication (lower risk for anti-epileptic comedication). Out of the 36 baseline variables, predictors of increased risk specifically after antipsychotic discontinuation were increased prolactin concentration, higher number of hospitalisations, and smoking. Both prognostic factors and predictors with increased risk after discontinuation were oral antipsychotic treatment (lower risk for long-acting injectables), higher last dosage of the antipsychotic study drug, shorter duration of antipsychotic treatment, and higher score on the Clinical Global Impression (CGI) severity scale The predictive performance (concordance index) for participants who were not used to train the model was 0·707 (chance level is 0·5).

INTERPRETATION

Routinely available general prognostic factors of psychotic relapse and predictors specific for treatment discontinuation could be used to support personalised treatment. Abrupt discontinuation of higher dosages of oral antipsychotics, especially for individuals with recurring hospitalisations, higher scores on the CGI severity scale, and increased prolactin concentrations, should be avoided to reduce the risk of relapse.

FUNDING

German Research Foundation and Berlin Institute of Health.

The relation between prediction model performance measures and patient selection outcomes for proton therapy in head and neck cancer

BACKGROUND

Normal-tissue complication probability (NTCP) models predict complication risk in patients receiving radiotherapy, considering radiation dose to healthy tissues, and are used to select patients for proton therapy, based on their expected reduction in risk after proton therapy versus photon radiotherapy (ΔNTCP). Recommended model evaluation measures include area under the receiver operating characteristic curve (AUC), overall calibration (CITL), and calibration slope (CS), whose precise relation to patient selection is still unclear. We investigated how each measure relates to patient selection outcomes.

METHODS

The model validation and consequent patient selection process was simulated within empirical head and neck cancer patient data. By manipulating performance measures independently via model perturbations, the relation between model performance and patient selection was studied.

RESULTS

Small reductions in AUC (-0.02) yielded mean changes in ΔNTCP between 0.9-3.2 %, and single-model patient selection differences between 2-19 %. Deviations (-0.2 or +0.2) in CITL or CS yielded mean changes in ΔNTCP between 0.3-1.4 %, and single-model patient selection differences between 1-10 %.

CONCLUSIONS

Each measure independently impacts ΔNTCP and patient selection and should thus be assessed in a representative sufficiently large external sample. Our suggested practical model selection approach is considering the model with the highest AUC, and recalibrating it if needed.

Heterogeneous treatment effects analysis for social scientists: A review

Social scientists have long been interested in the varying responses to a specific intervention, motivating the enterprise of heterogeneous treatment effects (HTE) analysis. Over the past five decades, the rapid development of HTE methods, from conventional multiplicative interactions in linear models to explorations based on machine learning techniques, has been witnessed. This article presents a systematic review of major HTE methods, including multiplicative interaction modeling, generalized additive modeling, propensity-score-based methods, marginal treatment effect, separate LASSO constraints, causal trees, causal forests, Bayesian additive regression trees, and meta-learners (i.e., the S-learner, T-learner, X-learner, and R-learner). These methods, as described roughly in a chronological order to emphasize methodological developments, are addressed to highlight their respective strengths and limitations. Following an illustrative example, this article reflects on future methodological developments.

The 5-STS is a prognostic factor of sub-acute stroke patients who will not become community walkers at discharge from rehabilitation

BACKGROUND

The recovery of community ambulation is a common concern among individuals after stroke.

OBJECTIVES

(1) To develop a potential readily applicable prognostic model able to correctly discriminate stroke patients who will not become independent community walkers at discharge; (2) To investigate the effects of early reassessment during the first month of treatment on the prediction accuracy of this model.

METHODS

This was a prospective cohort study. A consecutive sample of 80 patients at ≤60 days poststroke were assessed at baseline of outpatient physical rehabilitation and reassessed one month later. Non-functional community ambulation was measured.

RESULTS

Seventy-four patients were followed until discharge. Of these, 47 patients were non-functional community walkers at discharge. A prediction model based on baseline performance in the five repetition sit-to-stand [5-STS] test was able to discriminate those patients of the sample (Area-under-curve = 0.956), and again with data from reassessment (AUC = 0.952). A time of 21 s at baseline was a highly prognostic cut-off point for discrimination (sensitivity = 87.2% and 85.1%). The combined use of baseline and reassessment data improved sensitivity (98.1%)CONCLUSION:Early findings of the 5-STS among stroke patients is an independent prognostic factor associated with independent community walking at discharge. It could discriminate individuals who will not become community walkers at discharge.

A Causal Framework for Making Individualized Treatment Decisions in Oncology

Simple Summary

Physicians routinely make individualized treatment decisions by accounting for the joint effects of patient prognostic covariates and treatments on clinical outcomes. Ideally, this is performed using historical randomized clinical trial (RCT) data. Randomization ensures that unbiased estimates of causal treatment effect parameters can be obtained from the historical RCT data and used to predict each new patient’s outcome based on the joint effect of their baseline covariates and each treatment being considered. However, this process becomes problematic if a patient seen in the clinic is very different from the patients who were enrolled in the RCT. That is, if a new patient does not satisfy the entry criteria of the RCT, then the patient does not belong to the population represented by the patients who were studied in the RCT. In such settings, it still may be possible to utilize the RCT data to help choose a new patient’s treatment. This may be achieved by combining the RCT data with data from other clinical trials, or possibly preclinical experiments, and using the combined dataset to predict the patient’s expected outcome for each treatment being considered. In such settings, combining data from multiple sources in a way that is statistically reliable is not entirely straightforward, and correctly identifying and estimating the effects of treatments and patient covariates on clinical outcomes can be complex. Causal diagrams provide a rational basis to guide this process. The first step is to construct a causal diagram that reflects the plausible relationships between treatment variables, patient covariates, and clinical outcomes. If the diagram is correct, it can be used to determine what additional data may be needed, how to combine data from multiple sources, how to formulate a statistical model for clinical outcomes as a function of treatment and covariates, and how to compute an unbiased treatment effect estimate for each new patient. We use adjuvant therapy of renal cell carcinoma to illustrate how causal diagrams may be used to guide these steps.

Abstract

We discuss how causal diagrams can be used by clinicians to make better individualized treatment decisions. Causal diagrams can distinguish between settings where clinical decisions can rely on a conventional additive regression model fit to data from a historical randomized clinical trial (RCT) to estimate treatment effects and settings where a different approach is needed. This may be because a new patient does not meet the RCT’s entry criteria, or a treatment’s effect is modified by biomarkers or other variables that act as mediators between treatment and outcome. In some settings, the problem can be addressed simply by including treatment–covariate interaction terms in the statistical regression model used to analyze the RCT dataset. However, if the RCT entry criteria exclude a new patient seen in the clinic, it may be necessary to combine the RCT data with external data from other RCTs, single-arm trials, or preclinical experiments evaluating biological treatment effects. For example, external data may show that treatment effects differ between histological subgroups not recorded in an RCT. A causal diagram may be used to decide whether external observational or experimental data should be obtained and combined with RCT data to compute statistical estimates for making individualized treatment decisions. We use adjuvant treatment of renal cell carcinoma as our motivating example to illustrate how to construct causal diagrams and apply them to guide clinical decisions.

Using individual participant data to improve network meta-analysis projects

A network meta-analysis combines the evidence from existing randomised trials about the comparative efficacy of multiple treatments. It allows direct and indirect evidence about each comparison to be included in the same analysis, and provides a coherent framework to compare and rank treatments. A traditional network meta-analysis uses aggregate data (eg, treatment effect estimates and standard errors) obtained from publications or trial investigators. An alternative approach is to obtain, check, harmonise and meta-analyse the individual participant data (IPD) from each trial. In this article, we describe potential advantages of IPD for network meta-analysis projects, emphasising five key benefits: (1) improving the quality and scope of information available for inclusion in the meta-analysis, (2) examining and plotting distributions of covariates across trials (eg, for potential effect modifiers), (3) standardising and improving the analysis of each trial, (4) adjusting for prognostic factors to allow a network meta-analysis of conditional treatment effects and (5) including treatment-covariate interactions (effect modifiers) to allow relative treatment effects to vary by participant-level covariate values (eg, age, baseline depression score). A running theme of all these benefits is that they help examine and reduce heterogeneity (differences in the true treatment effect between trials) and inconsistency (differences in the true treatment effect between direct and indirect evidence) in the network. As a consequence, an IPD network meta-analysis has the potential for more precise, reliable and informative results for clinical practice and even allows treatment comparisons to be made for individual patients and targeted populations conditional on their particular characteristics.

Clinical prediction models for mortality in patients with covid-19: external validation and individual participant data meta-analysis

OBJECTIVE

To externally validate various prognostic models and scoring rules for predicting short term mortality in patients admitted to hospital for covid-19.

DESIGN

Two stage individual participant data meta-analysis.

SETTING

Secondary and tertiary care.

PARTICIPANTS

46 914 patients across 18 countries, admitted to a hospital with polymerase chain reaction confirmed covid-19 from November 2019 to April 2021.

DATA SOURCES

Multiple (clustered) cohorts in Brazil, Belgium, China, Czech Republic, Egypt, France, Iran, Israel, Italy, Mexico, Netherlands, Portugal, Russia, Saudi Arabia, Spain, Sweden, United Kingdom, and United States previously identified by a living systematic review of covid-19 prediction models published in The BMJ, and through PROSPERO, reference checking, and expert knowledge.

MODEL SELECTION AND ELIGIBILITY CRITERIA

Prognostic models identified by the living systematic review and through contacting experts. A priori models were excluded that had a high risk of bias in the participant domain of PROBAST (prediction model study risk of bias assessment tool) or for which the applicability was deemed poor.

METHODS

Eight prognostic models with diverse predictors were identified and validated. A two stage individual participant data meta-analysis was performed of the estimated model concordance (C) statistic, calibration slope, calibration-in-the-large, and observed to expected ratio (O:E) across the included clusters.

MAIN OUTCOME MEASURES

30 day mortality or in-hospital mortality.

RESULTS

Datasets included 27 clusters from 18 different countries and contained data on 46 914patients. The pooled estimates ranged from 0.67 to 0.80 (C statistic), 0.22 to 1.22 (calibration slope), and 0.18 to 2.59 (O:E ratio) and were prone to substantial between study heterogeneity. The 4C Mortality Score by Knight et al (pooled C statistic 0.80, 95% confidence interval 0.75 to 0.84, 95% prediction interval 0.72 to 0.86) and clinical model by Wang et al (0.77, 0.73 to 0.80, 0.63 to 0.87) had the highest discriminative ability. On average, 29% fewer deaths were observed than predicted by the 4C Mortality Score (pooled O:E 0.71, 95% confidence interval 0.45 to 1.11, 95% prediction interval 0.21 to 2.39), 35% fewer than predicted by the Wang clinical model (0.65, 0.52 to 0.82, 0.23 to 1.89), and 4% fewer than predicted by Xie et al's model (0.96, 0.59 to 1.55, 0.21 to 4.28).

CONCLUSION

The prognostic value of the included models varied greatly between the data sources. Although the Knight 4C Mortality Score and Wang clinical model appeared most promising, recalibration (intercept and slope updates) is needed before implementation in routine care.

Evolving Role of Adjuvant Systemic Therapy for Kidney and Urothelial Cancers

The role of adjuvant therapy in renal cell carcinoma and urothelial carcinoma is rapidly evolving. To date, the U.S. Food and Drug Administration has approved sunitinib and pembrolizumab in the adjuvant setting for renal cell carcinoma and nivolumab for urothelial carcinoma based on disease-free survival benefit. The U.S. Food and Drug Administration held a joint workshop with the National Cancer Institute and the Society of Urologic Oncology in 2017 to harmonize design elements, including eligibility and radiologic assessments across adjuvant trials in renal cell carcinoma and urothelial carcinoma. Considerations from the discussion at these workshops led the U.S. Food and Drug Administration to draft guidances to help inform subsequent adjuvant trial design for renal cell carcinoma and urothelial carcinoma. Patient-centered decision-making is crucial when determining therapeutic choices in the adjuvant setting; utility functions can be used to help quantify each patient's goals, values, and risk/benefit trade-offs to ensure that the decision regarding adjuvant therapy is informed by their preferences and the evolving outcomes data.

Individual differences in the effects of the ACTION-PAC intervention: an application of personalized medicine in the prevention and treatment of obesity

There is an increased interest in the use of personalized medicine approaches in the prevention or treatment of obesity, however, few studies have used these approaches to identify individual differences in treatment effects. The current study demonstrates the use of the predicted individual treatment effects framework to test for individual differences in the effects of the ACTION-PAC intervention, which targeted the treatment and prevention of obesity in a high school setting. We show how methods for personalized medicine can be used to test for significant individual differences in responses to an intervention and we discuss the potential and limitations of these methods. In our example, 25% of students in the preventive intervention, were predicted to have their BMI z-score reduced by 0.39 or greater, while at other end of the spectrum, 25% were predicted to have their BMI z-score increased by 0.09 or more. In this paper, we demonstrate and discuss the process of using methods for personalized medicine with interventions targeting adiposity and discuss the lessons learned from this application. Ultimately, these methods have the potential to be useful for clinicians and clients in choosing between treatment options, however they are limited in their ability to help researchers understand the mechanisms underlying these predictions.