Predictive approaches to heterogeneous treatment effects: a scoping review

Precision medicine evaluation of heterogeneity of treatment effect for a time-to-event outcome with application in a trial of Initial treatment for people living with HIV

BackgroundEvaluation of heterogeneity of treatment effect among participants in large randomized clinical trials may provide insights as to the value of individualizing clinical decisions. The effect modeling approach to predictive heterogeneity of treatment effect analysis offers a promising framework for heterogeneity of treatment effect estimation by simultaneously considering multiple patient characteristics and their interactions with treatment to predict differences in outcomes between randomized treatments. However, its implementation in clinical research remains limited and so we provide a detailed example of its application in a randomized trial that compared raltegravir-based vs darunavir/ritonavir-based therapy as initial antiretroviral treatments for people living with HIV.MethodsThe heterogeneity of treatment effect analysis used a two-step procedure, in which a working proportional hazards model was first selected to construct an index score for ranking the treatment difference for individuals, and then a second calibration step used a non-parametric kernel approach to estimate the true treatment difference for participants with similar index scores. Sensitivity and supplemental analyses were conducted to evaluate the robustness of the results. We further explored the impact of covariates on heterogeneity of treatment effect and the choice between treatments.ResultsThe heterogeneity of treatment effect analysis showed that while there is a clear trend favoring raltegravir-based therapy over darunavir/ritonavir-based therapy for the vast majority of the target population, there were a small subset of patients, characterized by more advanced HIV disease status, for whom the choice between the two treatments might be equivocal.ConclusionsThrough this example, we illustrate how an exploratory heterogeneity of treatment effect analysis might provide further insights into the comparative efficacy of treatments evaluated in a randomized trial. We also highlight some of the issues in implementing and interpreting effect modeling analyses in randomized trials.

Did a digital quality of life (QOL) assessment and practice support system in home health care improve the QOL of older adults living with life-limiting conditions and of their family caregivers? A mixed-methods pragmatic randomized controlled trial

We aimed to improve the quality of life (QOL) of homecare patients (≥ 55 years of age) who had chronic life-limiting conditions and that of their family caregivers by making QOL assessment data available via a digital QOL and practice support system (QPSS). We hypothesized that access to QPSS data in home health care would result in improved QOL for patients or their family caregivers. We further sought to understand how to integrate the use of QOL information into home health care. Our mixed-methods study, including a pragmatic randomized controlled trial (PrCT; registered at ClinicalTrials.gov #NCT02940951), was conducted with nine home healthcare teams in Canadian urban areas. The qualitative research included focus groups and interviews with home healthcare teams (N = 118) to determine how to integrate the QPSS into their practice. Participating homecare patients were assigned to an intervention group (N = 166), where home healthcare teams had access to patients' and their family caregivers' QOL data via the QPSS, or a usual care group (N = 165). Where possible, one family caregiver per patient was recruited (intervention N = 62; usual care N = 51). Primary outcomes, measured every two months for one year, were patients' and family caregivers' QOL trajectories. Longitudinal structural equation models were used to compare the trajectories. The home healthcare teams preferred to have QOL scores presented as tables and graphs, with family caregivers' data linked to each patient. Despite the enthusiasm expressed by the home healthcare teams, and efforts to satisfy their preferences, they infrequently accessed the QOL information. While we observed substantial individual-level variability in patients' and family caregivers' QOL trajectories, the average trajectories for the PrCT groups were similar. Making QOL assessment data available via a digital platform may not be sufficient to achieve measurable improvements for patients and family caregivers.

Characterizing Treatment Effect Heterogeneity Using Real-World Data

Characterizing heterogeneity of treatment effects (HTE) is a fundamental goal of pharmacoepidemiology, addressing why medications work differently across patient populations. This paper reviews state-of-the-art methods for studying HTE using real-world data (RWD), which offer larger study sizes and more diverse patient populations compared to randomized clinical trials. The paper first defines HTE and discusses its measurement. It then examines three leading approaches to studying HTE: subgroup analysis, disease risk score (DRS) methods, and effect modeling methods. Subgroup analyses offer simplicity, transparency, and provide insights into drug mechanisms. However, they face difficulties in resolving which subgroup or combination of characteristics should be the basis for clinical decision making when multiple effect modifiers are present. DRS methods address some of these limitations by incorporating multiple patient characteristics into a summary score of outcome risk but may obscure insights into mechanisms. Effect modeling methods directly predict individual treatment effects, offering potential for precise HTE characterization, but are prone to model misspecification and may not provide mechanistic insights. The methods each have tradeoffs. Subgroup analysis is straightforward but can lead to spurious associations and does not account for multiple characteristics at once. DRS methods are relatively simple to implement and clinically useful, but may not completely describe HTE or provide mechanistic insight. Effect modeling approaches have great potential for characterizing HTE but are still being developed. Understanding HTE is essential for personalizing treatment strategies to improve patient outcomes. Researchers must weigh the strengths and limitations of each approach when using RWD to study HTE.

Heterogeneity of Survival Benefit Conferred By Letermovir

Variation in treatment effects based on individual patient characteristics-known as treatment effect heterogeneity or effect modification-has recently gained significant attention. A previous clinical trial and its post hoc analysis suggested that letermovir (LTV) may reduce mortality more in some patients than in others. We hypothesized that the survival benefit of LTV differs according to each patient's specific characteristics. This study aimed to identify patient characteristics that are associated with significant survival benefits from LTV. Patients who underwent transplantation between 2018 and 2022 were randomly divided into training (n = 5779) and validation groups (n = 2865). We developed two models: one using a proportional hazards model with interaction terms (PI), and another using a modern machine learning (ML) approach to detect heterogeneity in the survival benefit-specifically, to identify patient characteristics associated with greater benefit from LTV. In our cohort, 60% of patients received LTV as prophylaxis. In the training cohort, the final PI model, using additive interactions, identified advanced age (≥60), high comorbidities (HCT-CI ≥3), umbilical cord blood (UCB), and haploidentical HCT with post cyclophosphamide (PTCy Haplo) as highly beneficial factors. Meanwhile, the ML model, using a causal forest algorithm, classified the top 60% of patients based on the estimated individual treatment effect as the high benefit group. In the validation group, 67.1% and 59.9% of patients were considered to be high benefit by the PI and ML models, respectively. The absolute difference in 6-month NRM (LTV versus no LTV) in the high benefit group (PI model: 9.8% versus 16.3%; ML model: 11.3% versus 16.3%) was greater than that in the low benefit group (PI model: 4.3% versus 6.9%; ML model: 4.1% versus 6.2%). Most patients (>80%) with advanced age, high comorbidities, or UCB were classified as high benefit by the ML model, supporting the robustness of the PI model. Our models successfully identified patients who could be expected to experience lower NRM with LTV prophylaxis, underscoring the importance of personalized medicine.

Measuring the Performance of Survival Models to Personalize Treatment Choices

Various statistical and machine learning algorithms can be used to predict treatment effects at the patient level using data from randomized clinical trials (RCTs). Such predictions can facilitate individualized treatment decisions. Recently, a range of methods and metrics were developed for assessing the accuracy of such predictions. Here, we extend these methods, focusing on the case of survival (time-to-event) outcomes. We start by providing alternative definitions of the participant-level treatment benefit; subsequently, we summarize existing and propose new measures for assessing the performance of models estimating participant-level treatment benefits. We explore metrics assessing discrimination and calibration for benefit and decision accuracy. These measures can be used to assess the performance of statistical as well as machine learning models and can be useful during model development (i.e., for model selection or for internal validation) or when testing a model in new settings (i.e., in an external validation). We illustrate methods using simulated data and real data from the OPERAM trial, an RCT in multimorbid older people, which randomized participants to either standard care or a pharmacotherapy optimization intervention. We provide R codes for implementing all models and measures.

Qini Curves for Potential Impact Assessment of Risk Predictive Models Informing Intervention Policies

OBJECTIVES

Predictive models in medicine help make decisions about which individual to treat with a given therapeutic or preventive intervention. Before being tested in large field studies and recommended for clinical adoption, it is important to evaluate not only their statistical accuracy but also the impact they may have when used to inform health intervention policies. We aim to provide simple methods for the potential impact assessment of health intervention policies based on predictive models.

METHODS

We propose an analytic framework based on Qini curves wherein prediction-based policies are analyzed on 2 impact endpoints: (1) the fraction of the population that would be selected for the intervention (coverage) and (2) the effect on the clinical outcomes of interest (disutility). The drivers of values are the disease prevalence, the predictive performance of the model, and the effectiveness of the intervention.

RESULTS

We present simple formulas for calculating coverage and disutility from either observational or randomized controlled data. We illustrate possible value measures arising from geometrical properties on the Qini plane: delta coverage and disutility, number needed to treat, and integrated difference between Qini curves. We show the applicability of the Qini analysis by providing examples about the prevention of falls in older adults and prevention of secondary cardiovascular events with pioglitazone.

CONCLUSIONS

Coverage and disutility capture key value components of prediction-based policies. The method can be used for comparing models or tuning risk thresholds for managing trade-offs between conflicting objectives (eg, clinical benefits, side effects, and healthcare resources).

Individualized multi-treatment response curves estimation using RBF-net with shared neurons

Heterogeneous treatment effect estimation is an important problem in precision medicine. Specific interests lie in identifying the differential effect of different treatments based on some external covariates. We propose a novel non-parametric treatment effect estimation method in a multi-treatment setting. Our non-parametric modeling of the response curves relies on radial basis function-nets with shared hidden neurons. Our model thus facilitates modeling commonality among the treatment outcomes. The estimation and inference schemes are developed under a Bayesian framework using thresholded best linear projections and implemented via an efficient Markov chain Monte Carlo algorithm, appropriately accommodating uncertainty in all aspects of the analysis. The numerical performance of the method is demonstrated through simulation experiments. Applying our proposed method to MIMIC data, we obtain several interesting findings related to the impact of different treatment strategies on the length of intensive care unit stay and 12-h Sequential Organ Failure Assessment score for sepsis patients who are home-discharged.

Multivariable prognostic prediction of efficacy and safety outcomes and response to fingolimod in people with relapsing-remitting multiple sclerosis

BACKGROUND

The individual treatment response in people with relapsing-remitting multiple sclerosis (RRMS) remain unpredictable. In order to support medical decisions, we aimed to predict response to fingolimod compared to placebo, by developing and validating prognostic multivariable models.

METHODS

We included two-year follow-up from intention-to-treat populations of two multi-country placebo-controlled randomized controlled trials (RCT) of daily fingolimod 0.5 mg. The data was accessed via ClinicalStudyDataRequest.com (Proposal Number: 11223) The RCTs were in adult RRMS patients with active disease. We used four Cox proportional hazards based penalized (elastic net and grouped lasso) and tree methods (transformation tree and forest) to predict time-to relapse and other relevant efficacy and safety endpoints in data from the RCT FREEDOMS. Treatment arm, 80 baseline variables and their interaction with treatment were considered as candidate predictors in the models. A nested cross-validation scheme ensured independent tuning parameter optimization and internal model performance evaluation. The generalizability of the models with the highest cross-validated time-dependent area under the receiver operating curve (AUC) was further evaluated in terms of discrimination (AUC), calibration (plots, intercept, slope), clinical utility (decision curve analysis), and treatment response plots by external validation in data from the RCT FREEDOMS II.

RESULTS

The best performing model predicting relapse risk (331 events) in the development sample (n=843) was an elastic net regression with main terms for four predictors alongside treatment: EDSS score, volume of Gadolinium enhanced T1 lesions, number of relapses in the last 2 years, and number of prior MS treatments. In external validation (n=713), it had an AUC of 0.68 (95% CI 0.63-0.72), but the predictions were overestimating the actual risk (358 events) with a calibration-in-the-large of -0.17 (-0.3 - -0.04) and a slope of 1.06 (0.78-1.35). Almost no heterogeneity (variability 0.001) was detected in the predicted relapse risk change in response to fingolimod. FREEDOMS II participants were predicted to have 0.21 to 0.31 absolute relapse risk reduction with fingolimod compared to placebo. The selected model predicting new or enlarging T2 magnetic resonance imaging (MRI) lesions had an AUC of 0.74 (0.70-0.78), moderate calibration, but no treatment response variability. The final model predicting confirmed disability progression had an AUC of 0.59 (0.54-0.64) and the predicted treatment response heterogeneity was not significant. The overall safety outcome could not be predicted with sufficient discrimination. However, the final model predicting infections or neoplasms had an AUC of 0.69 (0.63-0.74) and non-significant treatment response heterogeneity. For the efficacy outcomes, important predictors were related to (para)clinical disease activity or disability. Unexpected influential predictors included concomitant disorders.

CONCLUSION

Relapse and new or enlarging T2 MRI lesions were moderately predictable in an independent sample with the developed prognostic models. Fingolimod was expected to decrease the risk of these events for all patients, with no predictable heterogeneity. Disability and safety outcomes could not be well-predicted and it is yet unresolved whether the change in their risk as response to fingolimod is heterogeneous or not.

A 2-stage model of heterogenous treatment effects for brain atrophy in multiple sclerosis utilizing the MS PATHS research network

BACKGROUND

Two-stage models of heterogenous treatment effects (HTE) may advance personalized medicine in multiple sclerosis (MS). Brain atrophy is a relatively objective outcome measure that has strong relationships to MS prognosis and treatment effects and is enabled by standardized MRI.

OBJECTIVES

To predict brain atrophy outcomes for patients initiating disease-modifying therapies (DMT) with different efficacies, considering the patients' baseline brain atrophy risk measured via brain parenchymal fraction (BPF).

METHODS

Analyses included patients enrolled in the Multiple Sclerosis Partners Advancing Technology and Health Solutions (MS PATHS) network who started DMT and had complete baseline data and ≥ 6-month brain MRI follow-up. All brain MRIs were acquired using standardized acquisition sequences on Siemens 3T scanners. BPF change risk was derived by linear mixed effects models using baseline covariates. Model performance was assessed by predicted versus actual BPF change R2. Propensity score (PS) weighting was used to balance covariates between groups defined by DMT efficacy (high: natalizumab, alemtuzumab, ocrelizumab, and rituximab; moderate: dimethyl fumarate, fingolimod, and cladribine; low: teriflunomide, interferons, and glatiramer acetate). HTE models predicting 1 year change in BPF were built using a weighted linear mixed effects model with low-efficacy DMT as the reference.

RESULTS

Analyses included 581 high-, 183 moderate-, and 106 low-efficacy DMT-treated patients. The mean and median number of brain MRI observations per treatment period were 2.9 and 3.0, respectively. Risk model performance R2=0.55. After PS weighting, covariate standardized mean differences were <10 %, indicating excellent balance across measured variables. Changes in BPF between baseline and follow-up were found to be statistically significant (p < 0.001), suggesting a pathological change. Patients with low brain atrophy risk had a similar outcome regardless of DMT selection. In patients with high brain atrophy risk, high- and moderate-efficacy DMTs performed similarly, while a 2-fold worse BPF change was predicted for patients selecting low-efficacy DMTs (p < 0.001). Similar results were observed in a sensitivity analysis adjusting for pseudoatrophy effects in a sub-population of patients treated with natalizumab.

CONCLUSIONS

The relative benefit of selecting higher efficacy treatments may vary depending on patients' baseline brain atrophy risk. Poor outcomes are predicted in individuals with high baseline risk who are treated with low-efficacy DMTs.

Evaluating individualized treatment effect predictions: A model-based perspective on discrimination and calibration assessment

In recent years, there has been a growing interest in the prediction of individualized treatment effects. While there is a rapidly growing literature on the development of such models, there is little literature on the evaluation of their performance. In this paper, we aim to facilitate the validation of prediction models for individualized treatment effects. The estimands of interest are defined based on the potential outcomes framework, which facilitates a comparison of existing and novel measures. In particular, we examine existing measures of discrimination for benefit (variations of the c-for-benefit), and propose model-based extensions to the treatment effect setting for discrimination and calibration metrics that have a strong basis in outcome risk prediction. The main focus is on randomized trial data with binary endpoints and on models that provide individualized treatment effect predictions and potential outcome predictions. We use simulated data to provide insight into the characteristics of the examined discrimination and calibration statistics under consideration, and further illustrate all methods in a trial of acute ischemic stroke treatment. The results show that the proposed model-based statistics had the best characteristics in terms of bias and accuracy. While resampling methods adjusted for the optimism of performance estimates in the development data, they had a high variance across replications that limited their accuracy. Therefore, individualized treatment effect models are best validated in independent data. To aid implementation, a software implementation of the proposed methods was made available in R.

Emerging Analytical Approaches for Personalized Medicine Using Machine Learning In Pediatric and Congenital Heart Disease

Precision and personalized medicine, the process by which patient management is tailored to individual circumstances, are now terms that are familiar to cardiologists, despite it still being an emerging field. Although precision medicine relies most often on the underlying biology and pathophysiology of a patient's condition, personalized medicine relies on digital biomarkers generated through algorithms. Given the complexity of the underlying data, these digital biomarkers are most often generated through machine-learning algorithms. There are a number of analytic considerations regarding the creation of digital biomarkers that are discussed in this review, including data preprocessing, time dependency and gating, dimensionality reduction, and novel methods, both in the realm of supervised and unsupervised machine learning. Some of these considerations, such as sample size requirements and measurements of model performance, are particularly challenging in small and heterogeneous populations with rare outcomes such as children with congenital heart disease. Finally, we review analytic considerations for the deployment of digital biomarkers in clinical settings, including the emerging field of clinical artificial intelligence (AI) operations, computational needs for deployment, efforts to increase the explainability of AI, algorithmic drift, and the needs for distributed surveillance and federated learning. We conclude this review by discussing a recent simulation study that shows that, despite these analytic challenges and complications, the use of digital biomarkers in managing clinical care might have substantial benefits regarding individual patient outcomes.

Toward Precision in Critical Care Research: Methods for Observational and Interventional Studies

Critical care trials evaluate the effect of interventions in patients with diverse personal histories and causes of illness, often under the umbrella of heterogeneous clinical syndromes, such as sepsis or acute respiratory distress syndrome. Given this variation, it is reasonable to expect that the effect of treatment on outcomes may differ for individuals with variable characteristics. However, in randomized controlled trials, efficacy is typically assessed by the average treatment effect (ATE), which quantifies the average effect of the intervention on the outcome in the study population. Importantly, the ATE may hide variations of the treatment's effect on a clinical outcome across levels of patient characteristics, which may erroneously lead to the conclusion that an intervention does not work overall when it may in fact benefit certain patients. In this review, we describe methodological approaches for assessing heterogeneity of treatment effect (HTE), including expert-derived subgrouping, data-driven subgrouping, baseline risk modeling, treatment effect modeling, and individual treatment rule estimation. Next, we outline how insights from HTE analyses can be incorporated into the design of clinical trials. Finally, we propose a research agenda for advancing the field and bringing HTE approaches to the bedside.

Estimating and presenting hazard ratios and absolute risks from a Cox model with complex nonlinear interactions

Interaction analysis is a critical component of clinical and public health research and represents a key topic in precision health and medicine. In applied settings, however, interaction assessment is usually limited to the test of a product term in a regression model and to the presentation of results stratified by levels of additional covariates. Stratification of results often relies on categorizing or making linearity assumptions for continuous covariates, with substantial loss of precision and of relevant information. In time-to-event analysis, moreover, interaction assessment is often limited to the multiplicative hazard scale by inclusion of a product term in a Cox regression model, disregarding the clinically relevant information that is captured by the absolute risk scale. In this paper we present a user-friendly procedure, based on the prediction of individual absolute risks from the Cox model, for the estimation and presentation of interactive effects on both the multiplicative and additive scales in survival analysis. We describe how to flexibly incorporate interactions with continuous covariates, which potentially operate in a nonlinear fashion, provide software for replicating our procedure, and discuss different approaches to deriving CIs. The presented approach will allow clinical and public health researchers to assess complex relationships between multiple covariates as they relate to a clinical endpoint, and to provide a more intuitive and precise depiction of the results in applied research papers focusing on interaction and effect stratification.

Computational simulation of the potential improvement in clinical outcomes of cardiovascular diseases with the use of a personalized predictive medicine approach

IMPORTANCE AND OBJECTIVES

The current medical paradigm of evidence-based medicine relies on clinical guidelines derived from randomized clinical trials (RCTs), but these guidelines often overlook individual variations in treatment effects. Approaches have been proposed to develop models predicting the effects of individualized management, such as predictive allocation, individualizing treatment allocation. It is currently unknown whether widespread implementation of predictive allocation could result in better population-level outcomes over guideline-based therapy. We sought to simulate the potential effect of predictive allocation using data from previously conducted RCTs.

METHODS AND RESULTS

Data from 3 RCTs (positive trial, negative trial, trial stopped for futility) in pediatric cardiology were used in a computational simulation study to quantify the potential benefits of a personalized approach based on predictive allocation. Outcomes were compared when using a universal approach vs predictive allocation where each patient was allocated to the treatment associated with the lowest predicted probability of negative outcome. Compared to results from RCTs, predictive allocation yielded absolute risk reductions of 13.8% (95% confidence interval [CI] -1.9 to 29.5), 13.9% (95% CI 4.5-23.2), and 15.6% (95% CI 1.5-29.6), respectively, corresponding to a number needed to treat of 7.3, 7.2, and 6.4. The net benefit of predictive allocation was directly proportional to the performance of the prediction models and disappeared as model performance degraded below an area under the curve of 0.55.

DISCUSSION

These findings highlight that predictive allocation could result in improved group-level outcomes, particularly when highly predictive models are available. These findings will need to be confirmed in simulations of other trials with varying conditions and eventually in RCTs of predictive vs guideline-based treatment allocation.

Combining randomized and non-randomized data to predict heterogeneous effects of competing treatments

Some patients benefit from a treatment while others may do so less or do not benefit at all. We have previously developed a two-stage network meta-regression prediction model that synthesized randomized trials and evaluates how treatment effects vary across patient characteristics. In this article, we extended this model to combine different sources of types in different formats: aggregate data (AD) and individual participant data (IPD) from randomized and non-randomized evidence. In the first stage, a prognostic model is developed to predict the baseline risk of the outcome using a large cohort study. In the second stage, we recalibrated this prognostic model to improve our predictions for patients enrolled in randomized trials. In the third stage, we used the baseline risk as effect modifier in a network meta-regression model combining AD, IPD randomized clinical trial to estimate heterogeneous treatment effects. We illustrated the approach in the re-analysis of a network of studies comparing three drugs for relapsing-remitting multiple sclerosis. Several patient characteristics influence the baseline risk of relapse, which in turn modifies the effect of the drugs. The proposed model makes personalized predictions for health outcomes under several treatment options and encompasses all relevant randomized and non-randomized evidence.

Do machine learning methods lead to similar individualized treatment rules? A comparison study on real data

Identifying patients who benefit from a treatment is a key aspect of personalized medicine, which allows the development of individualized treatment rules (ITRs). Many machine learning methods have been proposed to create such rules. However, to what extent the methods lead to similar ITRs, that is, recommending the same treatment for the same individuals is unclear. In this work, we compared 22 of the most common approaches in two randomized control trials. Two classes of methods can be distinguished. The first class of methods relies on predicting individualized treatment effects from which an ITR is derived by recommending the treatment evaluated to the individuals with a predicted benefit. In the second class, methods directly estimate the ITR without estimating individualized treatment effects. For each trial, the performance of ITRs was assessed by various metrics, and the pairwise agreement between all ITRs was also calculated. Results showed that the ITRs obtained via the different methods generally had considerable disagreements regarding the patients to be treated. A better concordance was found among akin methods. Overall, when evaluating the performance of ITRs in a validation sample, all methods produced ITRs with limited performance, suggesting a high potential for optimism. For non-parametric methods, this optimism was likely due to overfitting. The different methods do not lead to similar ITRs and are therefore not interchangeable. The choice of the method strongly influences for which patients a certain treatment is recommended, drawing some concerns about their practical use.

Assessing the properties of patient-specific treatment effect estimates from causal forest algorithms under essential heterogeneity

BACKGROUND

Treatment variation from observational data has been used to estimate patient-specific treatment effects. Causal Forest Algorithms (CFAs) developed for this task have unknown properties when treatment effect heterogeneity from unmeasured patient factors influences treatment choice - essential heterogeneity.

METHODS

We simulated eleven populations with identical treatment effect distributions based on patient factors. The populations varied in the extent that treatment effect heterogeneity influenced treatment choice. We used the generalized random forest application (CFA-GRF) to estimate patient-specific treatment effects for each population. Average differences between true and estimated effects for patient subsets were evaluated.

RESULTS

CFA-GRF performed well across the population when treatment effect heterogeneity did not influence treatment choice. Under essential heterogeneity, however, CFA-GRF yielded treatment effect estimates that reflected true treatment effects only for treated patients and were on average greater than true treatment effects for untreated patients.

CONCLUSIONS

Patient-specific estimates produced by CFAs are sensitive to why patients in real-world practice make different treatment choices. Researchers using CFAs should develop conceptual frameworks of treatment choice prior to estimation to guide estimate interpretation ex post.

Use of conditional estimates of effect in cancer epidemiology: An application to lung cancer treatment

BACKGROUND

In oncology clinical trials, there is the assumption that randomization sufficiently balances confounding covariates and therefore average treatment effects are usually reported. This paper explores the wider benefits provided by conditioning on covariates for reasons other than mitigation of confounding.

METHODS

We reanalyzed the data from primary randomized controlled trials listed in two meta-analyses to explore the significance of conditioning on smoking status in terms of the effect magnitude of treatment on progression free survival in non-small cell lung cancer.

RESULTS

The reanalysis revealed that conditioning on smoking status using sub-group analyses provided the closest empiric estimate of individual treatment effect based on smoking status and significantly reduced the heterogeneity of treatment effect observed across studies. In addition, smoking status was determined to be a modifier of the effect of treatment.

CONCLUSION

Conditioning on prognostic covariates in randomized trials in oncology helps generate the closest empiric estimates of individual treatment benefit, addresses heterogeneity due to varying covariate distributions across trials and facilitates future decision making as well as evidence synthesis. Conditioning using sub-group analyses also allows examination for effect modification in meta-analysis.

Machine learning in precision diabetes care and cardiovascular risk prediction

Artificial intelligence and machine learning are driving a paradigm shift in medicine, promising data-driven, personalized solutions for managing diabetes and the excess cardiovascular risk it poses. In this comprehensive review of machine learning applications in the care of patients with diabetes at increased cardiovascular risk, we offer a broad overview of various data-driven methods and how they may be leveraged in developing predictive models for personalized care. We review existing as well as expected artificial intelligence solutions in the context of diagnosis, prognostication, phenotyping, and treatment of diabetes and its cardiovascular complications. In addition to discussing the key properties of such models that enable their successful application in complex risk prediction, we define challenges that arise from their misuse and the role of methodological standards in overcoming these limitations. We also identify key issues in equity and bias mitigation in healthcare and discuss how the current regulatory framework should ensure the efficacy and safety of medical artificial intelligence products in transforming cardiovascular care and outcomes in diabetes.

Performance metrics for models designed to predict treatment effect

BACKGROUND

Measuring the performance of models that predict individualized treatment effect is challenging because the outcomes of two alternative treatments are inherently unobservable in one patient. The C-for-benefit was proposed to measure discriminative ability. However, measures of calibration and overall performance are still lacking. We aimed to propose metrics of calibration and overall performance for models predicting treatment effect in randomized clinical trials (RCTs).

METHODS

Similar to the previously proposed C-for-benefit, we defined observed pairwise treatment effect as the difference between outcomes in pairs of matched patients with different treatment assignment. We match each untreated patient with the nearest treated patient based on the Mahalanobis distance between patient characteristics. Then, we define the Eavg-for-benefit, E50-for-benefit, and E90-for-benefit as the average, median, and 90th quantile of the absolute distance between the predicted pairwise treatment effects and local-regression-smoothed observed pairwise treatment effects. Furthermore, we define the cross-entropy-for-benefit and Brier-for-benefit as the logarithmic and average squared distance between predicted and observed pairwise treatment effects. In a simulation study, the metric values of deliberately "perturbed models" were compared to those of the data-generating model, i.e., "optimal model". To illustrate these performance metrics, different modeling approaches for predicting treatment effect are applied to the data of the Diabetes Prevention Program: 1) a risk modelling approach with restricted cubic splines; 2) an effect modelling approach including penalized treatment interactions; and 3) the causal forest.

RESULTS

As desired, performance metric values of "perturbed models" were consistently worse than those of the "optimal model" (Eavg-for-benefit ≥ 0.043 versus 0.002, E50-for-benefit ≥ 0.032 versus 0.001, E90-for-benefit ≥ 0.084 versus 0.004, cross-entropy-for-benefit ≥ 0.765 versus 0.750, Brier-for-benefit ≥ 0.220 versus 0.218). Calibration, discriminative ability, and overall performance of three different models were similar in the case study. The proposed metrics were implemented in a publicly available R-package "HTEPredictionMetrics".

CONCLUSION

The proposed metrics are useful to assess the calibration and overall performance of models predicting treatment effect in RCTs.

Estimating Patient-Specific Relative Benefit of Adding Biologics to Conventional Rheumatoid Arthritis Treatment: An Individual Participant Data Meta-Analysis

Importance

Current evidence remains ambiguous regarding whether biologics should be added to conventional treatment of rheumatoid arthritis for specific patients, which may cause potential overuse or treatment delay.

Objectives

To estimate the benefit of adding biologics to conventional antirheumatic drugs for the treatment of rheumatoid arthritis given baseline characteristics.

Data Sources

Cochrane CENTRAL, Scopus, MEDLINE, and the World Health Organization International Clinical Trials Registry Platform were searched for articles published from database inception to March 2, 2022.

Study Selection

Randomized clinical trials comparing certolizumab plus conventional antirheumatic drugs with placebo plus conventional drugs were selected.

Data Extraction and Synthesis

Individual participant data of the prespecified outcomes and covariates were acquired from the Vivli database. A 2-stage model was fitted to estimate patient-specific relative outcomes of adding certolizumab vs conventional drugs only. Stage 1 was a penalized logistic regression model to estimate the baseline expected probability of the outcome regardless of treatment using baseline characteristics. Stage 2 was a bayesian individual participant data meta-regression model to estimate the relative outcomes for a particular baseline expected probability. Patient-specific results were displayed interactively on an application based on a 2-stage model.

Main Outcomes and Measures

The primary outcome was low disease activity or remission at 3 months, defined by 3 disease activity indexes (ie, Disease Activity Score based on the evaluation of 28 joints, Clinical Disease Activity Index, or Simplified Disease Activity Index).

Results

Individual participant data were obtained from 3790 patients (2996 female [79.1%] and 794 male [20.9%]; mean [SD] age, 52.7 [12.3] years) from 5 large randomized clinical trials for moderate to high activity rheumatoid arthritis with usable data for 22 prespecified baseline covariates. Overall, adding certolizumab was associated with a higher probability of reaching low disease activity. The odds ratio for patients with an average baseline expected probability of the outcome was 6.31 (95% credible interval, 2.22-15.25). However, the benefits differed in patients with different baseline characteristics. For example, the estimated risk difference was smaller than 10% for patients with either low or high baseline expected probability.

Conclusions and Relevance

In this individual participant data meta-analysis, adding certolizumab was associated with more effectiveness for rheumatoid arthritis in general. However, the benefit was uncertain for patients with low or high baseline expected probability, for whom other evaluations were necessary. The interactive application displaying individual estimates may help with treatment selection.

Methodological concerns about "concordance-statistic for benefit" as a measure of discrimination in predicting treatment benefit

Prediction algorithms that quantify the expected benefit of a given treatment conditional on patient characteristics can critically inform medical decisions. Quantifying the performance of treatment benefit prediction algorithms is an active area of research. A recently proposed metric, the concordance statistic for benefit (cfb), evaluates the discriminative ability of a treatment benefit predictor by directly extending the concept of the concordance statistic from a risk model with a binary outcome to a model for treatment benefit. In this work, we scrutinize cfb on multiple fronts. Through numerical examples and theoretical developments, we show that cfb is not a proper scoring rule. We also show that it is sensitive to the unestimable correlation between counterfactual outcomes and to the definition of matched pairs. We argue that measures of statistical dispersion applied to predicted benefits do not suffer from these issues and can be an alternative metric for the discriminatory performance of treatment benefit predictors.

Measuring the performance of prediction models to personalize treatment choice

When data are available from individual patients receiving either a treatment or a control intervention in a randomized trial, various statistical and machine learning methods can be used to develop models for predicting future outcomes under the two conditions, and thus to predict treatment effect at the patient level. These predictions can subsequently guide personalized treatment choices. Although several methods for validating prediction models are available, little attention has been given to measuring the performance of predictions of personalized treatment effect. In this article, we propose a range of measures that can be used to this end. We start by defining two dimensions of model accuracy for treatment effects, for a single outcome: discrimination for benefit and calibration for benefit. We then amalgamate these two dimensions into an additional concept, decision accuracy, which quantifies the model's ability to identify patients for whom the benefit from treatment exceeds a given threshold. Subsequently, we propose a series of performance measures related to these dimensions and discuss estimating procedures, focusing on randomized data. Our methods are applicable for continuous or binary outcomes, for any type of prediction model, as long as it uses baseline covariates to predict outcomes under treatment and control. We illustrate all methods using two simulated datasets and a real dataset from a trial in depression. We implement all methods in the R package predieval. Results suggest that the proposed measures can be useful in evaluating and comparing the performance of competing models in predicting individualized treatment effect.

Decision Curve Analysis for Personalized Treatment Choice between Multiple Options

BACKGROUND

Decision curve analysis can be used to determine whether a personalized model for treatment benefit would lead to better clinical decisions. Decision curve analysis methods have been described to estimate treatment benefit using data from a single randomized controlled trial.

OBJECTIVES

Our main objective is to extend the decision curve analysis methodology to the scenario in which several treatment options exist and evidence about their effects comes from a set of trials, synthesized using network meta-analysis (NMA).

METHODS

We describe the steps needed to estimate the net benefit of a prediction model using evidence from studies synthesized in an NMA. We show how to compare personalized versus one-size-fit-all treatment decision-making strategies, such as "treat none" or "treat all patients with a specific treatment" strategies. First, threshold values for each included treatment need to be defined (i.e., the minimum risk difference compared with control that renders a treatment worth taking). The net benefit per strategy can then be plotted for a plausible range of threshold values to reveal the most clinically useful strategy. We applied our methodology to an NMA prediction model for relapsing-remitting multiple sclerosis, which can be used to choose between natalizumab, dimethyl fumarate, glatiramer acetate, and placebo.

RESULTS

We illustrated the extended decision curve analysis methodology using several threshold value combinations for each available treatment. For the examined threshold values, the "treat patients according to the prediction model" strategy performs either better than or close to the one-size-fit-all treatment strategies. However, even small differences may be important in clinical decision making. As the advantage of the personalized model was not consistent across all thresholds, improving the existing model (by including, for example, predictors that will increase discrimination) is needed before advocating its clinical usefulness.

CONCLUSIONS

This novel extension of decision curve analysis can be applied to NMA-based prediction models to evaluate their use to aid treatment decision making.

HIGHLIGHTS

Decision curve analysis is extended into a (network) meta-analysis framework.Personalized models predicting treatment benefit are evaluated when several treatment options are available and evidence about their effects comes from a set of trials.Detailed steps to compare personalized versus one-size-fit-all treatment decision-making strategies are outlined.This extension of decision curve analysis can be applied to (network) meta-analysis-based prediction models to evaluate their use to aid treatment decision making.

A standardized framework for risk-based assessment of treatment effect heterogeneity in observational healthcare databases

Treatment effects are often anticipated to vary across groups of patients with different baseline risk. The Predictive Approaches to Treatment Effect Heterogeneity (PATH) statement focused on baseline risk as a robust predictor of treatment effect and provided guidance on risk-based assessment of treatment effect heterogeneity in a randomized controlled trial. The aim of this study is to extend this approach to the observational setting using a standardized scalable framework. The proposed framework consists of five steps: (1) definition of the research aim, i.e., the population, the treatment, the comparator and the outcome(s) of interest; (2) identification of relevant databases; (3) development of a prediction model for the outcome(s) of interest; (4) estimation of relative and absolute treatment effect within strata of predicted risk, after adjusting for observed confounding; (5) presentation of the results. We demonstrate our framework by evaluating heterogeneity of the effect of thiazide or thiazide-like diuretics versus angiotensin-converting enzyme inhibitors on three efficacy and nine safety outcomes across three observational databases. We provide a publicly available R software package for applying this framework to any database mapped to the Observational Medical Outcomes Partnership Common Data Model. In our demonstration, patients at low risk of acute myocardial infarction receive negligible absolute benefits for all three efficacy outcomes, though they are more pronounced in the highest risk group, especially for acute myocardial infarction. Our framework allows for the evaluation of differential treatment effects across risk strata, which offers the opportunity to consider the benefit-harm trade-off between alternative treatments.

Estimating individualized treatment effects from randomized controlled trials: a simulation study to compare risk-based approaches

BACKGROUND

Baseline outcome risk can be an important determinant of absolute treatment benefit and has been used in guidelines for "personalizing" medical decisions. We compared easily applicable risk-based methods for optimal prediction of individualized treatment effects.

METHODS

We simulated RCT data using diverse assumptions for the average treatment effect, a baseline prognostic index of risk, the shape of its interaction with treatment (none, linear, quadratic or non-monotonic), and the magnitude of treatment-related harms (none or constant independent of the prognostic index). We predicted absolute benefit using: models with a constant relative treatment effect; stratification in quarters of the prognostic index; models including a linear interaction of treatment with the prognostic index; models including an interaction of treatment with a restricted cubic spline transformation of the prognostic index; an adaptive approach using Akaike's Information Criterion. We evaluated predictive performance using root mean squared error and measures of discrimination and calibration for benefit.

RESULTS

The linear-interaction model displayed optimal or close-to-optimal performance across many simulation scenarios with moderate sample size (N = 4,250; ~ 785 events). The restricted cubic splines model was optimal for strong non-linear deviations from a constant treatment effect, particularly when sample size was larger (N = 17,000). The adaptive approach also required larger sample sizes. These findings were illustrated in the GUSTO-I trial.

CONCLUSIONS

An interaction between baseline risk and treatment assignment should be considered to improve treatment effect predictions.

Analgesic and Anxiolytic Effects of Virtual Reality During Minor Procedures in an Emergency Department: A Randomized Controlled Study

STUDY OBJECTIVE

We aimed to assess the analgesic and anxiolytic efficacy of distraction, a nonpharmacologic intervention provided by 3-dimensional (3D) virtual reality (VR) compared with that provided by 2-dimensional (2D) VR during minor emergency department (ED) procedures.

METHODS

This randomized controlled study conducted in the ED of a teaching hospital included patients aged more than or equal to 18 years undergoing minor procedures. The patients watched the same computer-generated VR world either in 3D in a head-mounted display (intervention) or in 2D on a laptop screen (control). Our main outcomes were pain and anxiety during the procedure, assessed on a 100-mm visual analog scale. Secondary outcomes included the impression of telepresence in the computer-generated world assessed using the Igroup Presence Questionnaire, and the prevalence and intensity of cybersickness measured on a 100-mm visual analog scale.

RESULTS

The final analysis included 117 patients. The differences in median procedural pain and anxiety levels between the 2D and 3D VR groups were not significant: -3 mm (95% confidence interval [CI] -14 to 8) and -4 mm (95% CI -15 to 3), respectively; the difference in telepresence was 2.0 point (95% CI 0 to 2.0), and the proportion difference of cybersickness was -4% (95% CI -22 to 14), with an intensity difference of -5 mm (95% CI -9 to 3).

CONCLUSION

During minor procedures in adult patients in the ED, distraction by viewing a 3D virtual world in a head-mounted VR display did not result in lower average levels of procedural pain and anxiety than that by 2D viewing on a screen despite a higher sense of telepresence. There were no significant differences in the prevalence and intensity of cybersickness between the 2 groups.

Redefining effect modification

The odds ratio (OR) has been misunderstood in evidence based medicine and clinical epidemiology. Currently, "noncollapsibility" is considered a problem with interpretation of the OR and it is thought that the OR is rarely the parameter of interest for causal inference or interpretation of effect modification. The current focus on the relative risk (RR) and risk difference (RD) suffers from an important limitation: they are not solely measures of effect and vary numerically with baseline risk. In this paper, generalized linear models are examined in terms of the three binary effect measures commonly used in epidemiology to demonstrate that ORs may be the only way to interpret effect modification and have properties that should make them the parameter of interest for causal inference. We look forward to discussion, debate, and counter-views on this issue from the epidemiology community.

Assessing the ability of an instrumental variable causal forest algorithm to personalize treatment evidence using observational data: the case of early surgery for shoulder fracture

Background

Comparative effectiveness research (CER) using observational databases has been suggested to obtain personalized evidence of treatment effectiveness. Inferential difficulties remain using traditional CER approaches especially related to designating patients to reference classes a priori. A novel Instrumental Variable Causal Forest Algorithm (IV-CFA) has the potential to provide personalized evidence using observational data without designating reference classes a priori, but the consistency of the evidence when varying key algorithm parameters remains unclear. We investigated the consistency of IV-CFA estimates through application to a database of Medicare beneficiaries with proximal humerus fractures (PHFs) that previously revealed heterogeneity in the effects of early surgery using instrumental variable estimators.

Methods

IV-CFA was used to estimate patient-specific early surgery effects on both beneficial and detrimental outcomes using different combinations of algorithm parameters and estimate variation was assessed for a population of 72,751 fee-for-service Medicare beneficiaries with PHFs in 2011. Classification and regression trees (CART) were applied to these estimates to create ex-post reference classes and the consistency of these classes were assessed. Two-stage least squares (2SLS) estimators were applied to representative ex-post reference classes to scrutinize the estimates relative to known 2SLS properties.

Results

IV-CFA uncovered substantial early surgery effect heterogeneity across PHF patients, but estimates for individual patients varied with algorithm parameters. CART applied to these estimates revealed ex-post reference classes consistent across algorithm parameters. 2SLS estimates showed that ex-post reference classes containing older, frailer patients with more comorbidities, and lower utilizers of healthcare were less likely to benefit and more likely to have detriments from higher rates of early surgery.

Conclusions

IV-CFA provides an illuminating method to uncover ex-post reference classes of patients based on treatment effects using observational data with a strong instrumental variable. Interpretation of treatment effect estimates within each ex-post reference class using traditional CER methods remains conditional on the extent of measured information in the data.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01663-0.

Reported Challenges in Health Technology Assessment of Complex Health Technologies

OBJECTIVES

With complex health technologies entering the market, methods for health technology assessment (HTA) may require changes. This study aimed to identify challenges in HTA of complex health technologies.

METHODS

A survey was sent to European HTA organizations participating in European Network for HTA (EUnetHTA). The survey contained open questions and used predefined potentially complex health technologies and 7 case studies to identify types of complex health technologies and challenges faced during HTA. The survey was validated, tested for reliability by an expert panel, and pilot tested before dissemination.

RESULTS

A total of 22 HTA organizations completed the survey (67%). Advanced therapeutic medicinal products (ATMPs) and histology-independent therapies were considered most challenging based on the predefined complex health technologies and case studies. For the case studies, more than half of the reported challenges were "methodological," equal in relative effectiveness assessments as in cost-effectiveness assessments. Through the open questions, we found that most of these challenges actually rooted in data unavailability. Data were reported as "absent," "insufficient," "immature," or "low quality" by 18 of 20 organizations (90%), in particular data on quality of life. Policy and organizational challenges and challenges because of societal or political pressure were reported by 8 (40%) and 4 organizations (20%), respectively. Modeling issues were reported least often (n = 2, 4%).

CONCLUSIONS

Most challenges in HTA of complex health technologies root in data insufficiencies rather than in the complexity of health technologies itself. As the number of complex technologies grows, the urgency for new methods and policies to guide HTA decision making increases.

Prediction of treatment outcome in clinical trials under a personalized medicine perspective

A central problem in most data-driven personalized medicine scenarios is the estimation of heterogeneous treatment effects to stratify individuals into subpopulations that differ in their susceptibility to a particular disease or response to a specific treatment. In this work, with an illustrative example on type 2 diabetes we showed how the increasing ability to access and analyzed open data from randomized clinical trials (RCTs) allows to build Machine Learning applications in a framework of personalized medicine. An ensemble machine learning predictive model is first developed and then applied to estimate the expected treatment response according to the medication that would be prescribed. Machine learning is quickly becoming indispensable to bridge science and clinical practice, but it is not sufficient on its own. A collaborative effort is requested to clinicians, statisticians, and computer scientists to strengthen tools built on machine learning to take advantage of this evidence flow.

Pragmatic Precision Psychiatry-A New Direction for Optimizing Treatment Selection

IMPORTANCE

Clinical trials have identified numerous prescriptive predictors of mental disorder treatment response, ie, predictors of which treatments are best for which patients. However, none of these prescriptive predictors is strong enough alone to guide precision treatment planning. This has prompted growing interest in developing precision treatment rules (PTRs) that combine information across multiple prescriptive predictors, but this work has been much less successful in psychiatry than some other areas of medicine. Study designs and analysis schemes used in research on PTR development in other areas of medicine are reviewed, key challenges for implementing similar studies of mental disorders are highlighted, and recent methodological advances to address these challenges are described here.

OBSERVATIONS

Discovering prescriptive predictors requires large samples. Three approaches have been used in other areas of medicine to do this: conduct very large randomized clinical trials, pool individual-level results across multiple smaller randomized clinical trials, and develop preliminary PTRs in large observational treatment samples that are then tested in smaller randomized clinical trials. The third approach is most feasible for research on mental disorders. This approach requires working with large real-world observational electronic health record databases; carefully selecting samples to emulate trials; extracting information about prescriptive predictors from electronic health records along with other inexpensive data augmentation strategies; estimating preliminary PTRs in the observational data using appropriate methods; implementing pragmatic trials to validate the preliminary PTRs; and iterating between subsequent observational studies and quality improvement pragmatic trials to refine and expand the PTRs. New statistical methods exist to address the methodological challenges of implementing this approach.

CONCLUSIONS AND RELEVANCE

Advances in pragmatic precision psychiatry will require moving beyond the current focus on randomized clinical trials and adopting an iterative discovery-confirmation process that integrates observational and experimental studies in real-world clinical populations.

Leveraging machine learning predictive biomarkers to augment the statistical power of clinical trials with baseline magnetic resonance imaging

A key factor in designing randomized clinical trials is the sample size required to achieve a particular level of power to detect the benefit of a treatment. Sample size calculations depend upon the expected benefits of a treatment (effect size), the accuracy of measurement of the primary outcome, and the level of power specified by the investigators. In this study, we show that radiomic models, which leverage complex brain MRI patterns and machine learning, can be utilized in clinical trials with protocols that incorporate baseline MR imaging to significantly increase statistical power to detect treatment effects. Akin to the historical control paradigm, we propose to utilize a radiomic prediction model to generate a pseudo-control sample for each individual in the trial of interest. Because the variability of expected outcome across patients can mask our ability to detect treatment effects, we can increase the power to detect a treatment effect in a clinical trial by reducing that variability through using radiomic predictors as surrogates. We illustrate this method with simulations based on data from two cohorts in different neurologic diseases, Alzheimer's disease and glioblastoma multiforme. We present sample size requirements across a range of effect sizes using conventional analysis and models that include a radiomic predictor. For our Alzheimer's disease cohort, at an effect size of 0.35, total sample size requirements for 80% power declined from 246 to 212 for the endpoint cognitive decline. For our glioblastoma multiforme cohort, at an effect size of 1.65 with the endpoint survival time, total sample size requirements declined from 128 to 74. This methodology can decrease the required sample sizes by as much as 50%, depending on the strength of the radiomic predictor. The power of this method grows with increased accuracy of radiomic prediction, and furthermore, this method is most helpful when treatment effect sizes are small. Neuroimaging biomarkers are a powerful and increasingly common suite of tools that are, in many cases, highly predictive of disease outcomes. Here, we explore the possibility of using MRI-based radiomic biomarkers for the purpose of improving statistical power in clinical trials in the contexts of brain cancer and prodromal Alzheimer's disease. These methods can be applied to a broad range of neurologic diseases using a broad range of predictors of outcome to make clinical trials more efficient.

Harrell Jr F, Moons KGM, Debray TPA, Reitsma JB. A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint

Randomized trials typically estimate average relative treatment effects, but decisions on the benefit of a treatment are possibly better informed by more individualized predictions of the absolute treatment effect. In case of a binary outcome, these predictions of absolute individualized treatment effect require knowledge of the individual's risk without treatment and incorporation of a possibly differential treatment effect (ie, varying with patient characteristics). In this article, we lay out the causal structure of individualized treatment effect in terms of potential outcomes and describe the required assumptions that underlie a causal interpretation of its prediction. Subsequently, we describe regression models and model estimation techniques that can be used to move from average to more individualized treatment effect predictions. We focus mainly on logistic regression-based methods that are both well-known and naturally provide the required probabilistic estimates. We incorporate key components from both causal inference and prediction research to arrive at individualized treatment effect predictions. While the separate components are well known, their successful amalgamation is very much an ongoing field of research. We cut the problem down to its essentials in the setting of a randomized trial, discuss the importance of a clear definition of the estimand of interest, provide insight into the required assumptions, and give guidance with respect to modeling and estimation options. Simulated data illustrate the potential of different modeling options across scenarios that vary both average treatment effect and treatment effect heterogeneity. Two applied examples illustrate individualized treatment effect prediction in randomized trial data.

A two-stage prediction model for heterogeneous effects of treatments

Treatment effects vary across different patients, and estimation of this variability is essential for clinical decision-making. We aimed to develop a model estimating the benefit of alternative treatment options for individual patients, extending a risk modeling approach in a network meta-analysis framework. We propose a two-stage prediction model for heterogeneous treatment effects by combining prognosis research and network meta-analysis methods where individual patient data are available. In the first stage, a prognostic model to predict the baseline risk of the outcome. In the second stage, we use the baseline risk score from the first stage as a single prognostic factor and effect modifier in a network meta-regression model. We apply the approach to a network meta-analysis of three randomized clinical trials comparing the relapses in Natalizumab, Glatiramer Acetate, and Dimethyl Fumarate, including 3590 patients diagnosed with relapsing-remitting multiple sclerosis. We find that the baseline risk score modifies the relative and absolute treatment effects. Several patient characteristics, such as age and disability status, impact the baseline risk of relapse, which in turn moderates the benefit expected for each of the treatments. For high-risk patients, the treatment that minimizes the risk of relapse in 2 years is Natalizumab, whereas Dimethyl Fumarate might be a better option for low-risk patients. Our approach can be easily extended to all outcomes of interest and has the potential to inform a personalized treatment approach.