Look before you leap: systematic evaluation of tree-based statistical methods in subgroup identification

Modern approaches for evaluating treatment effect heterogeneity from clinical trials and observational data

In this paper, we review recent advances in statistical methods for the evaluation of the heterogeneity of treatment effects (HTE), including subgroup identification and estimation of individualized treatment regimens, from randomized clinical trials and observational studies. We identify several types of approaches using the features introduced in Lipkovich et al (Stat Med 2017;36: 136-196) that distinguish the recommended principled methods from basic methods for HTE evaluation that typically rely on rules of thumb and general guidelines (the methods are often referred to as common practices). We discuss the advantages and disadvantages of various principled methods as well as common measures for evaluating their performance. We use simulated data and a case study based on a historical clinical trial to illustrate several new approaches to HTE evaluation.

Inference on tree-structured subgroups with subgroup size and subgroup effect relationship in clinical trials

When multiple candidate subgroups are considered in clinical trials, we often need to make statistical inference on the subgroups simultaneously. Classical multiple testing procedures might not lead to an interpretable and efficient inference on the subgroups as they often fail to take subgroup size and subgroup effect relationship into account. In this paper, built on the selective traversed accumulation rules (STAR), we propose a data-adaptive and interactive multiple testing procedure for subgroups which can take subgroup size and subgroup effect relationship into account under prespecified tree structure. The proposed method is easy-to-implement and can lead to a more interpretable and efficient inference on prespecified tree-structured subgroups. Possible accommodations to post hoc identified tree-structure subgroups are also discussed in the paper. We demonstrate the merit of our proposed method by re-analyzing the panitumumab trial with the proposed method.

Predictors of abstinence, no heavy drinking days, and a 2-level reduction in World Health Organization drinking levels during treatment for alcohol use disorder in the COMBINE study

BACKGROUND

Data from trials of medications for alcohol use disorder (AUD) can be used to identify predictors of drinking outcomes regardless of treatment, which can inform the design of future trials with heterogeneous populations. Here, we identified predictors of abstinence, no heavy drinking days, and a 2-level reduction in World Health Organization (WHO) drinking levels during treatment for AUD in the Combined Pharmacotherapies and Behavioral Interventions (COMBINE) Study.

METHODS

We utilized data from the COMBINE Study, a randomized placebo-controlled trial evaluating the efficacy of naltrexone and acamprosate, both alone and in combination, for AUD (n = 1168). A tree-based machine learning algorithm was used to construct classification trees predicting abstinence, no heavy drinking days, and a 2-level reduction in WHO drinking levels in the last 4 weeks of treatment, based on 89 baseline variables.

RESULTS

The final tree for predicting abstinence had one split based on consecutive days abstinent prior to randomization, with a higher proportion of subjects achieving abstinence among those classified as abstinent for >2 versus ≤2 consecutive weeks prior to randomization (66% vs. 29%). The final tree for predicting no heavy drinking days in the last 4 weeks of treatment had three splits based on consecutive days abstinent, age, and total Alcohol Dependence Scale score at baseline. Seventy-three percent of the subjects classified as abstinent for >2 consecutive weeks prior to randomization had no heavy drinking days in the last 4 weeks of treatment. Among those classified as abstinent ≤2 consecutive weeks prior, three additional splits showed that younger subjects (age ≤44 years; 37%), and older subjects (age >44) with a total Alcohol Dependence Scale score >13 and complete abstinence (56%) or other drinking goals (35%), were less likely to have no heavy drinking days than older subjects with a total Alcohol Dependence Scale score ≤13 (67%). The final tree for predicting a 2-level reduction in WHO levels had no splits.

CONCLUSIONS

Consecutive days abstinent prior to randomization may predict abstinence and no heavy drinking days and total Alcohol Dependence Scale score and age may predict no heavy drinking days. The 2-level reduction in WHO levels outcome may be less likely to discriminate based on multiple patient characteristics.

An Application of the Patient Rule-Induction Method to Detect Clinically Meaningful Subgroups from Failed Phase III Clinical Trials

Background

Phase III superiority clinical trials have negative results (new treatment is not statistically better than standard of care) due to a number of factors, including patient and disease heterogeneity. However, even a treatment regime that fails to show population-level clinical improvement will have a subgroup of patients that attain a measurable clinical benefit.

Objective

The goal of this paper is to modify the Patient Rule-Induction Method to identify statistically significant subgroups, defined by clinical and/or demographic factors, of the clinical trial population where the experimental treatment performs better than the standard of care and better than observed in the entire clinical trial sample.

Results

We illustrate this method using part A of the SUCCESS clinical trial, which showed no overall difference between treatment arms: HR (95% CI) = 0.97 (0.78, 1.20). Using PRIM, we identified one subgroup defined by the mutational profile in BRCA1 which resulted in a significant benefit for adding Gemcitabine to the standard treatment: HR (95% CI) = 0.59 (0.40, 0.87).

Conclusion

This result demonstrates that useful information can be extracted from existing databases that could provide insight into why a phase III trial failed and assist in the design of future clinical trials involving the experimental treatment.

Subgroup discovery in non-inferiority trials

Approaches and guidelines for performing subgroup analysis to assess heterogeneity of treatment effect in clinical trials have been the topic of numerous papers in the statistical and clinical literature, but have been discussed predominantly in the context of conventional superiority trials. Concerns about treatment heterogeneity are the same if not greater in non-inferiority (NI) trials, especially since overall similarity between two treatment arms in a successful NI trial could be due to the existence of qualitative interactions that are more likely when comparing two active therapies. Even in unsuccessful NI trials, subgroup analyses can yield important insights about the potential reasons for failure to demonstrate non-inferiority of the experimental therapy. Recent NI trials have performed a priori subgroup analyses using standard statistical tests for interaction, but there is increasing interest in more flexible machine learning approaches for post-hoc subgroup discovery. The performance and practical application of such methods in NI trials have not been systematically explored, however. We considered the Virtual Twin method for the NI setting, an algorithm for subgroup identification that combines random forest with classification and regression trees, and conducted extensive simulation studies to examine its performance under different NI trial conditions and to devise decision rules for selecting the final subgroups. We illustrate the utility of the method with data from a NI trial that was conducted to compare two acupuncture treatments for chronic musculoskeletal pain.