Multiplicity issues in exploratory subgroup analysis

Differential Treatment Effects of Subgroup Analyses in Phase 3 Oncology Trials From 2004 to 2020

Importance

Subgroup analyses are often performed in oncology to investigate differential treatment effects and may even constitute the basis for regulatory approvals. Current understanding of the features, results, and quality of subgroup analyses is limited.

Objective

To evaluate forest plot interpretability and credibility of differential treatment effect claims among oncology trials.

Design, Setting, and Participants

This cross-sectional study included randomized phase 3 clinical oncology trials published prior to 2021. Trials were screened from ClinicalTrials.gov.

Main Outcomes and Measures

Missing visual elements in forest plots were defined as a missing point estimate or use of a linear x-axis scale for hazard and odds ratios. Multiplicity of testing control was recorded. Differential treatment effect claims were rated using the Instrument for Assessing the Credibility of Effect Modification Analyses. Linear and logistic regressions evaluated associations with outcomes.

Results

Among 785 trials, 379 studies (48%) enrolling 331 653 patients reported a subgroup analysis. The forest plots of 43% of trials (156 of 363) were missing visual elements impeding interpretability. While 4148 subgroup effects were evaluated, only 1 trial (0.3%) controlled for multiple testing. On average, trials that did not meet the primary end point conducted 2 more subgroup effect tests compared with trials meeting the primary end point (95% CI, 0.59-3.43 tests; P = .006). A total of 101 differential treatment effects were claimed across 15% of trials (55 of 379). Interaction testing was missing in 53% of trials (29 of 55) claiming differential treatment effects. Trials not meeting the primary end point were associated with greater odds of no interaction testing (odds ratio, 4.47; 95% CI, 1.42-15.55, P = .01). The credibility of differential treatment effect claims was rated as low or very low in 93% of cases (94 of 101).

Conclusions and Relevance

In this cross-sectional study of phase 3 oncology trials, nearly half of trials presented a subgroup analysis in their primary publication. However, forest plots of these subgroup analyses largely lacked essential features for interpretation, and most differential treatment effect claims were not supported. Oncology subgroup analyses should be interpreted with caution, and improvements to the quality of subgroup analyses are needed.

A novel concept of screening for subgrouping factors for the association between socioeconomic status and respiratory allergies

BACKGROUND

The new subgroup screening tool "subscreen" aims to understand the unclear and complex association between socioeconomic status (SES) and childhood allergy. This software R package has been successfully used in clinical trials but not in large population-based studies.

OBJECTIVE

To screen and identify subgrouping factors explaining their impact on the association between SES and respiratory allergies in childhood and youth.

METHODS

Using the national German childhood and youth survey dataset (KiGGS Wave 2), we included 56 suspected subgrouping factors to investigate the association between SES (low vs. high) and allergic rhinitis and/or asthma in an exploratory manner. The package enabled a comprehensive overview of odds ratios when considering the SES impact per subgroup and analogously all disease proportions per subgroup.

RESULT

Among the 56 candidate factors, striking subgrouping factors were identified; e.g., if mothers were younger and in the low SES group, their children had a higher risk of asthma. In addition children of the teen's age were associated with increased risks in the low SES group. For the crude proportions, factors such as (parental) smoking or having had no "contact with farm animals" were identified as strong risk factors for rhinitis.

SIGNIFICANCE

The "subscreen" package enabled the detection of notable subgroups for further investigations exemplarily for similar epidemiological research questions.

Effectiveness and safety of massage for chronic pain in patients with knee osteoarthritis: A protocol for systematic review and meta-analysis

BACKGROUND

Chronic pain (CP) is a common and debilitating symptom in patients with knee osteoarthritis (KOA). Massage has been supported as a non-pharmacological intervention for the individual symptom relief of CP. However, relevant evidence of using massage for CP in patients with KOA has been lacking.

METHODS

A systematic search will be performed in the following electronic databases for randomized controlled trials to evaluate the effectiveness and safety of massage for CP of KOA: China National Knowledge Infrastructure (CNKI), Wan Fang, PubMed, EMBASE, PsycINFO, and the Cochrane Library databases from their inception to December 2021. The entire process will include study selection, data extraction, risk of bias assessment and meta-analysis.

RESULTS

This proposed study will evaluate the effectiveness and safety of massage for CP in patients with KOA. Changes in pain relief and adverse effect will be included in our outcomes.

CONCLUSIONS

This systematic review will provide evidence for assessing the credibility of massage for CP in patients with KOA.

DISSEMINATION AND ETHICS

The results of this review will be disseminated through peer-reviewed publication. This review does not require ethical approval because all of the data used in this systematic review and meta-analysis have already been published. Furthermore, all of these data will be analyzed anonymously during the review process.

Effectiveness and safety of Tai Chi for chronic pain of knee osteoarthritis: A protocol for systematic review and meta-analysis

BACKGROUND

Chronic pain (CP) has been a major area of interest in the field of knee osteoarthritis (KOA), further aggravating the dysthymia, stiffness, and dysfunction of KOA patients. As an important part of complementary and alternative medicine, Tai Chi has a positive effect on KOA patients. The systematic review is to evaluate the effectiveness and safety of Tai Chi for KOA patients with CP.

METHODS

A systematic search will be performed in the following electronic databases for randomized controlled trials to evaluate the effectiveness and safety of Tai Chi in treating chronic pain of knee osteoarthritis: the Cochrane Library, PubMed, EMBASE, OVID-MEDLINE, and four Chinese databases (Wan Fang, CNKI, CBMdisc and VIP). Each database will be searched from inception to Dec. 2021. The process will include study selection, data extraction, risk of bias assessment and meta-analysis.

RESULTS

This proposed study will evaluate the effectiveness and safety of Tai Chi for KOA patients with CP. Improvement in pain and adverse effects of KOA will be included in our measure.

CONCLUSIONS

This proposed systematic review and meta-analysis will evaluate the existing evidence on the effectiveness and safety of Tai Chi for KOA patients with CP.

DISSEMINATION AND ETHICS

The results of this review will be disseminated through peer-reviewed publication. This review does not require ethical approval because all of the data used in this systematic review and meta-analysis have already been published. Furthermore, all of these data will be analyzed anonymously during the review process.

INPLASY REGISTRATION NUMBER

INPLASY2021120020.

Effectiveness and safety of massage for athletic injuries: A protocol for systematic review and meta-analysis

BACKGROUND

Athletic injuries have been a major area of interest in the field of sports and clinical medicine. Implemented on people's skin, muscles, and joints as an important part of complementary and alternative medicine (CAM), massage therapy has a positive effect on athletic injuries. This protocol is to provide the methods used to evaluate the effectiveness and safety of massage therapy for patients with athletic injuries.

METHODS

A systematic search will be performed in the following electronic databases for randomized controlled trials (RCTs) to evaluate the effectiveness and safety of massage therapy in treating athletic injuries: PubMed, the Cochrane Library, EMBASE and four Chinese databases (CNKI, Wan Fang, CBMdisc and VIP). Each database will be searched from inception to July 2021. The entire process will include study selection, data extraction, risk of bias assessment and meta-analysis.

RESULTS

A high-quality synthesis of current evidence of massage therapy for patients with athletic injuries will be provided.

CONCLUSIONS

This systematic review will provide evidence for assessing the credibility of massage therapy for patients with athletic injuries.

DISSEMINATION AND ETHICS

The results of this review will be disseminated through peer-reviewed publication. This review does not require ethical approval because all the data used in this systematic review and meta-analysis have already been published. Furthermore, all of these data will be analyzed anonymously during the review process.

INPLASY REGISTRATION NUMBER

INPLASY202170066.

Learning and confirming a class of treatment responders in clinical trials

Clinical trials require substantial effort and time to complete, and regulatory agencies may require two successful efficacy trials before approving a new drug. One way to improve the chance of follow-up success is to identify a subpopulation among whom treatment effects are estimated to be beneficial, and enrolling future studies from this subpopulation. In this article we study confirmable responder class (CRC) learning, where the objective is to learn in a random half of the dataset (training set) a subpopulation among whom the predicted conditional ATE (CATE) suggests clinically meaningful benefit, with maximum power of being confirmed via hypothesis test in the other half (test set). We studied a set of CRC learners across simulated datasets in which either all patients benefited, or only some did. Performance metrics included the rate of confirmation in the test set, and the classification accuracy of the CRC compared with the group with true CATE>0. In trials where all patients benefitted, only two learners were able to consistently identify most of the population as the CRC. One of these also performed especially well when only some patients benefitted, having relatively high confirmation rates, and showing robustness to the dimension of the covariate vector and population characteristics. This learner is based on cross-validation, and is a possible avenue for further development of model selection criteria for CRC learning. We also show that the performance of all methods can be improved by using both halves of the original dataset as training and test sets in turn.

Development of the Instrument to assess the Credibility of Effect Modification Analyses (ICEMAN) in randomized controlled trials and meta-analyses

BACKGROUND

Most randomized controlled trials (RCTs) and meta-analyses of RCTs examine effect modification (also called a subgroup effect or interaction), in which the effect of an intervention varies by another variable (e.g., age or disease severity). Assessing the credibility of an apparent effect modification presents challenges; therefore, we developed the Instrument for assessing the Credibility of Effect Modification Analyses (ICEMAN).

METHODS

To develop ICEMAN, we established a detailed concept; identified candidate credibility considerations in a systematic survey of the literature; together with experts, performed a consensus study to identify key considerations and develop them into instrument items; and refined the instrument based on feedback from trial investigators, systematic review authors and journal editors, who applied drafts of ICEMAN to published claims of effect modification.

RESULTS

The final instrument consists of a set of preliminary considerations, core questions (5 for RCTs, 8 for meta-analyses) with 4 response options, 1 optional item for additional considerations and a rating of credibility on a visual analogue scale ranging from very low to high. An accompanying manual provides rationales, detailed instructions and examples from the literature. Seventeen potential users tested ICEMAN; their suggestions improved the user-friendliness of the instrument.

INTERPRETATION

The Instrument for assessing the Credibility of Effect Modification Analyses offers explicit guidance for investigators, systematic reviewers, journal editors and others considering making a claim of effect modification or interpreting a claim made by others.

Q-Finder: An Algorithm for Credible Subgroup Discovery in Clinical Data Analysis - An Application to the International Diabetes Management Practice Study

Addressing the heterogeneity of both the outcome of a disease and the treatment response to an intervention is a mandatory pathway for regulatory approval of medicines. In randomized clinical trials (RCTs), confirmatory subgroup analyses focus on the assessment of drugs in predefined subgroups, while exploratory ones allow a posteriori the identification of subsets of patients who respond differently. Within the latter area, subgroup discovery (SD) data mining approach is widely used-particularly in precision medicine-to evaluate treatment effect across different groups of patients from various data sources (be it from clinical trials or real-world data). However, both the limited consideration by standard SD algorithms of recommended criteria to define credible subgroups and the lack of statistical power of the findings after correcting for multiple testing hinder the generation of hypothesis and their acceptance by healthcare authorities and practitioners. In this paper, we present the Q-Finder algorithm that aims to generate statistically credible subgroups to answer clinical questions, such as finding drivers of natural disease progression or treatment response. It combines an exhaustive search with a cascade of filters based on metrics assessing key credibility criteria, including relative risk reduction assessment, adjustment on confounding factors, individual feature's contribution to the subgroup's effect, interaction tests for assessing between-subgroup treatment effect interactions and tests adjustment (multiple testing). This allows Q-Finder to directly target and assess subgroups on recommended credibility criteria. The top-k credible subgroups are then selected, while accounting for subgroups' diversity and, possibly, clinical relevance. Those subgroups are tested on independent data to assess their consistency across databases, while preserving statistical power by limiting the number of tests. To illustrate this algorithm, we applied it on the database of the International Diabetes Management Practice Study (IDMPS) to better understand the drivers of improved glycemic control and rate of episodes of hypoglycemia in type 2 diabetics patients. We compared Q-Finder with state-of-the-art approaches from both Subgroup Identification and Knowledge Discovery in Databases literature. The results demonstrate its ability to identify and support a short list of highly credible and diverse data-driven subgroups for both prognostic and predictive tasks.

Reporting of health equity considerations in cluster and individually randomized trials

Background

The randomized controlled trial (RCT) is considered the gold standard study design to inform decisions about the effectiveness of interventions. However, a common limitation is inadequate reporting of the applicability of the intervention and trial results for people who are “socially disadvantaged” and this can affect policy-makers’ decisions. We previously developed a framework for identifying health-equity-relevant trials, along with a reporting guideline for transparent reporting. In this study, we provide a descriptive assessment of health-equity considerations in 200 randomly sampled equity-relevant trials.

Methods

We developed a search strategy to identify health-equity-relevant trials published between 2013 and 2015. We randomly sorted the 4316 records identified by the search and screened studies until 100 individually randomized (RCTs) and 100 cluster randomized controlled trials (CRTs) were identified. We developed and pilot-tested a data extraction form based on our initial work, to inform the development of our reporting guideline for equity-relevant randomized trials.

Results

In total, 39 trials (20%) were conducted in a low- and middle-income country and 157 trials (79%) in a high-income country focused on socially disadvantaged populations (78% CRTs, 79% RCTs). Seventy-four trials (37%) reported a subgroup analysis across a population characteristic associated with disadvantage (25% CRT, 49% RCTs), with 19% of included studies reporting subgroup analyses across sex, 9% across race/ethnicity/culture, and 4% across socioeconomic status. No subgroup analyses were reported for place of residence, occupation, religion, education, or social capital. One hundred and forty-one trials (71%) discussed the applicability of their results to one or more socially disadvantaged populations (68% of CRT, 73% of RCT).

Discussion

In this set of trials, selected for their relevance to health equity, data that were disaggregated for socially disadvantaged populations were rarely reported. We found that even when the data are available, opportunities to analyze health-equity considerations are frequently missed. The recently published equity extension of the Consolidated Reporting Standards for Randomized Trials (CONSORT-Equity) may help improve delineation of hypotheses related to socially disadvantaged populations, and transparency and completeness of reporting of health-equity considerations in RCTs. This study can serve as a baseline assessment of the reporting of equity considerations.

A critical review of graphics for subgroup analyses in clinical trials

Subgroup analyses are a routine part of clinical trials to investigate whether treatment effects are homogeneous across the study population. Graphical approaches play a key role in subgroup analyses to visualise effect sizes of subgroups, to aid the identification of groups that respond differentially, and to communicate the results to a wider audience. Many existing approaches do not capture the core information and are prone to lead to a misinterpretation of the subgroup effects. In this work, we critically appraise existing visualisation techniques, propose useful extensions to increase their utility and attempt to develop an effective visualisation approach. We focus on forest plots, UpSet plots, Galbraith plots, subpopulation treatment effect pattern plot, and contour plots, and comment on other approaches whose utility is more limited. We illustrate the methods using data from a prostate cancer study.

A Systematic Approach for Post Hoc Subgroup Analyses With Applications in Clinical Case Studies

BACKGROUND

The analysis of subgroups in clinical trials is essential to assess differences in treatment effects for distinct patient clusters, that is, to detect patients with greater treatment benefit or patients where the treatment seems to be ineffective.

METHODS

The software application subscreen (R package) has been developed to analyze the population of clinical trials in minute detail. The aim was to efficiently calculate point estimates (eg, hazard ratios) for multiple subgroups to identify groups that potentially differ from the overall trial result. The approach intentionally avoids inferential statistics such as P values or confidence intervals but intends to encourage discussions enriched with external evidence (eg, from other studies) about the exploratory results, which can be accompanied by further statistical methods in subsequent analyses. The subscreen application was applied to 2 clinical study data sets and used in a simulation study to demonstrate its usefulness.

RESULTS

The visualization of numerous combined subgroups illustrates the homogeneity or heterogeneity of potentially all subgroup estimates with the overall result. With this, the application leads to more targeted planning of future trials.

CONCLUSION

This described approach supports the current trend and requirements for the investigation of subgroup effects as discussed in the EMA draft guidance for subgroup analyses in confirmatory clinical trials (EMA 2014). The lack of a convenient tool to answer spontaneous questions from different perspectives can hinder an efficient discussion, especially in joint interdisciplinary study teams. With the new application, an easily executed but powerful tool is provided to fill this gap.

Early onset of efficacy with erenumab in patients with episodic and chronic migraine

Background

Subcutaneous erenumab reduced monthly migraine days and increased the likelihood of achieving a ≥ 50% reduction at all monthly assessment points tested in 2 pivotal trials in episodic migraine (EM) and chronic migraine (CM). Early efficacy of migraine preventive medications is an important treatment characteristic to patients. Delays in achievement of efficacy can result in failed adherence. The objective of these post-hoc analyses were to evaluate efficacy in the first 4 weeks after initial subcutaneous administration of erenumab 70 mg, erenumab 140 mg, or placebo.

Methods

There is no generally accepted methodology to measure onset of action for migraine preventive medications. We used a comprehensive approach with data from both studies to evaluate change from baseline in weekly migraine days (WMD), achievement of ≥ 50% reduction in WMD, and proportion of patients experiencing migraine measured on a daily basis. The 7-day moving averages were overlaid with observed data.

Results

In both studies (EM: N = 955; CM: N = 667), there was evidence of onset of efficacy of erenumab vs. placebo during the first week of treatment, which in some cases reached nominal significance. For EM the changes in WMD were (least squares mean [LSM] [95% CI]): placebo, − 0.1 (− 0.3, 0.0); erenumab 70 mg, − 0.3 (− 0.5, − 0.2) p = 0.130; erenumab 140 mg, − 0.6 (− 0.7, − 0.4) p < 0.001. For CM the changes were: placebo, − 0.5 (− 0.8, − 0.3); erenumab 70 mg, − 0.9 (− 1.2, − 0.7) p = 0.047; erenumab 140 mg, − 0.8 (− 1.1, − 0.5) p = 0.18. Achievement of ≥ 50% reduction in WMD was observed as early as Week 1 (adjusted OR [95% CI] erenumab vs placebo) in EM: erenumab 70 mg, 1.3 (1.0, 1.9) p = 0.097; erenumab 140 mg, 2.0 (1.4, 2.7) p < 0.001. A similar outcome was observed for CM: erenumab 70 mg, 1.8 (1.1, 2.8) p = 0.011; erenumab 140 mg, 1.9 (1.2, 2.9) p = 0.009. Seven-day moving averages of observed data showed each treatment arm differed from placebo by Week 1 (OR [95% CI]): in EM Day 3 for erenumab 140 mg, 0.7 (0.5, 1.0) p = 0.031 and at Day 7 for 70 mg, 0.6 (0.4, 0.8) p = 0.002; in CM: Day 6 for erenumab 70 mg, 0.6 (0.4, 0.9) p = 0.022 and at Day 7 for 140 mg, 0.7 (0.4, 1.0); p = 0.038.

Conclusion

Erenumab showed early onset of efficacy with separation from placebo within the first week of treatment in both chronic and episodic migraine patients.