Smoothed Nested Testing on Directed Acyclic Graphs

We consider the problem of multiple hypothesis testing when there is a logical nested structure to the hypotheses. When one hypothesis is nested inside another, the outer hypothesis must be false if the inner hypothesis is false. We model the nested structure as a directed acyclic graph, including chain and tree graphs as special cases. Each node in the graph is a hypothesis and rejecting a node requires also rejecting all of its ancestors. We propose a general framework for adjusting node-level test statistics using the known logical constraints. Within this framework, we study a smoothing procedure that combines each node with all of its descendants to form a more powerful statistic. We prove a broad class of smoothing strategies can be used with existing selection procedures to control the familywise error rate, false discovery exceedance rate, or false discovery rate, so long as the original test statistics are independent under the null. When the null statistics are not independent but are derived from positively-correlated normal observations, we prove control for all three error rates when the smoothing method is arithmetic averaging of the observations. Simulations and an application to a real biology dataset demonstrate that smoothing leads to substantial power gains.

Covariate adaptive familywise error rate control for genome-wide association studies

The familywise error rate has been widely used in genome-wide association studies. With the increasing availability of functional genomics data, it is possible to increase detection power by leveraging these genomic functional annotations. Previous efforts to accommodate covariates in multiple testing focused on false discovery rate control, while covariate-adaptive procedures controlling the familywise error rate remain underdeveloped. Here, we propose a novel covariate-adaptive procedure to control the familywise error rate that incorporates external covariates which are potentially informative of either the statistical power or the prior null probability. An efficient algorithm is developed to implement the proposed method. We prove its asymptotic validity and obtain the rate of convergence through a perturbation-type argument. Our numerical studies show that the new procedure is more powerful than competing methods and maintains robustness across different settings. We apply the proposed approach to the UK Biobank data and analyse 27 traits with 9 million single-nucleotide polymorphisms tested for associations. Seventy-five genomic annotations are used as covariates. Our approach detects more genome-wide significant loci than other methods in 21 out of the 27 traits.

Statistical considerations of phase 3 umbrella trials allowing adding one treatment arm mid-trial

Master protocols, in particular umbrella trials and platform trials, when evaluating multiple experimental treatments with a common control, could save patient resource, increase trial efficiency, and reduce drug development cost. Compared to the phase 3 platform trials that allow unlimited number of experimental arms to be added, it is more practical for individual companies to evaluate two experimental arms with a common control in an umbrella trial and allow the second experimental arm to be added at a later time. There have been limited research done in this type of trials in terms of statistical properties and guidance. In this article, we present statistical considerations of a phase 3 three-arm umbrella design including Type I error control and power, as well as the optimal allocation ratio. We intend to not only complement the existing literature, but more importantly to provide practical guidance to pave the way for its implementation by individual companies.

Voxel-based meta-analysis via permutation of subject images (PSI): Theory and implementation for SDM

Coordinate-based meta-analyses (CBMA) are very useful for summarizing the large number of voxel-based neuroimaging studies of normal brain functions and brain abnormalities in neuropsychiatric disorders. However, current CBMA methods do not conduct common voxelwise tests, but rather a test of convergence, which relies on some spatial assumptions that data may seldom meet, and has lower statistical power when there are multiple effects. Here we present a new algorithm that can use standard voxelwise tests and, importantly, conducts a standard permutation of subject images (PSI). Its main steps are: a) multiple imputation of study images; b) imputation of subject images; and c) subject-based permutation test to control the familywise error rate (FWER). The PSI algorithm is general and we believe that developers might implement it for several CBMA methods. We present here an implementation of PSI for seed-based d mapping (SDM) method, which additionally benefits from the use of effect sizes, random-effects models, Freedman-Lane-based permutations and threshold-free cluster enhancement (TFCE) statistics, among others. Finally, we also provide an empirical validation of the control of the FWER in SDM-PSI, which showed that it might be too conservative. We hope that the neuroimaging meta-analytic community will welcome this new algorithm and method.

A gatekeeping procedure to test a primary and a secondary endpoint in a group sequential design with multiple interim looks

Glimm et al. (2010) and Tamhane et al. (2010) studied the problem of testing a primary and a secondary endpoint, subject to a gatekeeping constraint, using a group sequential design (GSD) with K=2 looks. In this article, we greatly extend the previous results to multiple (K>2) looks. If the familywise error rate (FWER) is to be controlled at a preassigned α level then it is clear that the primary boundary must be of level α. We show under what conditions one α-level primary boundary is uniformly more powerful than another. Based on this result, we recommend the choice of the O'Brien and Fleming (1979) boundary over the Pocock (1977) boundary for the primary endpoint. For the secondary endpoint the choice of the boundary is more complicated since under certain conditions the secondary boundary can be refined to have a nominal level α'>α, while still controlling the FWER at level α, thus boosting the secondary power. We carry out secondary power comparisons via simulation between different choices of primary-secondary boundary combinations. The methodology is applied to the data from the RALES study (Pitt et al., 1999; Wittes et al., 2001). An R library package gsrsb to implement the proposed methodology is made available on CRAN.

Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies

Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.

Comparison of design strategies for a three-arm clinical trial with time-to-event endpoint: Power, time-to-analysis, and operational aspects

To optimize resources, randomized clinical trials with multiple arms can be an attractive option to simultaneously test various treatment regimens in pharmaceutical drug development. The motivation for this work was the successful conduct and positive final outcome of a three-arm randomized clinical trial primarily assessing whether obinutuzumab plus chlorambucil in patients with chronic lympocytic lymphoma and coexisting conditions is superior to chlorambucil alone based on a time-to-event endpoint. The inference strategy of this trial was based on a closed testing procedure. We compare this strategy to three potential alternatives to run a three-arm clinical trial with a time-to-event endpoint. The primary goal is to quantify the differences between these strategies in terms of the time it takes until the first analysis and thus potential approval of a new drug, number of required events, and power. Operational aspects of implementing the various strategies are discussed. In conclusion, using a closed testing procedure results in the shortest time to the first analysis with a minimal loss in power. Therefore, closed testing procedures should be part of the statistician's standard clinical trials toolbox when planning multiarm clinical trials.

Type I error rates of multi-arm multi-stage clinical trials: strong control and impact of intermediate outcomes

BACKGROUND

The multi-arm multi-stage (MAMS) design described by Royston et al. [Stat Med. 2003;22(14):2239-56 and Trials. 2011;12:81] can accelerate treatment evaluation by comparing multiple treatments with a control in a single trial and stopping recruitment to arms not showing sufficient promise during the course of the study. To increase efficiency further, interim assessments can be based on an intermediate outcome (I) that is observed earlier than the definitive outcome (D) of the study. Two measures of type I error rate are often of interest in a MAMS trial. Pairwise type I error rate (PWER) is the probability of recommending an ineffective treatment at the end of the study regardless of other experimental arms in the trial. Familywise type I error rate (FWER) is the probability of recommending at least one ineffective treatment and is often of greater interest in a study with more than one experimental arm.

METHODS

We demonstrate how to calculate the PWER and FWER when the I and D outcomes in a MAMS design differ. We explore how each measure varies with respect to the underlying treatment effect on I and show how to control the type I error rate under any scenario. We conclude by applying the methods to estimate the maximum type I error rate of an ongoing MAMS study and show how the design might have looked had it controlled the FWER under any scenario.

RESULTS

The PWER and FWER converge to their maximum values as the effectiveness of the experimental arms on I increases. We show that both measures can be controlled under any scenario by setting the pairwise significance level in the final stage of the study to the target level. In an example, controlling the FWER is shown to increase considerably the size of the trial although it remains substantially more efficient than evaluating each new treatment in separate trials.

CONCLUSIONS

The proposed methods allow the PWER and FWER to be controlled in various MAMS designs, potentially increasing the uptake of the MAMS design in practice. The methods are also applicable in cases where the I and D outcomes are identical.

Multiple comparisons with two controls for ordered categorical responses

In clinical studies, ordered categorical responses are common. To compare the efficacy of several treatments with a control for ordinal responses, the normal latent variable model has recently been proposed. This approach conceptualizes the responses as manifestations of an underlying continuous normal variable. In this article, we extend this idea to develop the multiple comparison method for use when there are two controls in the clinical trial. The proposed method is constructed such that the familywise type I error rate is controlled at a prespecified level. In addition, for a given level of test power, the procedure to evaluate the required sample size is provided. The proposed testing procedure is also illustrated by an example from a clinical study.

Allocating recycled significance levels in group sequential procedures for multiple endpoints

Graphical approaches have been proposed in the literature for testing hypotheses on multiple endpoints by recycling significance levels from rejected hypotheses to unrejected ones. Recently, they have been extended to group sequential procedures (GSPs). Our focus in this paper is on the allocation of recycled significance levels from rejected hypotheses to the stages of the GSPs for unrejected hypotheses. We propose a delayed recycling method that allocates the recycled significance level from Stage r onward, where r is prespecified. We show that r cannot be chosen adaptively to coincide with the random stage at which the hypothesis from which the significance level is recycled is rejected. Such an adaptive GSP does not always control the FWER. One can choose r to minimize the expected sample size for a given power requirement. We illustrate how a simulation approach can be used for this purpose. Several examples, including a clinical trial example, are given to illustrate the proposed procedure.

Noninferiority Studies with Multiple New Treatments and Heterogeneous Variances

The objective of a noninferiority (NI) trial is to affirm the efficacy of a new treatment compared with an active control by verifying that the new treatment maintains a considerable portion of the treatment effect of the control. Compensation by benefits other than efficacy is usually the justification for using a new treatment, as long as the loss of efficacy is within an acceptable margin (NI margin) from the standard treatment. A popular approach is to express this margin in terms of the efficacy difference between the new treatment and the active control. Based on this approach and the realization that NI trials often comprise several new treatments, statistical procedures that simultaneously conduct NI tests of several new treatments have been developed. However, these procedures rely on the assumption that the variances of the treatments are homogeneous. In this article, we discuss the undesirable effect of using these procedures on the familywise Type I error rate when the treatment responses have heterogeneous variances. To alleviate this problem, we reveal potential procedures that are more appropriate. Further, a power study is conducted to compare the different procedures to provide guidance on the selection of adequate testing procedures in NI trials. Clinical examples are given for illustrative purposes.

A step-up test procedure to find the minimum effective dose

It is of great interest to find the minimum effective dose (MED) in dose-response studies. A sequence of decreasing null hypotheses to find the MED is formulated under the assumption of nondecreasing dose response means. A step-up multiple test procedure that controls the familywise error rate (FWER) is constructed based on the maximum likelihood estimators for the monotone normal means. When the MED is equal to one, the proposed test is uniformly more powerful than Hsu and Berger's test (1999). Also, a simulation study shows a substantial power improvement for the proposed test over four competitors. Three R-codes are provided in Supplemental Materials for this article. Go to the publishers online edition of Journal of Biopharmaceutical Statistics to view the files.

A hybrid method to estimate the minimum effective dose for monotone and non-monotone dose-response relationships

This article proposes a new multiple-testing approach for estimation of the minimum effective dose allowing for non-monotonous dose-response shapes. The presented approach combines the advantages of two commonly used methods. It is shown that the new approach controls the error rate of underestimating the true minimum effective dose. Monte Carlo simulations indicate that the proposed method outperforms alternative methods in many cases and is only marginally worse in the remaining situations.

Step-up procedures for non-inferiority tests with multiple experimental treatments

Non-inferiority (NI) trials are becoming more popular. The NI of a new treatment compared with a standard treatment is established when the new treatment maintains a substantial fraction of the treatment effect of the standard treatment. A valid NI trial is also required to show assay sensitivity, the demonstration of the standard treatment having the expected effect with a size comparable to those reported in previous placebo-controlled studies. A three-arm NI trial is a clinical study that includes a new treatment, a standard treatment and a placebo. Most of the statistical methods developed for three-arm NI trials are designed for the existence of only one new treatment. Recently, a single-step procedure was developed to deal with NI trials with multiple new treatments with the overall familywise error rate controlled at a specified level. In this article, we extend the single-step procedure to two new step-up procedures for NI trials with multiple new treatments. A comparative study of test power shows that both proposed step-up procedures provide a significant improvement of power when compared to the single-step procedure. One of the two proposed step-up procedures also allows the flexibility of allocating different error rates between the sensitivity hypothesis and the NI hypotheses so that the assignment of fewer patients to the placebo becomes possible when designing NI trials. We illustrate the new procedures using data from a clinical trial.

Asymptotics of Bonferroni for Dependent Normal Test Statistics

The Bonferroni adjustment is sometimes used to control the familywise error rate (FWE) when the number of comparisons is huge. In genome wide association studies, researchers compare cases to controls with respect to thousands of single nucleotide polymorphisms. It has been claimed that the Bonferroni adjustment is only slightly conservative if the comparisons are nearly independent. We show that the veracity of this claim depends on how one defines "nearly." Specifically, if the test statistics' pairwise correlations converge to 0 as the number of tests tend to ∞, the conservatism of the Bonferroni procedure depends on their rate of convergence. The type I error rate of Bonferroni can tend to 0 or 1 - exp(-α) ≈ α, depending on that rate. We show using elementary probability theory what happens to the distribution of the number of errors when using Bonferroni, as the number of dependent normal test statistics gets large. We also use the limiting behavior of Bonferroni to shed light on properties of other commonly used test statistics.