A note on confidence bounds after fixed-sequence multiple tests

On generalized fixed sequence procedures for controlling the FWER

Testing a sequence of pre-ordered hypotheses to decide which of these can be rejected or accepted while controlling the familywise error rate (FWER) is of importance in many scientific studies such as clinical trials. In this paper, we first introduce a generalized fixed sequence procedure whose critical values are defined by using a function of the numbers of rejections and acceptances, and which allows follow-up hypotheses to be tested even if some earlier hypotheses are not rejected. We then construct the least favorable configuration for this generalized fixed sequence procedure and present a sufficient condition for the FWER control under arbitrary dependence. Based on the condition, we develop three new generalized fixed sequence procedures controlling the FWER under arbitrary dependence. We also prove that each generalized fixed sequence procedure can be described as a specific closed testing procedure. Through simulation studies and a clinical trial example, we compare the power performance of these proposed procedures with those of the existing FWER controlling procedures. Finally, when the pairwise joint distributions of the true null p-values are known, we further improve these procedures by incorporating pairwise correlation information while maintaining the control of the FWER. Copyright © 2015 John Wiley & Sons, Ltd.

Informative simultaneous confidence intervals in hierarchical testing

BACKGROUND AND OBJECTIVES

In clinical trials involving multiple tests it is often difficult to obtain informative simultaneous confidence intervals (SCIs). In particular in hierarchical testing, no quantification of effects is possible for the first tested (and most important) hypothesis after its rejection. Our goal is a construction of SCIs that are always informative.

METHODS

We present an approach where the level is split after rejection of each hypothesis to obtain an informative confidence bound. The splitting weights are continuous functions of the parameters. Our method is realizable by a simple algorithm and is illustrated by an intuitive graphical representation.

RESULTS

We show theoretically and by an example that the new SCIs always provide information when a hypothesis is rejected. The power to reject the first hypothesis is not smaller than for the classical fixed-sequence procedure. The price for the extra information is a small power loss in the hypotheses proceeding the most important one.

CONCLUSIONS

Given the substantial gain in information, a small loss of power for the non-primary hypotheses seems often acceptable. Especially in the context of non-inferiority trials, this method is a useful alternative. The flexibility in the choice of the weight functions makes the procedure attractive for applications.

The thresholds for statistical and clinical significance - a five-step procedure for evaluation of intervention effects in randomised clinical trials

BACKGROUND

Thresholds for statistical significance are insufficiently demonstrated by 95% confidence intervals or P-values when assessing results from randomised clinical trials. First, a P-value only shows the probability of getting a result assuming that the null hypothesis is true and does not reflect the probability of getting a result assuming an alternative hypothesis to the null hypothesis is true. Second, a confidence interval or a P-value showing significance may be caused by multiplicity. Third, statistical significance does not necessarily result in clinical significance. Therefore, assessment of intervention effects in randomised clinical trials deserves more rigour in order to become more valid.

METHODS

Several methodologies for assessing the statistical and clinical significance of intervention effects in randomised clinical trials were considered. Balancing simplicity and comprehensiveness, a simple five-step procedure was developed.

RESULTS

For a more valid assessment of results from a randomised clinical trial we propose the following five-steps: (1) report the confidence intervals and the exact P-values; (2) report Bayes factor for the primary outcome, being the ratio of the probability that a given trial result is compatible with a 'null' effect (corresponding to the P-value) divided by the probability that the trial result is compatible with the intervention effect hypothesised in the sample size calculation; (3) adjust the confidence intervals and the statistical significance threshold if the trial is stopped early or if interim analyses have been conducted; (4) adjust the confidence intervals and the P-values for multiplicity due to number of outcome comparisons; and (5) assess clinical significance of the trial results.

CONCLUSIONS

If the proposed five-step procedure is followed, this may increase the validity of assessments of intervention effects in randomised clinical trials.