Discounting phase 2 results when planning phase 3 clinical trials

Improving the assessment of the probability of success in late stage drug development

There are several steps to confirming the safety and efficacy of a new medicine. A sequence of trials, each with its own objectives, is usually required. Quantitative risk metrics can be useful for informing decisions about whether a medicine should transition from one stage of development to the next. To obtain an estimate of the probability of regulatory approval, pharmaceutical companies may start with industry-wide success rates and then apply to these subjective adjustments to reflect program-specific information. However, this approach lacks transparency and fails to make full use of data from previous clinical trials. We describe a quantitative Bayesian approach for calculating the probability of success (PoS) at the end of phase II which incorporates internal clinical data from one or more phase IIb studies, industry-wide success rates, and expert opinion or external data if needed. Using an example, we illustrate how PoS can be calculated accounting for differences between the phase II data and future phase III trials, and discuss how the methods can be extended to accommodate accelerated drug development pathways.

Adjusting for bias in the mean for primary and secondary outcomes when trials are in sequence

When designing a clinical trial, one key aspect of the design is the sample size calculation. The sample size calculation tends to rely on a target or expected difference. The expected difference can be based on the observed data from previous studies, which results in bias. It has been reported that large treatment effects observed in trials are often not replicated in subsequent trials. If these values are used to design subsequent studies, the sample sizes may be biased which results in an unethical study. Regression to the mean (RTM) is one explanation for this. If only health technologies which meet a particular continuation criterion (such as p < 0.05 in the first study) are progressed to a second confirmatory trial, it is highly likely that the observed effect in the second trial will be lower than that observed in the first trial. It will be shown how when moving from one trial to the next, a truncated normal distribution is inherently imposed on the first study. This results in a lower observed effect size in the second trial. A simple adjustment method is proposed based on the mathematical properties of the truncated normal distribution. This adjustment method was confirmed using simulations in R and compared with other previous adjustments. The method can be applied to the observed effect in a trial, which is being used in the design of a second confirmatory trial, resulting in a more stable estimate for the 'true' treatment effect. The adjustment accounts for the bias in the primary and secondary endpoints in the first trial with the bias being affected by the power of that study. Tables of results have been provided to aid implementation, along with a worked example. In summary, there is a bias introduced when the point estimate from one trial is used to assist the design of a second trial. It is recommended that any observed point estimates be used with caution and the adjustment method developed in this article be implemented to significantly reduce this bias.

A New Comprehensive Approach to Assess the Probability of Success of Development Programs Before Pivotal Trials

The point at which clinical development programs transition from early phase to pivotal trials is a critical milestone. Substantial uncertainty about the outcome of pivotal trials may remain even after seeing positive early phase data, and companies may need to make difficult prioritization decisions for their portfolio. The probability of success (PoS) of a program, a single number expressed as a percentage reflecting the multitude of risks that may influence the final program outcome, is a key decision-making tool. Despite its importance, companies often rely on crude industry benchmarks that may be "adjusted" by experts based on undocumented criteria and which are typically misaligned with the definition of success used to drive commercial forecasts, leading to overly optimistic expected net present value calculations. We developed a new framework to assess the PoS of a program before pivotal trials begin. Our definition of success encompasses the successful outcome of pivotal trials, regulatory approval and meeting the requirements for market access as outlined in the target product profile. The proposed approach is organized in four steps and uses an innovative Bayesian approach to synthesize all relevant evidence. The new PoS framework is systematic and transparent. It will help organizations to make more informed decisions. In this paper, we outline the rationale and elaborate on the structure of the proposed framework, provide examples, and discuss the benefits and challenges associated with its adoption.

Do strict decision criteria hamper productivity in the pharmaceutical industry?

The discouragingly high rates of attrition in drug development, and in particular in Phase 2, warrant a closer look at the decision criteria applied for investment in the next phase (Phase 3). We have in this article evaluated Stop/Go criteria after Phase 2, based on a model encompassing both Phase 2 and 3, as well as the eventual outcome on the market. The results indicate that the value of a drug project is often maximized if rather liberal decision criteria are applied. The routine adherence to standard criteria, e.g. requiring significance at 5% level, may lead to an unduly high rate of false negative decisions. This might ultimately hamper the productivity of drug development and leading to potentially useful drugs not being taken forward to benefit the intended patients.

Selection bias, investment decisions and treatment effect distributions

When making decisions regarding the investment and design for a Phase 3 programme in the development of a new drug, the results from preceding Phase 2 trials are an important source of information. However, only projects in which the Phase 2 results show promising treatment effects will typically be considered for a Phase 3 investment decision. This implies that, for those projects where Phase 3 is pursued, the underlying Phase 2 estimates are subject to selection bias. We will in this article investigate the nature of this selection bias based on a selection of distributions for the treatment effect. We illustrate some properties of Bayesian estimates, providing shrinkage of the Phase 2 estimate to counteract the selection bias. We further give some empirical guidance regarding the choice of prior distribution and comment on the consequences for decision-making in investment and planning for Phase 3 programmes.

Optimal designs for phase II/III drug development programs including methods for discounting of phase II results

Background

Go/no-go decisions after phase II and sample size chosen for phase III are usually based on phase II results (e.g., the treatment effect estimate of phase II). Due to the decision rule (only promising phase II results lead to phase III), treatment effect estimates from phase II that initiate a phase III trial commonly overestimate the true treatment effect. Underpowered phase III trials are the consequence. Optimistic findings may then not be reproduced, leading to the failure of potentially expensive drug development programs. For some disease areas these failure rates are described to be quite high: 62.5%.

Methods

We integrate the ideas of multiplicative and additive adjustment of treatment effect estimates after go decisions in a utility-based framework for optimizing drug development programs. The design of a phase II/III program, i.e., the “right amount of adjustment”, the allocation of the resources to phase II and III in terms of sample size, and the rule applied to decide whether to stop or to proceed with phase III influences its success considerably. Given specific drug development program characteristics (e.g., fixed and variable per patient costs for phase II and III, probable gain in case of market launch), optimal designs with respect to the maximal expected utility can be identified by the proposed Bayesian-frequentist approach. The method will be illustrated by application to practical examples characteristic for oncological studies.

Results

In general, our results show that the program set-ups with adjusted treatment effect estimate used for phase III planning are superior to the “naïve” program set-ups with respect to the maximal expected utility. Therefore, we recommend considering an adjusted phase II treatment effect estimate for the phase III sample size calculation. However, there is no one-fits-all design.

Conclusion

Individual drug development planning for a specific program is necessary to find the optimal design. The optimal choice of the design parameters for a specific drug development program at hand can be found by our user friendly R Shiny application and package (both assessable open-source via [1]).

Efficacy outcomes in phase 2 and phase 3 randomized controlled trials in rheumatology

Phase 3 trials are the mainstay of drug development across medicine but have often not met expectations set by preceding phase 2 studies. A systematic meta-analysis evaluated all randomized controlled, double-blind trials investigating targeted disease-modifying anti-rheumatic drugs in rheumatoid and psoriatic arthritis. Primary outcomes of American College of Rheumatology (ACR) 20 responses were compared by mixed-model logistic regression, including exploration of potential determinants of efficacy overestimation. In rheumatoid arthritis, phase 2 trial outcomes systematically overestimated subsequent phase 3 results (odds ratio comparing ACR20 in phase 2 versus phase 3: 1.39, 95% confidence interval: 1.25-1.57, P < 0.001). Data for psoriatic arthritis trials were similar, but not statistically significant (odds ratio comparing ACR20 in phase 2 versus phase 3: 1.35, 95% confidence interval: 0.94-1.94, P = 0.09). Differences in inclusion criteria largely explained the observed differences in efficacy findings. Our findings have implications for all stakeholders in new therapeutic development and testing, as well as potential ethical implications.

Selection bias for treatments with positive Phase 2 results

In drug development, treatments are most often selected at Phase 2 for further development when an initial trial of a new treatment produces a result that is considered positive. This selection due to a positive result means, however, that an estimator of the treatment effect, which does not take account of the selection is likely to over-estimate the true treatment effect (ie, will be biased). This bias can be large and researchers may face a disappointingly lower estimated treatment effect in further trials. In this paper, we review a number of methods that have been proposed to correct for this bias and introduce three new methods. We present results from applying the various methods to two examples and consider extensions of the examples. We assess the methods with respect to bias of estimation of the treatment effect and compare the probabilities that a bias-corrected treatment effect estimate will exceed a decision threshold. Following previous work, we also compare average power for the situation where a Phase 3 trial is launched given that the bias-corrected observed Phase 2 treatment effect exceeds a launch threshold. Finally, we discuss our findings and potential application of the bias correction methods.

Empowering phase II clinical trials to reduce phase III failures

The large number of failures in phase III clinical trials, which occur at a rate of approximately 45%, is studied herein relative to possible countermeasures. First, the phenomenon of failures is numerically described. Second, the main reasons for failures are reported, together with some generic improvements suggested in the related literature. This study shows how statistics explain, but do not justify, the high failure rate observed. The rate of failures due to a lack of efficacy that are not expected, is considered to be at least 10%. Expanding phase II is the simplest and most intuitive way to reduce phase III failures since it can reduce phase III false negative findings and launches of phase III trials when the treatment is positive but suboptimal. Moreover, phase II enlargement is discussed using an economic profile. As resources for research are often limited, enlarging phase II should be evaluated on a case-by-case basis. Alternative strategies, such as biomarker-based enrichments and adaptive designs, may aid in reducing failures. However, these strategies also have very low application rates with little likelihood of rapid growth.

Practical help for specifying the target difference in sample size calculations for RCTs: the DELTA2 five-stage study, including a workshop

BACKGROUND

The randomised controlled trial is widely considered to be the gold standard study for comparing the effectiveness of health interventions. Central to its design is a calculation of the number of participants needed (the sample size) for the trial. The sample size is typically calculated by specifying the magnitude of the difference in the primary outcome between the intervention effects for the population of interest. This difference is called the 'target difference' and should be appropriate for the principal estimand of interest and determined by the primary aim of the study. The target difference between treatments should be considered realistic and/or important by one or more key stakeholder groups.

OBJECTIVE

The objective of the report is to provide practical help on the choice of target difference used in the sample size calculation for a randomised controlled trial for researchers and funder representatives.

METHODS

The Difference ELicitation in TriAls2 (DELTA2) recommendations and advice were developed through a five-stage process, which included two literature reviews of existing funder guidance and recent methodological literature; a Delphi process to engage with a wider group of stakeholders; a 2-day workshop; and finalising the core document.

RESULTS

Advice is provided for definitive trials (Phase III/IV studies). Methods for choosing the target difference are reviewed. To aid those new to the topic, and to encourage better practice, 10 recommendations are made regarding choosing the target difference and undertaking a sample size calculation. Recommended reporting items for trial proposal, protocols and results papers under the conventional approach are also provided. Case studies reflecting different trial designs and covering different conditions are provided. Alternative trial designs and methods for choosing the sample size are also briefly considered.

CONCLUSIONS

Choosing an appropriate sample size is crucial if a study is to inform clinical practice. The number of patients recruited into the trial needs to be sufficient to answer the objectives; however, the number should not be higher than necessary to avoid unnecessary burden on patients and wasting precious resources. The choice of the target difference is a key part of this process under the conventional approach to sample size calculations. This document provides advice and recommendations to improve practice and reporting regarding this aspect of trial design. Future work could extend the work to address other less common approaches to the sample size calculations, particularly in terms of appropriate reporting items.

FUNDING

Funded by the Medical Research Council (MRC) UK and the National Institute for Health Research as part of the MRC-National Institute for Health Research Methodology Research programme.

Quantitative decision-making in randomized Phase II studies with a time-to-event endpoint

One of the most critical decision points in clinical development is Go/No-Go decision-making after a proof-of-concept study. Traditional decision-making relies on a formal hypothesis testing with control of type I and type II error rates, which is limited by assessing the strength of efficacy evidence in a small isolated trial. In this article, we propose a quantitative Bayesian/frequentist decision framework for Go/No-Go criteria and sample size evaluation in Phase II randomized studies with a time-to-event endpoint. By taking the uncertainty of treatment effect into consideration, we propose an integrated quantitative approach for a program when both the Phase II and Phase III trials share a common endpoint while allowing a discount of the observed Phase II data. Our results confirm the argument that an increase in the sample size of a Phase II trial will result in greater increase in the probability of success of a Phase III trial than increasing the Phase III trial sample size by equal amount. We illustrate the steps in quantitative decision-making with a real example of a randomized Phase II study in metastatic pancreatic cancer.

Assessment of Bayesian expected power via Bayesian bootstrap

The Bayesian expected power (BEP) has become increasingly popular in assessing the probability of success for a future trial. While the traditional power assumes a single value for the unknown effect size Δ and is thus highly dependent on the assumed value, the BEP embraces the uncertainty around Δ given the prior information and is therefore a less subjective measure for the probability of success than the traditional power especially when the prior information is not rich. Current methods for assessing BEP are often based in a parametric framework by imposing a model on the pilot data to derive and sample from the posterior distributions of Δ. The model-based approach can be analytically challenging and computationally costly especially for multivariate data sets, and it also runs the risk of generating misleading BEP if the model is misspecified. We propose an approach based on the Bayesian bootstrap (BBS) technique to simulate future trials in the presence of individual-level pilot data, based on which the empirical BEP can be calculated. The BBS approach is model-free with no assumptions about the distribution of the prior data and also circumvents the analytical and computational complexity associated with obtaining the posterior distribution of the Δ. Information from multiple pilot studies is also straightforward to combine. We also propose the double bootstrap technique, a frequentist counterpart to the BBS, that shares similar properties and achieves the same goal as the BBS for BEP assessment. Simulation and case studies are presented to demonstrate the implementation of the BBS technique and the double bootstrap technique and to compare the BEP results with model-based approach.

Interval and point estimation in adaptive Phase II trials with binary endpoint

Phase II clinical trials are concerned with making decision of whether a treatment is sufficiently efficacious to be worth further investigations in late large scale Phase III trials. In oncology Phase II trials, frequentist single-arm two-stage group-sequential designs with a binary endpoint are commonly used. To allow for more flexibility, adaptive versions of these designs have been proposed. In this paper, we propose point and interval estimation for adaptive designs in which the second stage sample size is a pre-specified function of first stage's number of responses. Our approach is based on sample space orderings, from which we derive p-values, and point and interval estimates. Simulation studies show that our proposed methods perform better, in terms of bias and root mean square error, than the fixed-sample maximum likelihood estimator.

A decision theoretical modeling for Phase III investments and drug licensing

For a new candidate drug to become an approved medicine, several decision points have to be passed. In this article, we focus on two of them: First, based on Phase II data, the commercial sponsor decides to invest (or not) in Phase III. Second, based on the outcome of Phase III, the regulator determines whether the drug should be granted market access. Assuming a population of candidate drugs with a distribution of true efficacy, we optimize the two stakeholders' decisions and study the interdependence between them. The regulator is assumed to seek to optimize the total public health benefit resulting from the efficacy of the drug and a safety penalty. In optimizing the regulatory rules, in terms of minimal required sample size and the Type I error in Phase III, we have to consider how these rules will modify the commercial optimization made by the sponsor. The results indicate that different Type I errors should be used depending on the rarity of the disease.

Simulation-based adjustment after exploratory biomarker subgroup selection in phase II

As part of the evaluation of phase II trials, it is common practice to perform exploratory subgroup analyses with the aim of identifying patient populations with a beneficial treatment effect. When investigating targeted therapies, these subgroups are typically defined by biomarkers. Promising results may lead to the decision to select the respective subgroup as the target population for a subsequent phase III trial. However, a selection based on a large observed treatment effect may potentially induce an upwards-bias leading to over-optimistic expectations on the success probability of the phase III trial. We describe how Approximate Bayesian Computation techniques can be used to derive a simulation-based bias adjustment method in this situation. Recommendations for the implementation of the approach are given. Simulation studies show that the proposed method reduces bias substantially compared with the maximum likelihood estimator. The procedure is illustrated with data from an oncology trial. Copyright © 2017 John Wiley & Sons, Ltd.

Profit Evaluations When Adaptation by Design Is Applied

BACKGROUND

Adaptation by design consists in conservatively estimating the phase III sample size on the basis of phase II data; it is also called conservative sample size estimation (CSSE). The usual assumptions are that the effect size is the same in both phases and that phase II data are not used for phase III confirmatory analysis. CSSE has been introduced to increase the rate of successful trials, and it can be applied in most clinical areas. CSSE reduces the probability of underpowered experiments and can improve the overall success probability of phase II and III, but it also increases phase III sample size, increasing the time and cost of experiments. Thus, the balance between higher revenue and greater cost is the issue.

METHODS

A profit model was built assuming that CSSE was applied and considering income per patient, annual incidence, time on market, market share, phase III success probability, fixed cost of the 2 phases, and cost per patient under treatment.

RESULTS

Profit turns out to be a random variable depending on phase II sample size and conservativeness. Profit moments are obtained in a closed formula. Profit utility, which is a linear function of profit expectation and volatility, is evaluated in accordance with the modern theory of investment performances. Indications regarding phase II sample size and conservativeness can be derived on the basis of utility, for example, through utility optimization.

CONCLUSIONS

CSSE can be adopted in many different statistical problems, and consequently the profit evaluations proposed here can be widely applied.

Sequential biases in accumulating evidence

Whilst it is common in clinical trials to use the results of tests at one phase to decide whether to continue to the next phase and to subsequently design the next phase, we show that this can lead to biased results in evidence synthesis. Two new kinds of bias associated with accumulating evidence, termed 'sequential decision bias' and 'sequential design bias', are identified. Both kinds of bias are the result of making decisions on the usefulness of a new study, or its design, based on the previous studies. Sequential decision bias is determined by the correlation between the value of the current estimated effect and the probability of conducting an additional study. Sequential design bias arises from using the estimated value instead of the clinically relevant value of an effect in sample size calculations. We considered both the fixed-effect and the random-effects models of meta-analysis and demonstrated analytically and by simulations that in both settings the problems due to sequential biases are apparent. According to our simulations, the sequential biases increase with increased heterogeneity. Minimisation of sequential biases arises as a new and important research area necessary for successful evidence-based approaches to the development of science. © 2015 The Authors. Research Synthesis Methods Published by John Wiley & Sons Ltd.

Sample size planning for phase II trials based on success probabilities for phase III

In recent years, high failure rates in phase III trials were observed. One of the main reasons is overoptimistic assumptions for the planning of phase III resulting from limited phase II information and/or unawareness of realistic success probabilities. We present an approach for planning a phase II trial in a time-to-event setting that considers the whole phase II/III clinical development programme. We derive stopping boundaries after phase II that minimise the number of events under side conditions for the conditional probabilities of correct go/no-go decision after phase II as well as the conditional success probabilities for phase III. In addition, we give general recommendations for the choice of phase II sample size. Our simulations show that unconditional probabilities of go/no-go decision as well as the unconditional success probabilities for phase III are influenced by the number of events observed in phase II. However, choosing more than 150 events in phase II seems not necessary as the impact on these probabilities then becomes quite small. We recommend considering aspects like the number of compounds in phase II and the resources available when determining the sample size. The lower the number of compounds and the lower the resources are for phase III, the higher the investment for phase II should be.

Bayesian methods for the design and analysis of noninferiority trials

The gold standard for evaluating treatment efficacy of a medical product is a placebo-controlled trial. However, when the use of placebo is considered to be unethical or impractical, a viable alternative for evaluating treatment efficacy is through a noninferiority (NI) study where a test treatment is compared to an active control treatment. The minimal objective of such a study is to determine whether the test treatment is superior to placebo. An assumption is made that if the active control treatment remains efficacious, as was observed when it was compared against placebo, then a test treatment that has comparable efficacy with the active control, within a certain range, must also be superior to placebo. Because of this assumption, the design, implementation, and analysis of NI trials present challenges for sponsors and regulators. In designing and analyzing NI trials, substantial historical data are often required on the active control treatment and placebo. Bayesian approaches provide a natural framework for synthesizing the historical data in the form of prior distributions that can effectively be used in design and analysis of a NI clinical trial. Despite a flurry of recent research activities in the area of Bayesian approaches in medical product development, there are still substantial gaps in recognition and acceptance of Bayesian approaches in NI trial design and analysis. The Bayesian Scientific Working Group of the Drug Information Association provides a coordinated effort to target the education and implementation issues on Bayesian approaches for NI trials. In this article, we provide a review of both frequentist and Bayesian approaches in NI trials, and elaborate on the implementation for two common Bayesian methods including hierarchical prior method and meta-analytic-predictive approach. Simulations are conducted to investigate the properties of the Bayesian methods, and some real clinical trial examples are presented for illustration.

The shrinking or disappearing observed treatment effect

It is frequently noted that an initial clinical trial finding was not reproduced in a later trial. This is often met with some surprise. Yet, there is a relatively straightforward reason partially responsible for this observation. In this article, we examine this reason by first reviewing some findings in a recent publication in the Journal of the American Medical Association. To help explain the non-negligible chance of failing to reproduce a previous positive finding, we compare a series of trials to successive diagnostic tests used for identifying a condition. To help explain the suspicion that the treatment effect, when observed in a subsequent trial, seems to have decreased in magnitude, we draw a conceptual analogy between phases II-III development stages and interim analyses of a trial with a group sequential design. Both analogies remind us that what we observed in an early trial could be a false positive or a random high. We discuss statistical sources for these occurrences and discuss why it is important for statisticians to take these into consideration when designing and interpreting trial results.

A Quantitative Process for Enhancing End of Phase 2 Decisions

The objectives of the phase 2 stage in a drug development program are to evaluate the safety and tolerability of different doses, select a promising dose range, and look for early signs of activity. At the end of phase 2, a decision to initiate phase 3 studies is made that involves the commitment of considerable resources. This multifactorial decision, generally made by balancing the current condition of a development organization's portfolio, the future cost of development, the competitive landscape, and the expected safety and efficacy benefits of a new therapy, needs to be a good one. In this article, we present a practical quantitative process that has been implemented for drugs entering phase 2 at Amgen Ltd. to ensure a consistent and explicit evidence-based approach is used to contribute to decisions for new drug candidates. Broadly following this process will also help statisticians increase their strategic influence in drug development programs. The process is illustrated using an example from the pancreatic cancer indication. Embedded within the process is a predominantly Bayesian approach to predicting the probability of efficacy success in a future (frequentist) phase 3 program.