Simulation Practices for Adaptive Trial Designs in Drug and Device Development

Using Bayesian statistics in confirmatory clinical trials in the regulatory setting: a tutorial review

Bayesian statistics plays a pivotal role in advancing medical science by enabling healthcare companies, regulators, and stakeholders to assess the safety and efficacy of new treatments, interventions, and medical procedures. The Bayesian framework offers a unique advantage over the classical framework, especially when incorporating prior information into a new trial with quality external data, such as historical data or another source of co-data. In recent years, there has been a significant increase in regulatory submissions using Bayesian statistics due to its flexibility and ability to provide valuable insights for decision-making, addressing the modern complexity of clinical trials where frequentist trials are inadequate. For regulatory submissions, companies often need to consider the frequentist operating characteristics of the Bayesian analysis strategy, regardless of the design complexity. In particular, the focus is on the frequentist type I error rate and power for all realistic alternatives. This tutorial review aims to provide a comprehensive overview of the use of Bayesian statistics in sample size determination, control of type I error rate, multiplicity adjustments, external data borrowing, etc., in the regulatory environment of clinical trials. Fundamental concepts of Bayesian sample size determination and illustrative examples are provided to serve as a valuable resource for researchers, clinicians, and statisticians seeking to develop more complex and innovative designs.

How to check a simulation study

Simulation studies are powerful tools in epidemiology and biostatistics, but they can be hard to conduct successfully. Sometimes unexpected results are obtained. We offer advice on how to check a simulation study when this occurs, and how to design and conduct the study to give results that are easier to check. Simulation studies should be designed to include some settings in which answers are already known. They should be coded in stages, with data-generating mechanisms checked before simulated data are analysed. Results should be explored carefully, with scatterplots of standard error estimates against point estimates surprisingly powerful tools. Failed estimation and outlying estimates should be identified and dealt with by changing data-generating mechanisms or coding realistic hybrid analysis procedures. Finally, we give a series of ideas that have been useful to us in the past for checking unexpected results. Following our advice may help to prevent errors and to improve the quality of published simulation studies.

Point estimation for adaptive trial designs II: Practical considerations and guidance

In adaptive clinical trials, the conventional end-of-trial point estimate of a treatment effect is prone to bias, that is, a systematic tendency to deviate from its true value. As stated in recent FDA guidance on adaptive designs, it is desirable to report estimates of treatment effects that reduce or remove this bias. However, it may be unclear which of the available estimators are preferable, and their use remains rare in practice. This article is the second in a two-part series that studies the issue of bias in point estimation for adaptive trials. Part I provided a methodological review of approaches to remove or reduce the potential bias in point estimation for adaptive designs. In part II, we discuss how bias can affect standard estimators and assess the negative impact this can have. We review current practice for reporting point estimates and illustrate the computation of different estimators using a real adaptive trial example (including code), which we use as a basis for a simulation study. We show that while on average the values of these estimators can be similar, for a particular trial realization they can give noticeably different values for the estimated treatment effect. Finally, we propose guidelines for researchers around the choice of estimators and the reporting of estimates following an adaptive design. The issue of bias should be considered throughout the whole lifecycle of an adaptive design, with the estimation strategy prespecified in the statistical analysis plan. When available, unbiased or bias-reduced estimates are to be preferred.

Sensitivity Analyses of Clinical Trial Designs: Selecting Scenarios and Summarizing Operating Characteristics

The use of simulation-based sensitivity analyses is fundamental for evaluating and comparing candidate designs of future clinical trials. In this context, sensitivity analyses are especially useful to assess the dependence of important design operating characteristics with respect to various unknown parameters. Typical examples of operating characteristics include the likelihood of detecting treatment effects and the average study duration, which depend on parameters that are unknown until after the onset of the clinical study, such as the distributions of the primary outcomes and patient profiles. Two crucial components of sensitivity analyses are (i) the choice of a set of plausible simulation scenarios and (ii) the list of operating characteristics of interest. We propose a new approach for choosing the set of scenarios to be included in a sensitivity analysis. We maximize a utility criterion that formalizes whether a specific set of sensitivity scenarios is adequate to summarize how the operating characteristics of the trial design vary across plausible values of the unknown parameters. Then, we use optimization techniques to select the best set of simulation scenarios (according to the criteria specified by the investigator) to exemplify the operating characteristics of the trial design. We illustrate our proposal in three trial designs.

An overview of methodological considerations regarding adaptive stopping, arm dropping, and randomization in clinical trials

BACKGROUND AND OBJECTIVES

Adaptive features may increase flexibility and efficiency of clinical trials, and improve participants' chances of being allocated to better interventions. Our objective is to provide thorough guidance on key methodological considerations for adaptive clinical trials.

METHODS

We provide an overview of key methodological considerations for clinical trials employing adaptive stopping, adaptive arm dropping, and response-adaptive randomization. We cover pros and cons of different decisions and provide guidance on using simulation to compare different adaptive trial designs. We focus on Bayesian multi-arm adaptive trials, although the same general considerations apply to frequentist adaptive trials.

RESULTS

We provide guidance on 1) interventions and possible common control, 2) outcome selection, follow-up duration and model choice, 3) timing of adaptive analyses, 4) decision rules for adaptive stopping and arm dropping, 5) randomization strategies, 6) performance metrics, their prioritization, and arm selection strategies, and 7) simulations, assessment of performance under different scenarios, and reporting. Finally, we provide an example using a newly developed R simulation engine that may be used to evaluate and compare different adaptive trial designs.

CONCLUSION

This overview may help trialists design better and more transparent adaptive clinical trials and to adequately compare them before initiation.

Bayesian Methods in Human Drug and Biological Products Development in CDER and CBER

The Center for Drug Evaluation and Research (CDER) and the Center for Biologics Evaluation and Research (CBER) of the U.S. Food and Drug Administration (FDA) have been leaders in protecting and promoting the U.S. public health by helping to ensure that safe and effective drugs and biological products are available in the United States for those who need them. The null hypothesis significance testing approach, along with other considerations, is typically used to demonstrate the effectiveness of a drug or biological product. The Bayesian framework presents an alternative approach to demonstrate the effectiveness of a treatment. This article discusses the Bayesian framework for drug and biological product development, highlights key settings in which Bayesian approaches may be appropriate, and provides recent examples of the use of Bayesian approaches within CDER and CBER.

How to Use and Interpret the Results of a Platform Trial: Users' Guide to the Medical Literature

Platform trials are a type of randomized clinical trial that allow simultaneous comparison of multiple intervention groups against a single control group that serves as a common control based on a prespecified interim analysis plan. The platform trial design enables introduction of new interventions after the trial is initiated to evaluate multiple interventions in an ongoing manner using a single overarching protocol called a master (or core) protocol. When multiple treatment candidates are available, rapid scientific therapeutic discoveries may be made. Platform trials have important potential advantages in creating an efficient trial infrastructure that can help address critical clinical questions as the evidence evolves. Platform trials have recently been used in investigations of evolving therapies for patients with COVID-19. The purpose of this Users' Guide to the Medical Literature is to describe fundamental concepts of platform trials and master protocols and review issues in the conduct and interpretation of these studies. This Users' Guide is intended to help clinicians and readers understand articles reporting on interventions evaluated using platform trial designs.

Randomised trials at the level of the individual

In global health research, short-term, small-scale clinical trials with fixed, two-arm trial designs that generally do not allow for major changes throughout the trial are the most common study design. Building on the introductory paper of this Series, this paper discusses data-driven approaches to clinical trial research across several adaptive trial designs, as well as the master protocol framework that can help to harmonise clinical trial research efforts in global health research. We provide a general framework for more efficient trial research, and we discuss the importance of considering different study designs in the planning stage with statistical simulations. We conclude this second Series paper by discussing the methodological and operational complexity of adaptive trial designs and master protocols and the current funding challenges that could limit uptake of these approaches in global health research.

Planning for the Next Pandemic: Ethics and Innovation Today for Improved Clinical Trials Tomorrow

The coronavirus pandemic has brought public attention to the steps required to produce valid scientific clinical research in drug development. Traditional ethical principles that guide clinical research remain the guiding compass for physicians, patients, public health officials, investigators, drug developers and the public. Accelerating the process of delivering safe and effective treatments and vaccines against COVID-19 is a moral imperative. The apparent clash between the regulated system of phased randomized clinical trials and urgent public health need requires leveraging innovation with ethical scientific rigor. We reflect on the Belmont principles of autonomy, beneficence and justice as the pandemic unfolds, and illustrate the role of innovative clinical trial designs in alleviating pandemic challenges. Our discussion highlights selected types of innovative trial design and correlates them with ethical parameters and public health benefits. Details are provided for platform trials and other innovative designs such as basket and umbrella trials, designs leveraging external data sources, multi-stage seamless trials, preplanned control arm data sharing between larger trials, and higher order systems of linked trials coordinated more broadly between individual trials and phases of development, recently introduced conceptually as "PIPELINEs."

Opportunity for efficiency in clinical development: An overview of adaptive clinical trial designs and innovative machine learning tools, with examples from the cardiovascular field

Modern data analysis tools and statistical modeling techniques are increasingly used in clinical research to improve diagnosis, estimate disease progression and predict treatment outcomes. What seems less emphasized is the importance of the study design, which can have a serious impact on the study cost, time and statistical efficiency. This paper provides an overview of different types of adaptive designs in clinical trials and their applications to cardiovascular trials. We highlight recent proliferation of work on adaptive designs over the past two decades, including some recent regulatory guidelines on complex trial designs and master protocols. We also describe the increasing role of machine learning and use of metaheuristics to construct increasingly complex adaptive designs or to identify interesting features for improved predictions and classifications.

Pharmacology-Guided Rule-Based Adaptive Dose Escalation in First-in-Human Studies

First-in-human (FIH) studies typically progress through cohorts of fixed, standard size throughout the escalation scheme. This work presents and tests a pharmacology-guided rule-based adaptive dose escalation design that aims at making "best use" of participants in early clinical drug evaluation; it is paper based, not requiring real-time access to computational methods. The design minimizes the number of participants exposed to dose levels with low likelihood of being therapeutically relevant. Using criteria based on dose-limiting adverse event rate and on target exposure or target pharmacodynamics, the design increases the sample size when approaching the dose range of potential clinical relevance. The adaptive escalation design was retrospectively tested on actual data from a sample of 40 recently executed FIH studies with novel small and large molecules, and it was evaluated by simulating trials with three compounds with different therapeutic windows, i.e., representing a promising, unacceptable, and dubious profile. In retrospective evaluation of the adaptive escalation design, none of the cases overshot the actually reported top dose; one case resulted in a top dose that was within 20% under the estimated maximum tolerated dose in the original study. The median reduction of total number of participants per study was 38%. Trial simulations confirmed the retrospective evaluation, showing a similar performance of the adaptive escalation design compared with the conventional 6 + 2 design, at a reduced study size for compounds with a presumed acceptable therapeutic window. The adaptive escalation design was shown to make "best use" of participants in FIH studies without compromising safety.

Modified Goldilocks Design with strict type I error control in confirmatory clinical trials

Goldilocks Design (GD) utilizes predictive probability to adaptively select a trial's sample size based on accumulating data. In order to control type I error at a desired level for a subset of the null space, extensive simulations at the study design stage are required to choose critical values, which is a challenge for this type of Bayesian adaptive design to be used for confirmatory trials. In this article, we propose a Modified Goldilocks Design (MGD) where type I error is analytically controlled over the entire null space. We do so by applying the conditional invariance principle and a combination test approach on [Formula: see text]-values that are obtained from independent cohorts of subjects. Simulation studies show that despite analytic control of type I error rate, the proposed MGD has similar power when compared with the original GD. We further apply it to an example trial with time-to-event endpoint in oncology.

A roadmap to using historical controls in clinical trials - by Drug Information Association Adaptive Design Scientific Working Group (DIA-ADSWG)

Historical controls (HCs) can be used for model parameter estimation at the study design phase, adaptation within a study, or supplementation or replacement of a control arm. Currently on the latter, there is no practical roadmap from design to analysis of a clinical trial to address selection and inclusion of HCs, while maintaining scientific validity. This paper provides a comprehensive roadmap for planning, conducting, analyzing and reporting of studies using HCs, mainly when a randomized clinical trial is not possible. We review recent applications of HC in clinical trials, in which either predominantly a large treatment effect overcame concerns about bias, or the trial targeted a life-threatening disease with no treatment options. In contrast, we address how the evidentiary standard of a trial can be strengthened with optimized study designs and analysis strategies, emphasizing rare and pediatric indications. We highlight the importance of simulation and sensitivity analyses for estimating the range of uncertainties in the estimation of treatment effect when traditional randomization is not possible. Overall, the paper provides a roadmap for using HCs.