When should factorial designs be used for late-phase randomised controlled trials?

BACKGROUND

A 2×2 factorial design evaluates two interventions (A versus control and B versus control) by randomising to control, A-only, B-only or both A and B together. Extended factorial designs are also possible (e.g. 3×3 or 2×2×2). Factorial designs often require fewer resources and participants than alternative randomised controlled trials, but they are not widely used. We identified several issues that investigators considering this design need to address, before they use it in a late-phase setting.

METHODS

We surveyed journal articles published in 2000-2022 relating to designing factorial randomised controlled trials. We identified issues to consider based on these and our personal experiences.

RESULTS

We identified clinical, practical, statistical and external issues that make factorial randomised controlled trials more desirable. Clinical issues are (1) interventions can be easily co-administered; (2) risk of safety issues from co-administration above individual risks of the separate interventions is low; (3) safety or efficacy data are wanted on the combination intervention; (4) potential for interaction (e.g. effect of A differing when B administered) is low; (5) it is important to compare interventions with other interventions balanced, rather than allowing randomised interventions to affect the choice of other interventions; (6) eligibility criteria for different interventions are similar. Practical issues are (7) recruitment is not harmed by testing many interventions; (8) each intervention and associated toxicities is unlikely to reduce either adherence to the other intervention or overall follow-up; (9) blinding is easy to implement or not required. Statistical issues are (10) a suitable scale of analysis can be identified; (11) adjustment for multiplicity is not required; (12) early stopping for efficacy or lack of benefit can be done effectively. External issues are (13) adequate funding is available and (14) the trial is not intended for licensing purposes. An overarching issue (15) is that factorial design should give a lower sample size requirement than alternative designs. Across designs with varying non-adherence, retention, intervention effects and interaction effects, 2×2 factorial designs require lower sample size than a three-arm alternative when one intervention effect is reduced by no more than 24%-48% in the presence of the other intervention compared with in the absence of the other intervention.

CONCLUSIONS

Factorial designs are not widely used and should be considered more often using our issues to consider. Low potential for at most small to modest interaction is key, for example, where the interventions have different mechanisms of action or target different aspects of the disease being studied.

Medicines and Healthcare products Regulatory Agency's "Consultation on proposals for legislative changes for clinical trials": a response from the Trials Methodology Research Partnership Adaptive Designs Working Group, with a focus on data sharing

In the UK, the Medicines and Healthcare products Regulatory Agency consulted on proposals "to improve and strengthen the UK clinical trials legislation to help us make the UK the best place to research and develop safe and innovative medicines". The purpose of the consultation was to help finalise the proposals and contribute to the drafting of secondary legislation. We discussed these proposals as members of the Trials Methodology Research Partnership Adaptive Designs Working Group, which is jointly funded by the Medical Research Council and the National Institute for Health and Care Research. Two topics arose frequently in the discussion: the emphasis on legislation, and the absence of questions on data sharing. It is our opinion that the proposals rely heavily on legislation to change practice. However, clinical trials are heterogeneous, and as a result some trials will struggle to comply with all of the proposed legislation. Furthermore, adaptive design clinical trials are even more heterogeneous than their non-adaptive counterparts, and face more challenges. Consequently, it is possible that increased legislation could have a greater negative impact on adaptive designs than non-adaptive designs. Overall, we are sceptical that the introduction of legislation will achieve the desired outcomes, with some exceptions. Meanwhile the topic of data sharing - making anonymised individual-level clinical trial data available to other investigators for further use - is entirely absent from the proposals and the consultation in general. However, as an aspect of the wider concept of open science and reproducible research, data sharing is an increasingly important aspect of clinical trials. The benefits of data sharing include faster innovation, improved surveillance of drug safety and effectiveness and decreasing participant exposure to unnecessary risk. There are already a number of UK-focused documents that discuss and encourage data sharing, for example, the Concordat on Open Research Data and the Medical Research Council's Data Sharing Policy. We strongly suggest that data sharing should be the norm rather than the exception, and hope that the forthcoming proposals on clinical trials invite discussion on this important topic.

Point estimation for adaptive trial designs II: Practical considerations and guidance

In adaptive clinical trials, the conventional end-of-trial point estimate of a treatment effect is prone to bias, that is, a systematic tendency to deviate from its true value. As stated in recent FDA guidance on adaptive designs, it is desirable to report estimates of treatment effects that reduce or remove this bias. However, it may be unclear which of the available estimators are preferable, and their use remains rare in practice. This article is the second in a two-part series that studies the issue of bias in point estimation for adaptive trials. Part I provided a methodological review of approaches to remove or reduce the potential bias in point estimation for adaptive designs. In part II, we discuss how bias can affect standard estimators and assess the negative impact this can have. We review current practice for reporting point estimates and illustrate the computation of different estimators using a real adaptive trial example (including code), which we use as a basis for a simulation study. We show that while on average the values of these estimators can be similar, for a particular trial realization they can give noticeably different values for the estimated treatment effect. Finally, we propose guidelines for researchers around the choice of estimators and the reporting of estimates following an adaptive design. The issue of bias should be considered throughout the whole lifecycle of an adaptive design, with the estimation strategy prespecified in the statistical analysis plan. When available, unbiased or bias-reduced estimates are to be preferred.

Treatment selection in multi-arm multi-stage designs: With application to a postpartum haemorrhage trial

BACKGROUND

Multi-arm multi-stage trials are an efficient, adaptive approach for testing many treatments simultaneously within one protocol. In settings where numbers of patients available to be entered into trials and resources might be limited, such as primary postpartum haemorrhage, it may be necessary to select a pre-specified subset of arms at interim stages even if they are all showing some promise against the control arm. This will put a limit on the maximum number of patients required and reduce the associated costs. Motivated by the World Health Organization Refractory HaEmorrhage Devices trial in postpartum haemorrhage, we explored the properties of such a selection design in a randomised phase III setting and compared it with other alternatives. The objectives are: (1) to investigate how the timing of treatment selection affects the operating characteristics; (2) to explore the use of an information-rich (continuous) intermediate outcome to select the best-performing arm, out of four treatment arms, compared with using the primary (binary) outcome for selection at the interim stage; and (3) to identify factors that can affect the efficiency of the design.

METHODS

We conducted simulations based on the refractory haemorrhage devices multi-arm multi-stage selection trial to investigate the impact of the timing of treatment selection and applying an adaptive allocation ratio on the probability of correct selection, overall power and familywise type I error rate. Simulations were also conducted to explore how other design parameters will affect both the maximum sample size and trial timelines.

RESULTS

The results indicate that the overall power of the trial is bounded by the probability of 'correct' selection at the selection stage. The results showed that good operating characteristics are achieved if the treatment selection is conducted at around 17% of information time. Our results also showed that although randomising more patients to research arms before selection will increase the probability of selecting correctly, this will not increase the overall efficiency of the (selection) design compared with the fixed allocation ratio of 1:1 to all arms throughout.

CONCLUSIONS

Multi-arm multi-stage selection designs are efficient and flexible with desirable operating characteristics. We give guidance on many aspects of these designs including selecting the intermediate outcome measure, the timing of treatment selection, and choosing the operating characteristics.

Combining factorial and multi-arm multi-stage platform designs to evaluate multiple interventions efficiently

BACKGROUND

Factorial-MAMS design platform designs have many advantages, but the practical advantages and disadvantages of combining the two designs have not been explored.

METHODS

We propose practical methods for a combined design within the platform trial paradigm where some interventions are not expected to interact and could be given together.

RESULTS

We describe the combined design and suggest diagrams that can be used to represent it. Many properties are common both to standard factorial designs, including the need to consider interactions between interventions and the impact of intervention efficacy on power of other comparisons, and to standard multi-arm multi-stage designs, including the need to pre-specify procedures for starting and stopping intervention comparisons. We also identify some specific features of the factorial-MAMS design: timing of interim and final analyses should be determined by calendar time or total observed events; some non-factorial modifications may be useful; eligibility criteria should be broad enough to include any patient eligible for any part of the randomisation; stratified randomisation may conveniently be performed sequentially; and analysis requires special care to use only concurrent controls.

CONCLUSION

A combined factorial-MAMS design can combine the efficiencies of factorial trials and multi-arm multi-stage platform trials. It allows us to address multiple research questions under one protocol and to test multiple new treatment options, which is particularly important when facing a new emergent infection such as COVID-19.

RAMPART: A model for a regulatory-ready academic-led phase III trial in the adjuvant renal cell carcinoma setting

The development of therapeutics in oncology is a highly active research area for the pharmaceutical and biotechnology industries, but also has a strong academic base. Many new agents have been developed in recent years, most with specific biological targets. This has mandated the need to look at different ways to streamline the evaluation of new agents. One solution has been the development of adaptive trial designs that allow the evaluation of multiple agents, concentrating on the most promising agents while screening out those which are unlikely to benefit patients. Another way forward has been the growth of partnerships between academia and industry with the shared goal of designing and conducting high quality clinical trials which answer important clinical questions as efficiently as possible. The RAMPART trial (NCT03288532) brings together both of these processes in an attempt to improve outcomes for patients with locally advanced renal cell carcinoma (RCC), where no globally acceptable adjuvant strategy after nephrectomy currently exist. RAMPART is led by the MRC CTU at University College London (UCL), in collaboration with other international academic groups and industry. We aim to facilitate the use of data from RAMPART, (dependent on outcomes), for a future regulatory submission that will extend the license of the agents being investigated. We share our experience in order to lay the foundations for an effective trial design and conduct framework and to guide others who may be considering similar collaborations. Trial Registration: ISRCTN #: ISRCTN53348826, NCT #: NCT03288532, EUDRACT #: 2017-002329-39. CTA #: 20363/0380/001-0001. MREC #: 17/LO/1875. ClinicalTrials.gov Identifier: NCT03288532 RAMPART grant number: MC_UU_12023/25. . RAMPART Protocol version 5.0.

RAMPART: A phase III multi-arm multi-stage trial of adjuvant checkpoint inhibitors in patients with resected primary renal cell carcinoma (RCC) at high or intermediate risk of relapse

BACKGROUND

20-60% of patients with initially locally advanced Renal Cell Carcinoma (RCC) develop metastatic disease despite optimal surgical excision. Adjuvant strategies have been tested in RCC including cytokines, radiotherapy, hormones and oral tyrosine-kinase inhibitors (TKIs), with limited success. The predominant global standard-of-care after nephrectomy remains active monitoring. Immune checkpoint inhibitors (ICIs) are effective in the treatment of metastatic RCC; RAMPART will investigate these agents in the adjuvant setting.

METHODS/DESIGN

RAMPART is an international, UK-led trial investigating the addition of ICIs after nephrectomy in patients with resected locally advanced RCC. RAMPART is a multi-arm multi-stage (MAMS) platform trial, upon which additional research questions may be addressed over time. The target population is patients with histologically proven resected locally advanced RCC (clear cell and non-clear cell histological subtypes), with no residual macroscopic disease, who are at high or intermediate risk of relapse (Leibovich score 3-11). Patients with fully resected synchronous ipsilateral adrenal metastases are included. Participants are randomly assigned (3,2:2) to Arm A - active monitoring (no placebo) for one year, Arm B - durvalumab (PD-L1 inhibitor) 4-weekly for one year; or Arm C - combination therapy with durvalumab 4-weekly for one year plus two doses of tremelimumab (CTLA-4 inhibitor) at day 1 of the first two 4-weekly cycles. The co-primary outcomes are disease-free-survival (DFS) and overall survival (OS). Secondary outcomes include safety, metastasis-free survival, RCC specific survival, quality of life, and patient and clinician preferences. Tumour tissue, plasma and urine are collected for molecular analysis (TransRAMPART).

TRIAL REGISTRATION

ISRCTN #: ISRCTN53348826, NCT #: NCT03288532, EUDRACT #: 2017-002329-39, CTA #: 20363/0380/001-0001, MREC #: 17/LO/1875, ClinicalTrials.gov Identifier: NCT03288532, RAMPART grant number: MC_UU_12023/25, TransRAMPART grant number: A28690 Cancer Research UK, RAMPART Protocol version 5.0.

Adding new experimental arms to randomised clinical trials: Impact on error rates

BACKGROUND

Experimental treatments pass through various stages of development. If a treatment passes through early-phase experiments, the investigators may want to assess it in a late-phase randomised controlled trial. An efficient way to do this is adding it as a new research arm to an ongoing trial while the existing research arms continue, a so-called multi-arm platform trial. The familywise type I error rate is often a key quantity of interest in any multi-arm platform trial. We set out to clarify how it should be calculated when new arms are added to a trial some time after it has started.

METHODS

We show how the familywise type I error rate, any-pair and all-pairs powers can be calculated when a new arm is added to a platform trial. We extend the Dunnett probability and derive analytical formulae for the correlation between the test statistics of the existing pairwise comparison and that of the newly added arm. We also verify our analytical derivation via simulations.

RESULTS

Our results indicate that the familywise type I error rate depends on the shared control arm information (i.e. individuals in continuous and binary outcomes and primary outcome events in time-to-event outcomes) from the common control arm patients and the allocation ratio. The familywise type I error rate is driven more by the number of pairwise comparisons and the corresponding (pairwise) type I error rates than by the timing of the addition of the new arms. The familywise type I error rate can be estimated using Šidák's correction if the correlation between the test statistics of pairwise comparisons is less than 0.30.

CONCLUSIONS

The findings we present in this article can be used to design trials with pre-planned deferred arms or to add new pairwise comparisons within an ongoing platform trial where control of the pairwise error rate or familywise type I error rate (for a subset of pairwise comparisons) is required.

Proposals on Kaplan-Meier plots in medical research and a survey of stakeholder views: KMunicate

OBJECTIVES

To examine reactions to the proposed improvements to standard Kaplan-Meier plots, the standard way to present time-to-event data, and to understand which (if any) facilitated better depiction of (1) the state of patients over time, and (2) uncertainty over time in the estimates of survival.

DESIGN

A survey of stakeholders' opinions on the proposals.

SETTING

A web-based survey, open to international participation, for those with an interest in visualisation of time-to-event data.

PARTICIPANTS

1174 people participated in the survey over a 6-week period. Participation was global (although primarily Europe and North America) and represented a wide range of researchers (primarily statisticians and clinicians).

MAIN OUTCOME MEASURES

Two outcome measures were of principal importance: (1) participants' opinions of each proposal compared with a 'standard' Kaplan-Meier plot; and (2) participants' overall ranking of the proposals (including the standard).

RESULTS

Most proposals were more popular than the standard Kaplan-Meier plot. The most popular proposals in the two categories, respectively, were an extended table beneath the plot depicting the numbers at risk, censored and having experienced an event at periodic timepoints, and CIs around each Kaplan-Meier curve.

CONCLUSIONS

This study produced a high response number, reflecting the importance of graphics for time-to-event data. Those producing and publishing Kaplan-Meier plots-both authors and journals-should, as a starting point, consider using the combination of the two favoured proposals.

Combined test versus logrank/Cox test in 50 randomised trials

BACKGROUND

The logrank test and the Cox proportional hazards model are routinely applied in the design and analysis of randomised controlled trials (RCTs) with time-to-event outcomes. Usually, sample size and power calculations assume proportional hazards (PH) of the treatment effect, i.e. the hazard ratio is constant over the entire follow-up period. If the PH assumption fails, the power of the logrank/Cox test may be reduced, sometimes severely. It is, therefore, important to understand how serious this can become in real trials, and for a proven, alternative test to be available to increase the robustness of the primary test.

METHODS

We performed a systematic search to identify relevant articles in four leading medical journals that publish results of phase 3 clinical trials. Altogether, 50 articles satisfied our inclusion criteria. We digitised published Kaplan-Meier curves and created approximations to the original times to event or censoring at the individual patient level. Using the reconstructed data, we tested for non-PH in all 50 trials. We compared the results from the logrank/Cox test with those from the combined test recently proposed by Royston and Parmar.

RESULTS

The PH assumption was checked and reported only in 28% of the studies. Evidence of non-PH at the 0.10 level was detected in 31% of comparisons. The Cox test of the treatment effect was significant at the 0.05 level in 49% of comparisons, and the combined test in 55%. In four of five trials with discordant results, the interpretation would have changed had the combined test been used. The degree of non-PH and the dominance of the p value for the combined test were strongly associated. Graphical investigation suggested that non-PH was mostly due to a treatment effect manifesting in an early follow-up and disappearing later.

CONCLUSIONS

The evidence for non-PH is checked (and, hence, identified) in only a small minority of RCTs, but non-PH may be present in a substantial fraction of such trials. In our reanalysis of the reconstructed data from 50 trials, the combined test outperformed the Cox test overall. The combined test is a promising approach to making trial design and analysis more robust.

Assessing the impact of efficacy stopping rules on the error rates under the multi-arm multi-stage framework

BACKGROUND

The multi-arm multi-stage framework uses intermediate outcomes to assess lack-of-benefit of research arms at interim stages in randomised trials with time-to-event outcomes. However, the design lacks formal methods to evaluate early evidence of overwhelming efficacy on the definitive outcome measure. We explore the operating characteristics of this extension to the multi-arm multi-stage design and how to control the pairwise and familywise type I error rate. Using real examples and the updated nstage program, we demonstrate how such a design can be developed in practice.

METHODS

We used the Dunnett approach for assessing treatment arms when conducting comprehensive simulation studies to evaluate the familywise error rate, with and without interim efficacy looks on the definitive outcome measure, at the same time as the planned lack-of-benefit interim analyses on the intermediate outcome measure. We studied the effect of the timing of interim analyses, allocation ratio, lack-of-benefit boundaries, efficacy rule, number of stages and research arms on the operating characteristics of the design when efficacy stopping boundaries are incorporated. Methods for controlling the familywise error rate with efficacy looks were also addressed.

RESULTS

Incorporating Haybittle-Peto stopping boundaries on the definitive outcome at the interim analyses will not inflate the familywise error rate in a multi-arm design with two stages. However, this rule is conservative; in general, more liberal stopping boundaries can be used with minimal impact on the familywise error rate. Efficacy bounds in trials with three or more stages using an intermediate outcome may inflate the familywise error rate, but we show how to maintain strong control.

CONCLUSION

The multi-arm multi-stage design allows stopping for both lack-of-benefit on the intermediate outcome and efficacy on the definitive outcome at the interim stages. We provide guidelines on how to control the familywise error rate when efficacy boundaries are implemented in practice.

Adaptive designs in clinical trials: why use them, and how to run and report them

Adaptive designs can make clinical trials more flexible by utilising results accumulating in the trial to modify the trial's course in accordance with pre-specified rules. Trials with an adaptive design are often more efficient, informative and ethical than trials with a traditional fixed design since they often make better use of resources such as time and money, and might require fewer participants. Adaptive designs can be applied across all phases of clinical research, from early-phase dose escalation to confirmatory trials. The pace of the uptake of adaptive designs in clinical research, however, has remained well behind that of the statistical literature introducing new methods and highlighting their potential advantages. We speculate that one factor contributing to this is that the full range of adaptations available to trial designs, as well as their goals, advantages and limitations, remains unfamiliar to many parts of the clinical community. Additionally, the term adaptive design has been misleadingly used as an all-encompassing label to refer to certain methods that could be deemed controversial or that have been inadequately implemented.We believe that even if the planning and analysis of a trial is undertaken by an expert statistician, it is essential that the investigators understand the implications of using an adaptive design, for example, what the practical challenges are, what can (and cannot) be inferred from the results of such a trial, and how to report and communicate the results. This tutorial paper provides guidance on key aspects of adaptive designs that are relevant to clinical triallists. We explain the basic rationale behind adaptive designs, clarify ambiguous terminology and summarise the utility and pitfalls of adaptive designs. We discuss practical aspects around funding, ethical approval, treatment supply and communication with stakeholders and trial participants. Our focus, however, is on the interpretation and reporting of results from adaptive design trials, which we consider vital for anyone involved in medical research. We emphasise the general principles of transparency and reproducibility and suggest how best to put them into practice.

Testing many treatments within a single protocol over 10 years at MRC Clinical Trials Unit at UCL: Multi-arm, multi-stage platform, umbrella and basket protocols

There is real need to change how we do some of our clinical trials, as currently the testing and development process is too slow, too costly and too failure-prone often we find that a new treatment is no better than the current standard. Much of the focus on the development and testing pathway has been in improving the design of phase I and II trials. In this article, we present examples of new methods for improving the design of phase III trials (and the necessary lead up to them) as they are the most time-consuming and expensive part of the pathway. Key to all these methods is the aim to test many treatments and/or pose many therapeutic questions within one protocol.

Review and evaluation of performance measures for survival prediction models in external validation settings

Background

When developing a prediction model for survival data it is essential to validate its performance in external validation settings using appropriate performance measures. Although a number of such measures have been proposed, there is only limited guidance regarding their use in the context of model validation. This paper reviewed and evaluated a wide range of performance measures to provide some guidelines for their use in practice.

Methods

An extensive simulation study based on two clinical datasets was conducted to investigate the performance of the measures in external validation settings. Measures were selected from categories that assess the overall performance, discrimination and calibration of a survival prediction model. Some of these have been modified to allow their use with validation data, and a case study is provided to describe how these measures can be estimated in practice. The measures were evaluated with respect to their robustness to censoring and ease of interpretation. All measures are implemented, or are straightforward to implement, in statistical software.

Results

Most of the performance measures were reasonably robust to moderate levels of censoring. One exception was Harrell’s concordance measure which tended to increase as censoring increased.

Conclusions

We recommend that Uno’s concordance measure is used to quantify concordance when there are moderate levels of censoring. Alternatively, Gönen and Heller’s measure could be considered, especially if censoring is very high, but we suggest that the prediction model is re-calibrated first. We also recommend that Royston’s D is routinely reported to assess discrimination since it has an appealing interpretation. The calibration slope is useful for both internal and external validation settings and recommended to report routinely. Our recommendation would be to use any of the predictive accuracy measures and provide the corresponding predictive accuracy curves. In addition, we recommend to investigate the characteristics of the validation data such as the level of censoring and the distribution of the prognostic index derived in the validation setting before choosing the performance measures.

Type I error rates of multi-arm multi-stage clinical trials: strong control and impact of intermediate outcomes

BACKGROUND

The multi-arm multi-stage (MAMS) design described by Royston et al. [Stat Med. 2003;22(14):2239-56 and Trials. 2011;12:81] can accelerate treatment evaluation by comparing multiple treatments with a control in a single trial and stopping recruitment to arms not showing sufficient promise during the course of the study. To increase efficiency further, interim assessments can be based on an intermediate outcome (I) that is observed earlier than the definitive outcome (D) of the study. Two measures of type I error rate are often of interest in a MAMS trial. Pairwise type I error rate (PWER) is the probability of recommending an ineffective treatment at the end of the study regardless of other experimental arms in the trial. Familywise type I error rate (FWER) is the probability of recommending at least one ineffective treatment and is often of greater interest in a study with more than one experimental arm.

METHODS

We demonstrate how to calculate the PWER and FWER when the I and D outcomes in a MAMS design differ. We explore how each measure varies with respect to the underlying treatment effect on I and show how to control the type I error rate under any scenario. We conclude by applying the methods to estimate the maximum type I error rate of an ongoing MAMS study and show how the design might have looked had it controlled the FWER under any scenario.

RESULTS

The PWER and FWER converge to their maximum values as the effectiveness of the experimental arms on I increases. We show that both measures can be controlled under any scenario by setting the pairwise significance level in the final stage of the study to the target level. In an example, controlling the FWER is shown to increase considerably the size of the trial although it remains substantially more efficient than evaluating each new treatment in separate trials.

CONCLUSIONS

The proposed methods allow the PWER and FWER to be controlled in various MAMS designs, potentially increasing the uptake of the MAMS design in practice. The methods are also applicable in cases where the I and D outcomes are identical.

Quantifying the uptake of user-written commands over time

A major factor in the uptake of new statistical methods is the availability of user-friendly software implementations. One attractive feature of Stata is that users can write their own commands and release them to other users via Statistical Software Components at Boston College. Authors of statistical programs do not always get adequate credit, because programs are rarely cited properly. There is no obvious measure of a program's impact, but researchers are under increasing pressure to demonstrate the impact of their work to funders. In addition to encouraging proper citation of software, the number of downloads of a user-written package can be regarded as a measure of impact over time. In this article, we explain how such information can be accessed for any month from July 2007 and summarized using the new ssccount command.

The extension of total gain (TG) statistic in survival models: properties and applications

BACKGROUND

The results of multivariable regression models are usually summarized in the form of parameter estimates for the covariates, goodness-of-fit statistics, and the relevant p-values. These statistics do not inform us about whether covariate information will lead to any substantial improvement in prediction. Predictive ability measures can be used for this purpose since they provide important information about the practical significance of prognostic factors. R (2)-type indices are the most familiar forms of such measures in survival models, but they all have limitations and none is widely used.

METHODS

In this paper, we extend the total gain (TG) measure, proposed for a logistic regression model, to survival models and explore its properties using simulations and real data. TG is based on the binary regression quantile plot, otherwise known as the predictiveness curve. Standardised TG ranges from 0 (no explanatory power) to 1 ('perfect' explanatory power).

RESULTS

The results of our simulations show that unlike many of the other R (2)-type predictive ability measures, TG is independent of random censoring. It increases as the effect of a covariate increases and can be applied to different types of survival models, including models with time-dependent covariate effects. We also apply TG to quantify the predictive ability of multivariable prognostic models developed in several disease areas.

CONCLUSIONS

Overall, TG performs well in our simulation studies and can be recommended as a measure to quantify the predictive ability in survival models.

Measuring the impact of methodological research: a framework and methods to identify evidence of impact

Background

Providing evidence of impact highlights the benefits of medical research to society. Such evidence is increasingly requested by research funders and commonly relies on citation analysis. However, other indicators may be more informative. Although frameworks to demonstrate the impact of clinical research have been reported, no complementary framework exists for methodological research. Therefore, we assessed the impact of methodological research projects conducted or completed between 2009 and 2012 at the UK Medical Research Council Clinical Trials Unit Hub for Trials Methodology Research Hub, with a view to developing an appropriate framework.

Methods

Various approaches to the collection of data on research impact were employed. Citation rates were obtained using Web of Science (http://www.webofknowledge.com/) and analyzed descriptively. Semistructured interviews were conducted to obtain information on the rates of different types of research output that indicated impact for each project. Results were then pooled across all projects. Finally, email queries pertaining to methodology projects were collected retrospectively and their content analyzed.

Results

Simple citation analysis established the citation rates per year since publication for 74 methodological publications; however, further detailed analysis revealed more about the potential influence of these citations. Interviews that spanned 20 individual research projects demonstrated a variety of types of impact not otherwise collated, for example, applications and further developments of the research; release of software and provision of guidance materials to facilitate uptake; formation of new collaborations and broad dissemination. Finally, 194 email queries relating to 6 methodological projects were received from 170 individuals across 23 countries. They provided further evidence that the methodologies were impacting on research and research practice, both nationally and internationally. We have used the information gathered in this study to adapt an existing framework for impact of clinical research for use in methodological research.

Conclusions

Gathering evidence on research impact of methodological research from a variety of sources has enabled us to obtain multiple indicators and thus to demonstrate broad impacts of methodological research. The adapted framework developed can be applied to future methodological research and thus provides a tool for methodologists to better assess and report research impacts.

Electronic supplementary material

The online version of this article (doi:10.1186/1745-6215-15-464) contains supplementary material, which is available to authorized users.

Impact of lack-of-benefit stopping rules on treatment effect estimates of two-arm multi-stage (TAMS) trials with time to event outcome

Background

In 2011, Royston et al. described technical details of a two-arm, multi-stage (TAMS) design. The design enables a trial to be stopped part-way through recruitment if the accumulating data suggests a lack of benefit of the experimental arm. Such interim decisions can be made using data on an available ‘intermediate’ outcome. At the conclusion of the trial, the definitive outcome is analyzed. Typical intermediate and definitive outcomes in cancer might be progression-free and overall survival, respectively. In TAMS designs, the stopping rule applied at the interim stage(s) affects the sampling distribution of the treatment effect estimator, potentially inducing bias that needs addressing.

Methods

We quantified the bias in the treatment effect estimator in TAMS trials according to the size of the treatment effect and for different designs. We also retrospectively ‘redesigned’ completed cancer trials as TAMS trials and used the bootstrap to quantify bias.

Results

In trials in which the experimental treatment is better than the control and which continue to their planned end, the bias in the estimate of treatment effect is small and of no practical importance. In trials stopped for lack of benefit at an interim stage, the treatment effect estimate is biased at the time of interim assessment. This bias is markedly reduced by further patient follow-up and reanalysis at the planned ‘end’ of the trial.

Conclusions

Provided that all patients in a TAMS trial are followed up to the planned end of the trial, the bias in the estimated treatment effect is of no practical importance. Bias correction is then unnecessary.

A simulation study of predictive ability measures in a survival model II: explained randomness and predictive accuracy

Several R(2) -type measures have been proposed to evaluate the predictive ability of a survival model. In Part I, we classified the measures into four categories and studied the measures in the explained variation category. In this paper, we study the remaining measures in a similar fashion, discussing their strengths and shortcomings. Simulation studies are used to examine the performance of the measures with respect to the criteria we set out in Part I. Our simulation studies showed that among the measures studied in this paper, the measures proposed by Kent and O'Quigley ρ(W)(2) (and its approximation ρ(W,A)(2)) and Schemper and Kaider R(SK)(2) perform better with respect to our criteria. However, our investigations showed that ρ(W)(2) is adversely affected by the distribution of covariate and the presence of influential observations. The results show that the other measures perform poorly, primarily because they are affected either by the degree of censoring or the follow-up period.

A simulation study of predictive ability measures in a survival model I: explained variation measures

Measures of predictive ability play an important role in quantifying the clinical significance of prognostic factors. Several measures have been proposed to evaluate the predictive ability of survival models in the last two decades, but no single measure is consistently used. The proposed measures can be classified into the following categories: explained variation, explained randomness, and predictive accuracy. The three categories are conceptually different and are based on different principles. Several new measures have been proposed since Schemper and Stare's study in 1996 on some of the existing measures. This paper is the first of two papers that study the proposed measures systematically by applying a set of criteria that a measure of predictive ability should possess in the context of survival analysis. The present paper focuses on the explained variation category, and part II studies the proposed measures in the other categories. Simulation studies are used to examine the performance of five explained variation measures with respect to these criteria, discussing their strengths and shortcomings. Our simulation studies show that the measures proposed by Kent and O'Quigley, R(PM)(2), and Royston and Sauerbrei, R(D)(2), appear to be the best overall at quantifying predictive ability. However, it should be noted that neither measure is perfect; R(PM)(2) is sensitive to outliers and R(D)(2) to (marked) non-normality of the distribution of the prognostic index. The results show that the other measures perform poorly, primarily because they are adversely affected by censoring.

Designs for clinical trials with time-to-event outcomes based on stopping guidelines for lack of benefit

background

The pace of novel medical treatments and approaches to therapy has accelerated in recent years. Unfortunately, many potential therapeutic advances do not fulfil their promise when subjected to randomized controlled trials. It is therefore highly desirable to speed up the process of evaluating new treatment options, particularly in phase II and phase III trials. To help realize such an aim, in 2003, Royston and colleagues proposed a class of multi-arm, two-stage trial designs intended to eliminate poorly performing contenders at a first stage (point in time). Only treatments showing a predefined degree of advantage against a control treatment were allowed through to a second stage. Arms that survived the first-stage comparison on an intermediate outcome measure entered a second stage of patient accrual, culminating in comparisons against control on the definitive outcome measure. The intermediate outcome is typically on the causal pathway to the definitive outcome (i.e. the features that cause an intermediate event also tend to cause a definitive event), an example in cancer being progression-free and overall survival. Although the 2003 paper alluded to multi-arm trials, most of the essential design features concerned only two-arm trials. Here, we extend the two-arm designs to allow an arbitrary number of stages, thereby increasing flexibility by building in several 'looks' at the accumulating data. Such trials can terminate at any of the intermediate stages or the final stage.

Methods

We describe the trial design and the mathematics required to obtain the timing of the 'looks' and the overall significance level and power of the design. We support our results by extensive simulation studies. As an example, we discuss the design of the STAMPEDE trial in prostate cancer.

Results

The mathematical results on significance level and power are confirmed by the computer simulations. Our approach compares favourably with methodology based on beta spending functions and on monitoring only a primary outcome measure for lack of benefit of the new treatment.

Conclusions

The new designs are practical and are supported by theory. They hold considerable promise for speeding up the evaluation of new treatments in phase II and III trials.

The epidemiology and treatment of mesothelioma in South East England 1985-2002

OBJECTIVES

To describe trends in the incidence of mesothelioma for men and women in South East England and the geographical variation at the level of primary care trust. To describe treatment patterns by cancer network of residence, and relative survival by cancer network, disease stage and treatment modality.

METHODS

5753 cases were extracted from the Thames Cancer Registry database. We calculated age standardised incidence rates for each year, age specific incidence rates in 10 year age groups, and we used linear regression to compute the average annual percentage change in age standardised incidence. We used Poisson regression to analyse generational trends in incidence.

RESULTS

Men had five times higher incidence of mesothelioma than women. In men, there was an overall 4% increase per year between 1985 and 2002. Over the same period, the overall increase in incidence for women was 5% per year. The incidence was highest in men aged over 70 years, and men aged over 80 years had the highest increase of 8% per year. The incidence rate ratio increased for men born between 1892 and 1942 and started to slow for those born from 1947 onwards. Areas along the Thames and its estuary had the highest incidence. There was some variation by cancer network in the proportion of patients receiving cancer surgery, radiotherapy and chemotherapy. There were no discernable differences in relative survival by cancer network of residence or disease stage but those receiving combined treatment had higher 5 year survival.

CONCLUSIONS

Mesothelioma incidence has increased in South East England, particularly for men aged over 70 years. The highest incidence occurs along the Thames and its estuary, reflecting areas of asbestos use in shipbuilding and industry in the past. More research is needed to understand the interrelationships of prognostic factors, treatment choices and survival, and to determine the best care and support for these patients and their families.

Colorectal cancer incidence, mortality and survival in South-east England between 1972 and 2001

We report incidence, mortality and survival from colorectal cancer in South-east England using data from 162,022 incident cases and 97,697 deaths collected between 1972 and 2001 at the Thames Cancer Registry, which currently covers 14 million people. Overall, there was an increase in the incidence of colorectal cancer among men aged 50 years and over, and a decrease among the youngest age groups. In women, there was a clear decrease in incidence among those aged less than 60 years but a slight increase among those aged 60-79 years. Furthermore, there has been a steady decrease in mortality for all ages, larger in women than in men, and an increase in the 10-year relative survival for both sexes from just over 30% among those followed-up during 1981-1986 to just over 45% among those followed-up during 1997-2001.