Treatment estimands in clinical trials of patients hospitalised for COVID-19: ensuring trials ask the right questions

Core outcome sets in cancer clinical trials: current status and future opportunities-an EORTC perspective

BACKGROUND

Inconsistent, varied and selective outcome reporting is problematic in clinical trials. Core outcome sets (COS) standardise the outcomes that should be measured and reported in all trials in a specific area of health or health care. We reviewed available cancer COS and assessed their uptake in cancer clinical trials through surveying members of the European Organisation for Research and Treatment of Cancer (EORTC).

METHODS

This study employs an exploratory cross-sectional design across two phases. The Core Outcome Measures in Effectiveness Trials (COMET) Initiative database was searched for cancer-specific COS on June 1st, 2023. Awareness and use of COS amongst EORTC trialists was assessed in November 2023 via an online survey.

RESULTS

We identified a total of 85 cancer-related COS on the COMET database. Of these, 69 related to the tumour types as categorised by the EORTC and their disease orientated groups. A total of 710 EORTC members responded, of whom half (50%) stated they were unfamiliar with COS. Relevant COS were available to over a quarter of respondents, with a tenth utilising available COS. Those who chose not to use an available COS cited volume of outcomes, lack of time and infrastructure for implementation as key barriers.

CONCLUSIONS

While COS are becoming increasingly available to, and acknowledged by, cancer clinical trialists, their implementation is currently still limited. Our findings indicate that further development of COS to fill gaps for missing tumour types, greater involvement of trialists in the COS development process, and increased awareness and understanding of COS amongst trialists are all required to ensure widespread implementation of COS in cancer clinical trials.

Prioritizing attributes of approaches to analyzing patient-centered outcomes that are truncated due to death in critical care clinical trials: a Delphi study

BACKGROUND

A key challenge for many critical care clinical trials is that some patients will die before their outcome is fully measured. This is referred to as "truncation due to death" and must be accounted for in both the treatment effect definition (i.e. the estimand), as well as the statistical analysis approach. It is unknown which analytic approaches to this challenge are most relevant to stakeholders.

METHODS

Using a modified Delphi process, we sought to identify critical attributes of analytic methods used to account for truncation due to death in critical care clinical trials. The Delphi panel included stakeholders with diverse professional or personal experience in critical care-focused clinical trials. The research team generated an initial list of attributes and associated definitions. The attribute list and definitions were refined through two Delphi rounds. Panelists ranked and scored attributes and provided open-ended rationales for responses. A consensus threshold was set as ≥ 70% of respondents rating an attribute as "Critical" (i.e., score ≥ 7 on a 9-point Likert scale) and ≤ 15% of respondents rating the measure as "Not Important" (i.e., a score of ≤ 3).

RESULTS

Thirty-one (91%) of 34 invited individuals participated in one or both rounds. The response rate was 82% in Round 1 and 85% in Round 2. Participants included eight (26%) personal experience experts and 26 (84%) professional experience experts. After two Delphi rounds, four attributes met the criteria for consensus: accuracy (the approach will identify effects if they exist, but will not if they do not), interpretability (the approach enables a straightforward interpretation of the intervention's effect), clinical relevance (the approach can directly inform patient care), and patient-centeredness (the approach is relevant to patients and/or their families). Attributes that did not meet the consensus threshold included sensitivity, comparability, familiarity, mechanistic plausibility, and statistical simplicity.

CONCLUSIONS

We found that methods used to account for truncation due to death in the treatment effect definition and statistical approach in critical care trials should meet at least four defined criteria: accuracy, interpretability, clinical relevance, and patient-centeredness. Future work is needed to derive objective criteria to quantify how well existing estimands and analytic approaches encompass these attributes.

Application of the Estimand Framework to Anesthesia Trials

SUMMARY

Events occurring after randomization, such as use of rescue medication, treatment discontinuation, or death, are common in randomized trials. These events can change either the existence or interpretation of the outcome of interest. However, appropriate handling of these intercurrent events is often unclear. The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) E9(R1) addendum introduced the estimand framework, which aligns trial objectives with the design, conduct, statistical analysis, and interpretation of results. This article describes how the estimand framework can be used in anesthesia trials to precisely define the treatment effect to be estimated, key attributes of an estimand, common intercurrent events in anesthesia trials with strategies for handling them, and use of the estimand framework in a hypothetical anesthesia trial on postoperative delirium. When planning anesthesia trials, clearly defining the estimand is vital to ensure that what is being estimated is clearly understood, is clinically relevant, and helps answer the clinical questions of interest.

Revisiting ethnic discrepancies in COVID-19 hospitalized cohorts: a correction for collider bias

OBJECTIVES

Studies from the first waves of the coronavirus disease 2019 (COVID-19) pandemic suggest that individuals from minority ethnicities are at an increased risk of worse outcomes. Concerns exist that this relationship is potentially driven by bias from analyzing hospitalized patients only. We investigate this relationship and the possible presence of bias.

STUDY DESIGN AND SETTING

Using data from South London hospitals across two COVID-19 waves (February 2020 - May 2021), the relationship between ethnicity and COVID-19 outcomes was examined using regression models. Three iterations of each model were completed: 1) an unadjusted analysis, 2) adjusting for covariates (medical history and deprivation), and 3) adjusting for covariates and bias induced by conditioning on hospitalization.

RESULTS

Among 3,133 patients, those who were Asian had a two-fold increased risk of death during the hospital stay that was consistent across the two COVID-19 waves and was not affected by correcting for conditioning on hospitalization. However, wave-specific effects demonstrate significant differences between ethnic groups until bias from using a hospitalized cohort was corrected for.

CONCLUSION

Worsened COVID-19 outcomes in minority ethnicities may be minimized by correcting for bias induced by conditioning on hospitalization. Consideration of this bias should be a key component of study design.

Eliminating Ambiguous Treatment Effects Using Estimands

Most reported treatment effects in medical research studies are ambiguously defined, which can lead to misinterpretation of study results. This is because most authors do not attempt to describe what the treatment effect represents, and instead require readers to deduce this based on the reported statistical methods. However, this approach is challenging, because many methods provide counterintuitive results. For example, some methods include data from all patients, yet the resulting treatment effect applies only to a subset of patients, whereas other methods will exclude certain patients while results will apply to everyone. Additionally, some analyses provide estimates pertaining to hypothetical settings in which patients never die or discontinue treatment. Herein we introduce estimands as a solution to the aforementioned problem. An estimand is a clear description of what the treatment effect represents, thus saving readers the necessity of trying to infer this from study methods and potentially getting it wrong. We provide examples of how estimands can remove ambiguity from reported treatment effects and describe their current use in practice. The crux of our argument is that readers should not have to infer what investigators are estimating; they should be told explicitly.

Estimands in cluster-randomized trials: choosing analyses that answer the right question

BACKGROUND

Cluster-randomized trials (CRTs) involve randomizing groups of individuals (e.g. hospitals, schools or villages) to different interventions. Various approaches exist for analysing CRTs but there has been little discussion around the treatment effects (estimands) targeted by each.

METHODS

We describe the different estimands that can be addressed through CRTs and demonstrate how choices between different analytic approaches can impact the interpretation of results by fundamentally changing the question being asked, or, equivalently, the target estimand.

RESULTS

CRTs can address either the participant-average treatment effect (the average treatment effect across participants) or the cluster-average treatment effect (the average treatment effect across clusters). These two estimands can differ when participant outcomes or the treatment effect depends on the cluster size (referred to as 'informative cluster size'), which can occur for reasons such as differences in staffing levels or types of participants between small and large clusters. Furthermore, common estimators, such as mixed-effects models or generalized estimating equations with an exchangeable working correlation structure, can produce biased estimates for both the participant-average and cluster-average treatment effects when cluster size is informative. We describe alternative estimators (independence estimating equations and cluster-level analyses) that are unbiased for CRTs even when informative cluster size is present.

CONCLUSION

We conclude that careful specification of the estimand at the outset can ensure that the study question being addressed is clear and relevant, and, in turn, that the selected estimator provides an unbiased estimate of the desired quantity.

Is Intention to Treat Still the Gold Standard or Should Health Technology Assessment Agencies Embrace a Broader Estimands Framework?: Insights and Perspectives From the National Institute for Health and Care Excellence and Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen on the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use E9 (R1) Addendum

OBJECTIVES

The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) E9 (R1) addendum will have an important impact on the design and analysis of randomized controlled clinical trials, which represent crucial sources of evidence in health technology assessments, and on the intention-to-treat (ITT) principle in particular. This article brings together a task force of health economists and statisticians in academic institutes and the pharmaceutical industry, to examine the implications of the addendum from the perspective of the National Institute for Health and Care Excellence (NICE) and the Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (IQWiG) and to address the question of whether the ITT principle should be considered the gold standard for estimating treatment effects.

METHODS

We review the ITT principle, as introduced in the ICH E9 guideline. We then present an overview of the ICH E9 (R1) addendum and its estimand framework, highlighting its premise and the proposed strategies for handling intercurrent events, and examine some cases among submissions to IQWiG and NICE.

RESULTS

IQWiG and NICE appear to have diverging perspectives around the relevance of the ITT principle and, in particular, the acceptance of hypothetical strategies for estimating treatment effects, as suggested by examples where the sponsor proposed an alternative approach to the ITT principle when accounting for treatment switching for interventional oncology trials.

CONCLUSIONS

The ICH E9 (R1) addendum supports the use of methods that depart from the ITT principle. The relevance of estimands using these methods depends on the perspectives and objectives of payers. It is challenging to design a study that meets all stakeholders' research questions. Different estimands may serve to answer different relevant questions or decision problems.

Rethinking intercurrent events in defining estimands for tuberculosis trials

BACKGROUND/AIMS

Tuberculosis remains one of the leading causes of death from an infectious disease globally. Both choices of outcome definitions and approaches to handling events happening post-randomisation can change the treatment effect being estimated, but these are often inconsistently described, thus inhibiting clear interpretation and comparison across trials.

METHODS

Starting from the ICH E9(R1) addendum's definition of an estimand, we use our experience of conducting large Phase III tuberculosis treatment trials and our understanding of the estimand framework to identify the key decisions regarding how different event types are handled in the primary outcome definition, and the important points that should be considered in making such decisions. A key issue is the handling of intercurrent (i.e. post-randomisation) events (ICEs) which affect interpretation of or preclude measurement of the intended final outcome. We consider common ICEs including treatment changes and treatment extension, poor adherence to randomised treatment, re-infection with a new strain of tuberculosis which is different from the original infection, and death. We use two completed tuberculosis trials (REMoxTB and STREAM Stage 1) as illustrative examples. These trials tested non-inferiority of new tuberculosis treatment regimens versus a control regimen. The primary outcome was a binary composite endpoint, 'favourable' or 'unfavourable', which was constructed from several components.

RESULTS

We propose the following improvements in handling the above-mentioned ICEs and loss to follow-up (a post-randomisation event that is not in itself an ICE). First, changes to allocated regimens should not necessarily be viewed as an unfavourable outcome; from the patient perspective, the potential harms associated with a change in the regimen should instead be directly quantified. Second, handling poor adherence to randomised treatment using a per-protocol analysis does not necessarily target a clear estimand; instead, it would be desirable to develop ways to estimate the treatment effects more relevant to programmatic settings. Third, re-infection with a new strain of tuberculosis could be handled with different strategies, depending on whether the outcome of interest is the ability to attain culture negativity from infection with any strain of tuberculosis, or specifically the presenting strain of tuberculosis. Fourth, where possible, death could be separated into tuberculosis-related and non-tuberculosis-related and handled using appropriate strategies. Finally, although some losses to follow-up would result in early treatment discontinuation, patients lost to follow-up before the end of the trial should not always be classified as having an unfavourable outcome. Instead, loss to follow-up should be separated from not completing the treatment, which is an ICE and may be considered as an unfavourable outcome.

CONCLUSION

The estimand framework clarifies many issues in tuberculosis trials but also challenges trialists to justify and improve their outcome definitions. Future trialists should consider all the above points in defining their outcomes.

Evaluating how clear the questions being investigated in randomised trials are: systematic review of estimands

OBJECTIVES

To evaluate how often the precise research question being addressed about an intervention (the estimand) is stated or can be determined from reported methods, and to identify what types of questions are being investigated in phase 2-4 randomised trials.

DESIGN

Systematic review of the clarity of research questions being investigated in randomised trials in 2020 in six leading general medical journals.

DATA SOURCE

PubMed search in February 2021.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

Phase 2-4 randomised trials, with no restrictions on medical conditions or interventions. Cluster randomised, crossover, non-inferiority, and equivalence trials were excluded.

MAIN OUTCOME MEASURES

Number of trials that stated the precise primary question being addressed about an intervention (ie, the primary estimand), or for which the primary estimand could be determined unambiguously from the reported methods using statistical knowledge. Strategies used to handle post-randomisation events that affect the interpretation or existence of patient outcomes, such as intervention discontinuations or uses of additional drug treatments (known as intercurrent events), and the corresponding types of questions being investigated.

RESULTS

255 eligible randomised trials were identified. No trials clearly stated all the attributes of the estimand. In 117 (46%) of 255 trials, the primary estimand could be determined from the reported methods. Intercurrent events were reported in 242 (95%) of 255 trials; but the handling of these could only be determined in 125 (49%) of 255 trials. Most trials that provided this information considered the occurrence of intercurrent events as irrelevant in the calculation of the treatment effect and assessed the effect of the intervention regardless (96/125, 77%)-that is, they used a treatment policy strategy. Four (4%) of 99 trials with treatment non-adherence owing to adverse events estimated the treatment effect in a hypothetical setting (ie, the effect as if participants continued treatment despite adverse events), and 19 (79%) of 24 trials where some patients died estimated the treatment effect in a hypothetical setting (ie, the effect as if participants did not die).

CONCLUSIONS

The precise research question being investigated in most trials is unclear, mainly because of a lack of clarity on the approach to handling intercurrent events. Clear reporting of estimands is necessary in trial reports so that all stakeholders, including clinicians, patients and policy makers, can make fully informed decisions about medical interventions.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42021238053.

Re-randomisation trials in multi-episode settings: Estimands and independence estimators

Often patients may require treatment on multiple occasions. The re-randomisation design can be used in such multi-episode settings, as it allows patients to be re-enrolled and re-randomised for each new treatment episode they experience. We propose a set of estimands that can be used in multi-episode settings, focusing on issues unique to multi-episode settings, namely how each episode should be weighted, how the patient's treatment history in previous episodes should be handled, and whether episode-specific effects or average effects across all episodes should be used. We then propose independence estimators for each estimand, and show the manner in which many re-randomisation trials have been analysed in the past (a simple comparison between all intervention episodes vs. all control episodes) corresponds to a per-episode added-benefit estimand, that is, the average effect of the intervention across all episodes, over and above any benefit conferred from the intervention in previous episodes. We show this estimator is generally unbiased, and describe when other estimators will be unbiased. We conclude that (i) consideration of these estimands can help guide the choice of which analysis method is most appropriate; and (ii) the re-randomisation design with an independence estimator can be a useful approach in multi-episode settings.

Missing data in palliative care research: estimands and estimators

There are several methodological challenges when conducting randomised controlled trials in palliative care. These include worsening function and high mortality, leading to treatment discontinuation, some of which will be unrelated to the intervention being evaluated.Recently, a new framework for handling postrandomisation events, such as attrition, has been released. This framework aims to align trial objectives, design, conduct and analysis by clarifying what and how to estimate treatment effects in the presence of data affected by postrandomisation events.The purpose of this paper is to introduce palliative care researchers to this framework and how it can guide trial design, and efficacy and safety analysis in a palliative care context where individual withdrawal from the trial is common.In this paper, we describe the estimand framework and the background for it. We also consider postrandomisation events that are frequently encountered in palliative care trials and how these might affect objectives of interest. We then construct efficacy and safety estimands for a trial in palliative care. Better trial design and alignment of objectives with analysis can improve our understanding of what treatments do and do not work in palliative care.

Design aspects of COVID-19 treatment trials: Improving probability and time of favorable events

As a reaction to the pandemic of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), a multitude of clinical trials for the treatment of SARS-CoV-2 or the resulting corona disease 2019 (COVID-19) are globally at various stages from planning to completion. Although some attempts were made to standardize study designs, this was hindered by the ferocity of the pandemic and the need to set up clinical trials quickly. We take the view that a successful treatment of COVID-19 patients (i) increases the probability of a recovery or improvement within a certain time interval, say 28 days; (ii) aims to expedite favorable events within this time frame; and (iii) does not increase mortality over this time period. On this background, we discuss the choice of endpoint and its analysis. Furthermore, we consider consequences of this choice for other design aspects including sample size and power and provide some guidance on the application of adaptive designs in this particular context.

Functional loss and mortality in randomized clinical trials for amyotrophic lateral sclerosis: to combine, or not to combine – that is the estimand

Amyotrophic lateral sclerosis is a rapidly progressive disease leading to death in, on average, 3–5 years after first symptom onset. Consequently, there are frequently a non‐negligible number of patients who die during the course of a clinical trial. This introduces bias in end points such as daily functioning, muscle strength, and quality of life. In this paper, we outline how the choice of strategy to handle death affects the interpretation of the trial results. We provide a general overview of the considerations, positioned in the estimand framework, and discuss the possibility that not every strategy provides a clinically relevant answer in each setting. The relevance of a strategy changes as a function of the intended trial duration, hypothesized treatment effect, and population included. It is important to consider this trade‐off at the design stage of a clinical trial, as this will clarify the exact research question that is being answered, and better guide the planning, design, and analysis of the study.

Definitions and elements of endpoints in phase III randomized trials for the treatment of COVID-19: a cross-sectional analysis of trials registered in ClinicalTrials.gov

BACKGROUND

There are several challenges in designing clinical trials for the treatment of novel infectious diseases, such as COVID-19. In particular, the definition of endpoints related to the severity, time frame, and clinical course remains unclear. Therefore, we conducted a cross-sectional analysis of phase III randomized trials for COVID-19 registered at ClinicalTrials.gov .

METHODS

We collected the data from ClinicalTrials.gov on March 31, 2021, by specifying the following search conditions under Advanced Search: Condition or disease: (COVID-19) OR (SARS-CoV-2); Study type: Interventional Studies; Study Results: All Studies; Recruitment: Not yet recruiting, Recruiting, Enrolling by invitation, Active, Not recruiting, Suspended, Completed; Sex: All; and Phase: Phase 3. From the downloaded search results, we selected trials that met the following criteria: Primary Purpose: Treatment; Allocation: Randomized. We manually transcribed information not included in the downloaded file, such as Primary Outcome Measures, Secondary Outcome Measures, Time Frame, and Inclusion Criteria. In the analysis, we examined primary and secondary endpoints in trials with severe and non-severe patients, including the types of endpoints, time frame, clinical course, and sample size.

RESULTS

A total of 406 trials were included in the analysis. The median numbers of endpoints in trials with severe and non-severe patients were 9 and 7, respectively. Approximately 25% of the trials used multiple primary endpoints. Regardless of the type of endpoint, the time frames were longer in the trials with severe patients than in the trials with non-severe patients. In the evaluation of the clinical course, worsening was often considered in binary endpoints, and improvement was considered in time-to-event endpoints. The sample size was the largest in clinical trials using binary endpoints.

CONCLUSIONS

Endpoints can differ with respect to severity, and the clinical course and time frame are important for defining endpoints. This study provides information that can facilitate the achievement of a consensus for the endpoints in evaluating COVID-19 treatments.

Estimands in published protocols of randomised trials: urgent improvement needed

BACKGROUND

An estimand is a precise description of the treatment effect to be estimated from a trial (the question) and is distinct from the methods of statistical analysis (how the question is to be answered). The potential use of estimands to improve trial research and reporting has been underpinned by the recent publication of the ICH E9(R1) Addendum on the use of estimands in clinical trials in 2019. We set out to assess how well estimands are described in published trial protocols.

METHODS

We reviewed 50 trial protocols published in October 2020 in Trials and BMJ Open. For each protocol, we determined whether the estimand for the primary outcome was explicitly stated, not stated but inferable (i.e. could be constructed from the information given), or not inferable.

RESULTS

None of the 50 trials explicitly described the estimand for the primary outcome, and in 74% of trials, it was impossible to infer the estimand from the information included in the protocol. The population attribute of the estimand could not be inferred in 36% of trials, the treatment condition attribute in 20%, the population-level summary measure in 34%, and the handling of intercurrent events in 60% (the strategy for handling non-adherence was not inferable in 32% of protocols, and the strategy for handling mortality was not inferable in 80% of the protocols for which it was applicable). Conversely, the outcome attribute was stated for all trials. In 28% of trials, three or more of the five estimand attributes could not be inferred.

CONCLUSIONS

The description of estimands in published trial protocols is poor, and in most trials, it is impossible to understand exactly what treatment effect is being estimated. Given the utility of estimands to improve clinical research and reporting, this urgently needs to change.

Why ITT analysis is not always the answer for estimating treatment effects in clinical trials

For many years there has been a consensus among the Clinical Research community that ITT analysis represents the correct approach for the vast majority of trials. Recent worldwide regulatory guidance for pharmaceutical industry trials has allowed discussion of alternatives to the ITT approach to analysis; different treatment effects can be considered which may be more clinically meaningful and more relevant to patients and prescribers. The key concept is of a trial "estimand", a precise description of the estimated treatment effect. The strategy chosen to account for patients who discontinue treatment or take alternative medications which are not part of the randomised treatment regimen are important determinants of this treatment effect. One strategy to account for these events is treatment policy, which corresponds to an ITT approach. Alternative equally valid strategies address what the treatment effect is if the patient actually takes the treatment or does not use specific alternative medication. There is no single right answer to which strategy is most appropriate, the solution depends on the key clinical question of interest. The estimands framework discussed in the new guidance has been particularly useful in the context of the current COVID-19 pandemic and has clarified what choices are available to account for the impact of COVID-19 on clinical trials. Specifically, an ITT approach addresses a treatment effect that may not be generalisable beyond the current pandemic.

Infectious diseases epidemiology, quantitative methodology, and clinical research in the midst of the COVID-19 pandemic: Perspective from a European country

Starting from historic reflections, the current SARS-CoV-2 induced COVID-19 pandemic is examined from various perspectives, in terms of what it implies for the implementation of non-pharmaceutical interventions, the modeling and monitoring of the epidemic, the development of early-warning systems, the study of mortality, prevalence estimation, diagnostic and serological testing, vaccine development, and ultimately clinical trials. Emphasis is placed on how the pandemic had led to unprecedented speed in methodological and clinical development, the pitfalls thereof, but also the opportunities that it engenders for national and international collaboration, and how it has simplified and sped up procedures. We also study the impact of the pandemic on clinical trials in other indications. We note that it has placed biostatistics, epidemiology, virology, infectiology, and vaccinology, and related fields in the spotlight in an unprecedented way, implying great opportunities, but also the need to communicate effectively, often amidst controversy.