On estimands and the analysis of adverse events in the presence of varying follow-up times within the benefit assessment of therapies

A Comparison of Statistical Methods for Time-To-Event Analyses in Randomized Controlled Trials Under Non-Proportional Hazards

While well-established methods for time-to-event data are available when the proportional hazards assumption holds, there is no consensus on the best inferential approach under non-proportional hazards (NPH). However, a wide range of parametric and non-parametric methods for testing and estimation in this scenario have been proposed. To provide recommendations on the statistical analysis of clinical trials where non-proportional hazards are expected, we conducted a simulation study under different scenarios of non-proportional hazards, including delayed onset of treatment effect, crossing hazard curves, subgroups with different treatment effects, and changing hazards after disease progression. We assessed type I error rate control, power, and confidence interval coverage, where applicable, for a wide range of methods, including weighted log-rank tests, the MaxCombo test, summary measures such as the restricted mean survival time (RMST), average hazard ratios, and milestone survival probabilities, as well as accelerated failure time regression models. We found a trade-off between interpretability and power when choosing an analysis strategy under NPH scenarios. While analysis methods based on weighted logrank tests typically were favorable in terms of power, they do not provide an easily interpretable treatment effect estimate. Also, depending on the weight function, they test a narrow null hypothesis of equal hazard functions, and rejection of this null hypothesis may not allow for a direct conclusion of treatment benefit in terms of the survival function. In contrast, non-parametric procedures based on well-interpretable measures like the RMST difference had lower power in most scenarios. Model-based methods based on specific survival distributions had larger power; however, often gave biased estimates and lower than nominal confidence interval coverage. The application of the studied methods is illustrated in a case study with reconstructed data from a phase III oncologic trial.

Methodological challenges using routine clinical care data for real-world evidence: a rapid review utilizing a systematic literature search and focus group discussion

BACKGROUND

The integration of real-world evidence (RWE) from real-world data (RWD) in clinical research is crucial for bridging the gap between clinical trial results and real-world outcomes. Analyzing routinely collected data to generate clinical evidence faces methodological concerns like confounding and bias, similar to prospectively documented observational studies. This study focuses on additional limitations frequently reported in the literature, providing an overview of the challenges and biases inherent to analyzing routine clinical care data, including health claims data (hereafter: routine data).

METHODS

We conducted a literature search on routine data studies in four high-impact journals based on the Journal Citation Reports (JCR) category "Medicine, General & Internal" as of 2022 and three oncology journals, covering articles published from January 2018 to October 2023. Articles were screened and categorized into three scenarios based on their potential to provide meaningful RWE: (1) Burden of Disease, (2) Safety and Risk Group Analysis, and (3) Treatment Comparison. Limitations of this type of data cited in the discussion sections were extracted and classified according to different bias types: main bias categories in non-randomized studies (information bias, reporting bias, selection bias, confounding) and additional routine data-specific challenges (i.e., operationalization, coding, follow-up, missing data, validation, and data quality). These classifications were then ranked by relevance in a focus group meeting of methodological experts. The search was pre-specified and registered in PROSPERO (CRD42023477616).

RESULTS

In October 2023, 227 articles were identified, 69 were assessed for eligibility, and 39 were included in the review: 11 on the burden of disease, 17 on safety and risk group analysis, and 11 on treatment comparison. Besides typical biases in observational studies, we identified additional challenges specific to RWE frequently mentioned in the discussion sections. The focus group had varied opinions on the limitations of Safety and Risk Group Analysis and Treatment Comparison but agreed on the essential limitations for the Burden of Disease category.

CONCLUSION

This review provides a comprehensive overview of potential limitations and biases in analyzing routine data reported in recent high-impact journals. We highlighted key challenges that have high potential to impact analysis results, emphasizing the need for thorough consideration and discussion for meaningful inferences.

Tree-temporal scan statistics for safety signal detection in vaccine clinical trials

The evaluation of safety is critical in all clinical trials. However, the quantitative analysis of safety data in clinical trials poses statistical difficulties because of multiple potentially overlapping endpoints. Tree-temporal scan statistic approaches address this issue and have been widely employed in other data sources, but not to date in clinical trials. We evaluated the performance of three complementary scan statistical methods for routine quantitative safety signal detection: the self-controlled tree-temporal scan (SCTTS), a tree-temporal scan based on group comparison (BGTTS), and a log-rank based tree-temporal scan (LgRTTS). Each method was evaluated using data from two phase III clinical trials, and simulated data (simulation study). In the case study, the reference set was adverse events (AEs) in the Reference Safety Information of the evaluated vaccine. The SCTTS method had higher sensitivity than other methods, and after dose 1 detected 80 true positives (TP) with a positive predictive value (PPV) of 60%. The LgRTTS detected 49 TPs with 69% PPV. The BGTTS had 90% of PPV with 38 TPs. In the simulation study, with simulated reference sets of AEs, the SCTTS method had good sensitivity to detect transient effects. The LgRTTS method showed the best performance for the detection of persistent effects, with high sensitivity and expected probability of type I error. These three methods provide complementary approaches to safety signal detection in clinical trials or across clinical development programmes. All three methods formally adjust for multiple testing of large numbers of overlapping endpoints without being excessively conservative.

Estimand Framework and Statistical Considerations for Integrated Analysis of Clinical Trial Safety Data

BACKGROUND

Safety analyses play a pivotal role in drug development, ensuring the protection of patients while advancing innovative pharmaceuticals to market. A single study generally does not have sufficient sample size to evaluate all important safety events with reasonable precision and may not cover the full target population for the investigational treatment. Integrated analyses (pooled or meta-analysis) over several studies may be helpful in that regard. But without a structured conscious workflow accompanied with appropriate statistical methods for the integrated analysis, this can easily take a route compromising the interpretation.

METHODS

In this article we apply the ICH estimand framework to clinical trial integration and summarize respective critical statistical assumptions to ensure the integrated analyses are interpretable.

RESULTS

The estimand framework is valuable for developing principles for a deeper understanding of the critical statistical aspects of planning an integrated safety analysis. Our principles address the clinical question of interest, estimand and estimation. Special focus was given to the criteria for inclusion and exclusion of the component studies in the integrated analysis, and to integration of estimates pertaining to signal detection.

CONCLUSION

Performing an integrated analysis and its preparatory steps calls for a good understanding of the clinical question of interest and its estimand, care and sound practice, to enable interpretation and avoid introducing unnecessary bias. It is valuable to use the estimand framework not only for efficacy evaluations, but also for safety evaluations in clinical trials and for integrated safety analyses.

Time-to-event estimands and loss to follow-up in oncology in light of the estimands guidance

Time-to-event estimands are central to many oncology clinical trials. The estimands framework (addendum to the ICH E9 guideline) calls for precisely defining the treatment effect of interest to align with the clinical question of interest and requires predefining the handling of intercurrent events (ICEs) that occur after treatment initiation and "affect either the interpretation or the existence of the measurements associated with the clinical question of interest." We discuss a practical problem in clinical trial design and execution, that is, in some clinical contexts it is not feasible to systematically follow patients to an event of interest. Loss to follow-up in the presence of intercurrent events can affect the meaning and interpretation of the study results. We provide recommendations for trial design, stressing the need for close alignment of the clinical question of interest and study design, impact on data collection, and other practical implications. When patients cannot be systematically followed, compromise may be necessary to select the best available estimand that can be feasibly estimated under the circumstances. We discuss the use of sensitivity and supplementary analyses to examine assumptions of interest.

A safety estimand for late phase clinical trials where the analysis period varies over the subjects

BACKGROUND/AIMS

Evaluating safety is as important as evaluating efficacy in a clinical trial, yet the tradition for safety analysis is rudimentary. This article explores more complex methodologies for safety evaluation, with the aim of improving the interpretability, as well as generalizability, of the results.

METHODS

For studies where the analysis periods vary over the subjects, using the International Council for Harmonisation estimand framework, we construct a formal estimand that could be used in the setting of safety surveillance that answers the clinical question of 'What is the magnitude of the increase in risk of experiencing an adverse event if the treatment is taken, as prescribed, for a specific period of time?'. Estimation methodologies for this estimand are also discussed.

RESULTS

The proposed estimand is similar to that found in the efficacy analyses of time to event data (e.g. in outcome studies), with the key difference of utilization of hypothetical intercurrent event strategy for the intercurrent event of treatment discontinuation. This is motivated by what we perceive to be a key difference for the safety objective compared to efficacy objectives, namely a desire for sensitivity (i.e. greater possibility of detecting a negative impact of the drug, if such exists) as opposed to the need to prove a positive effect of the drug in a conservative manner.

CONCLUSION

It is valuable, and possible, to use the International Council for Harmonisation estimand framework not only for efficacy but also for safety evaluation, with the estimand driven by an interpretable, and relevant, clinical question.

A proposal for using benefit-risk methods to improve the prominence of adverse event results when reporting trials

Adverse events suffer from poor reporting within randomised controlled trials, despite them being crucial to the evaluation of a treatment. A recent update to the CONSORT harms checklist aims to improve reporting by providing structure and consistency to the information presented. We propose an extension wherein harms would be reported in conjunction with effectiveness outcome(s) rather than in silo to provide a more complete picture of the evidence acquired within a trial. Benefit-risk methods are designed to simultaneously consider both benefits and risks, and therefore, we believe these methods could be implemented to improve the prominence of adverse events when reporting trials. The aim of this article is to use case studies to demonstrate the practical utility of benefit-risk methods to present adverse events results alongside effectiveness results. Two randomised controlled trials have been selected as case studies, the Option-DM trial and the SANAD II trial. Using a previous review, a shortlist of 17 benefit-risk methods which could potentially be used for reporting RCTs was created. From this shortlist, three benefit-risk methods are applied across the two case studies. We selected these methods for their usefulness to achieve the aim of this paper and which are commonly used in the literature. The methods selected were the Benefit-Risk Action Team (BRAT) Framework, net clinical benefit (NCB), and the Outcome Measures in Rheumatology (OMERACT) 3 × 3 table. Results using the benefit-risk method added further context and detail to the clinical summaries made from the trials. In the case of the SANAD II trial, the clinicians concluded that despite the primary outcome being improved by the treatment, the increase in adverse events negated the improvement and the treatment was therefore not recommended. The benefit-risk methods applied to this case study outlined the data that this decision was based on in a clear and transparent way. Using benefit-risk methods to report the results of trials can increase the prominence of adverse event results by presenting them alongside the primary efficacy/effectiveness outcomes. This ensures that all the factors which would be used to determine whether a treatment would be recommended are transparent to the reader.

An Overview of Regression Models for Adverse Events Analysis

Over the last few years, several review articles described the adverse events analysis as sub-optimal in clinical trials. Indeed, the context surrounding adverse events analyses often imply an overwhelming number of events, a lack of power to find associations, but also a lack of specific training regarding those complex data. In randomized controlled trials or in observational studies, comparing the occurrence of adverse events according to a covariable of interest (e.g., treatment) is a recurrent question in the analysis of drug safety data, and adjusting other important factors is often relevant. This article is an overview of the existing regression models that may be considered to compare adverse events and to discuss model choice regarding the characteristics of the adverse events of interest. Many dimensions may be relevant to compare the adverse events between patients, (e.g., timing, recurrence, and severity). Recent efforts have been made to cover all of them. For chronic treatments, the occurrence of intercurrent events during the patient follow-up usually needs the modeling approach to be adapted (at least with regard to their interpretation). Moreover, analysis based on regression models should not be limited to the estimation of relative effects. Indeed, absolute risks stemming from the model should be presented systematically to help the interpretation, to validate the model, and to encourage comparison of studies.

[Special issue PRO] Considering endpoints for comparative tolerability of cancer treatments using patient report given the estimand framework

Regulatory agencies are advancing the use of systematic approaches to collect patient experience data, including patient-reported outcomes (PROs), in cancer clinical trials to inform regulatory decision-making. Due in part to clinician under-reporting of symptomatic adverse events, there is a growing recognition that evaluation of cancer treatment tolerability should include the patient experience, both in terms of the overall side effect impact and symptomatic adverse events. Methodologies around implementation, analysis, and interpretation of "patient" reported tolerability are under development, and current approaches are largely descriptive. There is robust guidance for use of PROs as efficacy endpoints to compare cancer treatments, but it is unclear to what extent this can be relied-upon to develop tolerability endpoints. An important consideration when developing endpoints to compare tolerability between treatments is the linkage of trial design, objectives, and statistical analysis. Despite interest in and frequent collection of PRO data in oncology trials, heterogeneity in analyses and unclear PRO objectives mean that design, objectives, and analysis may not be aligned, posing substantial challenges for the interpretation of results. The recent ICH E9 (R1) estimand framework represents an opportunity to help address these challenges. Efforts to apply the estimand framework in the context of PROs have primarily focused on efficacy outcomes. In this paper, we discuss considerations for comparing the patient-reported tolerability of different treatments in an oncology trial context.

Adverse events in single-arm clinical trials with non-fatal time-to-event efficacy endpoint: from clinical questions to methods for statistical analysis

BACKGROUND

In any single-arm trial on novel treatments, assessment of toxicity plays an important role as occurrence of adverse events (AEs) is relevant for application in clinical practice. In the presence of a non-fatal time-to-event(s) efficacy endpoint, the analysis should be broadened to consider AEs occurrence in time. The AEs analysis could be tackled with two approaches, depending on the clinical question of interest. Approach 1 focuses on the occurrence of AE as first event. Treatment ability to protect from the efficacy endpoint event(s) has an impact on the chance of observing AEs due to competing risks action. Approach 2 considers how treatment affects the occurrence of AEs in the potential framework where the efficacy endpoint event(s) could not occur.

METHODS

In the first part of the work we review the strategy of analysis for these two approaches. We identify theoretical quantities and estimators consistent with the following features: (a) estimators should address for the presence of right censoring; (b) theoretical quantities and estimators should be functions of time. In the second part of the work we propose the use of alternative methods (regression models, stratified Kaplan-Meier curves, inverse probability of censoring weighting) to relax the assumption of independence between the potential times to AE and to event(s) in the efficacy endpoint for addressing Approach 2.

RESULTS

We show through simulations that the proposed methods overcome the bias due to the dependence between the two potential times and related to the use of standard estimators.

CONCLUSIONS

We demonstrated through simulations that one can handle patients selection in the risk sets due to the competing event, and thus obtain conditional independence between the two potential times, adjusting for all the observed covariates that induce dependence.

A Novel Method for Deriving Adverse Event Prevalence in Randomized Controlled Trials: Potential for Improved Understanding of Benefit-Risk Ratio and Application to Drug Labels

INTRODUCTION

Adverse event (AE) data in randomized controlled trials (RCTs) allow quantification of a drug's safety risk relative to placebo and comparison across medications. The standard US label for Food and Drug Administration-approved drugs typically lists AEs by MedDRA Preferred Term that occur at ≥ 2% in drug and with greater incidence than in placebo. We suggest that the drug label can be more informative for both patients and physicians if it includes, in addition to AE incidence (percent of subjects who reported the AE out of the total subjects in treatment), the absolute prevalence (percent of subject-days spent with an AE out of the total subject-days spent in treatment) and expected duration (days required for AE incidence to be reduced by half). We also propose a new method to analyze AEs in RCTs using drug-placebo difference in AE prevalence to improve safety signal detection.

METHODS

AE data from six RCTs in schizophrenia were analyzed (five RCTs of the dopamine D2 receptor-based antipsychotic lurasidone and one RCT of the novel trace amine-associated receptor 1 [TAAR1] agonist ulotaront). We determined incidence, absolute prevalence, and expected duration of AEs for lurasidone and ulotaront vs respective placebo. We also calculated areas under the curve of drug-placebo difference in AE prevalence and mean percent contribution of each AE to this difference.

RESULTS

A number of AEs with the same incidence had different absolute prevalence and expected duration. When accounting for these two parameters, AEs that did not appear in the 2% incidence tables of the drug label turned out to contribute substantially to drug tolerability. The percent contribution of a drug-related AE to the overall side effect burden increased the drug-placebo difference in AE prevalence, whereas the percent contribution of a placebo-related AE decreased such difference, revealing a continuum of risk between drug and placebo. AE prevalence curves for drug were generally greater than those for placebo. Ulotaront exhibited a small drug-placebo difference in AE prevalence curves due to a relatively low incidence and short duration of AEs in the ulotaront treatment arm as well as the emergence of disease-related AEs in the placebo arm.

CONCLUSION

Reporting AE absolute prevalence and expected duration for each RCT and incorporating them in the drug label is possible, is clinically relevant, and allows standardized comparison of medications. Our new metric, the drug-placebo difference in AE prevalence, facilitates signal detection in RCTs. We piloted this metric in RCTs of several neuropsychiatric indications and drugs, offering a new way to compare AE burden and tolerability among treatments using existing clinical trial information.

Application of estimand framework in ICH E9 (R1) to safety evaluation

Defining the right question of interest is important to a clinical study. ICH E9 (R1) introduces the framework of an estimand and its five attributes, which provide a basis for connecting different components of a study with its clinical questions. Most of the applications of the estimand framework focus on efficacy instead of safety assessment. In this paper, we expand the estimand framework into the safety evaluation and compare/contrast the similarity and differences between safety and efficacy estimand. Furthermore, we present and discuss applications of a safety estimand to oncology trials and pooled data analyses. At last, we also discuss the potential usage of safety estimand to handle the impacts of COVID-19 pandemic on safety assessment.

Assessing adverse events in clinical trials during the era of the COVID-19 pandemic

Interpretation of safety data for clinical trials that were ongoing at the onset of the COVID-19 pandemic or were started subsequent to the beginning of the pandemic may be affected in a variety of ways. Pandemic-related issues can influence the extent of study participation and introduce data collection gaps. A SARS-CoV-2 infection among study subjects as a post-randomization event may introduce a number of confounding factors that can alter the frequency of adverse events, in some cases appearing as an increase in the frequency of an adverse event associated with a study drug relative to a comparator. The authors discuss clinical challenges and statistical concerns, specifically the estimand framework, including examples for consideration, to address these challenges in safety evaluation wrought by the COVID-19 pandemic. Our aim is to shed light on the importance of starting an early dialogue among the drug development team on the evaluation of safety, critical for benefit-risk evaluation throughout the drug development process.

Application of estimand framework in ICH E9 (R1) to vaccine trials

Over the past decades, the primary interest in vaccine efficacy or immunogenicity evaluation mostly focuses on the biological effect of immunization in complying with the vaccination schedule in a targeted population. The safety questions, which are essential for vaccines as they are generally given to large healthy populations, need to be clearly defined to reflect the risk assessment of interest. ICH E9 (R1) provides a structured framework to clarify the clinical questions and formulate the treatment effect as an estimand. This paper applies the estimand framework to vaccine clinical trials on common clinical questions regarding efficacy, immunogenicity, and safety.

Important Considerations for Signal Detection and Evaluation

Safety clinicians have a wealth of resources describing how to perform signal detection. Nevertheless, there are some nuances concerning approaches taken by regulatory authorities and statistical considerations that should be appreciated. New approaches, such as the FDA Medical Queries, illustrate the value of considering medical concepts over individual adverse events. One area which would benefit from further clarity is how safety signals may be evaluated for evidence of a causal relationship to the drug of interest. Just as such safety signals can take many forms, the types of tools and methods required to interrogate these signals are equally as diverse. An understanding of the complexity of this process can aid the safety reviewer in successfully characterizing the emerging safety profile of a drug during the pre-marketing phase of development.

Marking 2-Years of New Thinking in Clinical Trials: The Estimand Journey

The ICH E9(R1) addendum on Estimands and Sensitivity Analyses in Clinical Trials has introduced a new estimand framework for the design, conduct, analysis, and interpretation of clinical trials. We share Pharmaceutical Industry experiences of implementing the estimand framework in the first two years since the final guidance became available with key lessons learned and highlight what else needs to be done to continue the journey in embedding the estimand framework in clinical trials. Emerging best practices and points to consider on strategies for implementing a new estimand thinking process are provided. Whilst much of the focus of implementing ICH E9(R1) to date has been on defining estimands, we highlight some of the important aspects relating to the choice of statistical analysis methods and sensitivity analyses to ensure estimands can be estimated robustly with minimal bias. In particular, we discuss the implications if complete follow-up is not possible when the treatment policy strategy is being used to handle intercurrent events. ICH E9(R1) was introduced just before the start of the COVID-19 pandemic, but a positive outcome from the pandemic has been an acceleration in the adoption of the estimand framework, including differentiating intercurrent events related or not related to the pandemic. In summary, much has been learned on the estimand journey and continued sharing of case studies will help to further advance the understanding and increase awareness across all clinical researchers of the estimand framework.

Missing data in palliative care research: estimands and estimators

There are several methodological challenges when conducting randomised controlled trials in palliative care. These include worsening function and high mortality, leading to treatment discontinuation, some of which will be unrelated to the intervention being evaluated.Recently, a new framework for handling postrandomisation events, such as attrition, has been released. This framework aims to align trial objectives, design, conduct and analysis by clarifying what and how to estimate treatment effects in the presence of data affected by postrandomisation events.The purpose of this paper is to introduce palliative care researchers to this framework and how it can guide trial design, and efficacy and safety analysis in a palliative care context where individual withdrawal from the trial is common.In this paper, we describe the estimand framework and the background for it. We also consider postrandomisation events that are frequently encountered in palliative care trials and how these might affect objectives of interest. We then construct efficacy and safety estimands for a trial in palliative care. Better trial design and alignment of objectives with analysis can improve our understanding of what treatments do and do not work in palliative care.

Improving the analysis of adverse event data in randomised controlled trials

Analysing treatment harm is vital but problematic with the relatively small sample sizes afforded in randomised controlled trials (RCTs). Good analysis practice for efficacy outcomes are well established but there has been minimal progress for the analysis of adverse events (AEs). In this commentary we examine four key issues for AE analysis. Namely, why harm data in RCTs is undervalued, why AE analysis is difficult, what aspects of current analysis practice are unsatisfactory, and the challenges for selection and interpretation of AEs results in publications. We discuss how the value of harm data from RCTs could be better realised by reframing the research question to one for detecting signals of adverse reactions. We align established good statistical practice to current unsatisfactory practice. We encourage use of Bayesian analyses to enable cumulative assessment of harm across trial research phases and discourage selecting AEs to report based on arbitrary rules. We propose comprehendible summaries to be based on core outcome sets, serious and pre-specified events, and events leading to discontinuation. Analysis of AEs in contemporary clinical trials needs attention to progress. In the following we have outlined immediate, mid and longer-term strategies for trialists to adopt to support a change in practice.

Joint modelling of multivariate longitudinal clinical laboratory safety outcomes, concomitant medication and clinical adverse events: application to artemisinin-based treatment during pregnancy clinical trial

Background

In drug trials, clinical adverse events (AEs), concomitant medication and laboratory safety outcomes are repeatedly collected to support drug safety evidence. Despite the potential correlation of these outcomes, they are typically analysed separately, potentially leading to misinformation and inefficient estimates due to partial assessment of safety data. Using joint modelling, we investigated whether clinical AEs vary by treatment and how laboratory outcomes (alanine amino-transferase, total bilirubin) and concomitant medication are associated with clinical AEs over time following artemisinin-based antimalarial therapy.

Methods

We used data from a trial of artemisinin-based treatments for malaria during pregnancy that randomized 870 women to receive artemether–lumefantrine (AL), amodiaquine–artesunate (ASAQ) and dihydroartemisinin–piperaquine (DHAPQ). We fitted a joint model containing four sub-models from four outcomes: longitudinal sub-model for alanine aminotransferase, longitudinal sub-model for total bilirubin, Poisson sub-model for concomitant medication and Poisson sub-model for clinical AEs. Since the clinical AEs was our primary outcome, the longitudinal sub-models and concomitant medication sub-model were linked to the clinical AEs sub-model via current value and random effects association structures respectively. We fitted a conventional Poisson model for clinical AEs to assess if the effect of treatment on clinical AEs (i.e. incidence rate ratio (IRR)) estimates differed between the conventional Poisson and the joint models, where AL was reference treatment.

Results

Out of the 870 women, 564 (65%) experienced at least one AE. Using joint model, AEs were associated with the concomitant medication (log IRR 1.7487; 95% CI: 1.5471, 1.9503; p < 0.001) but not the total bilirubin (log IRR: -0.0288; 95% CI: − 0.5045, 0.4469; p = 0.906) and alanine aminotransferase (log IRR: 0.1153; 95% CI: − 0.0889, 0.3194; p = 0.269). The Poisson model underestimated the effects of treatment on AE incidence such that log IRR for ASAQ was 0.2118 (95% CI: 0.0082, 0.4154; p = 0.041) for joint model compared to 0.1838 (95% CI: 0.0574, 0.3102; p = 0.004) for Poisson model.

Conclusion

We demonstrated that although the AEs did not vary across the treatments, the joint model yielded efficient AE incidence estimates compared to the Poisson model. The joint model showed a positive relationship between the AEs and concomitant medication but not with laboratory outcomes.

Trial registration

ClinicalTrials.gov: NCT00852423

Confidence intervals for exposure-adjusted rate differences in randomized trials

Exposure-adjusted event rate is a quantity often used in clinical trials to describe average event count per unit of person-time. The event count may represent the number of patients experiencing first (incident) event episode, or the total number of event episodes, including recurring events. For inference about difference in the exposure-adjusted rates between interventions, many methods of interval estimation rely on the assumption of Poisson distribution for the event counts. These intervals may suffer from substantial undercoverage both, asymptotically due to extra-Poisson variation, and in the settings with rare events even when the Poisson assumption is satisfied. We review asymptotically robust methods of interval estimation for the rate difference that do not depend on distributional assumptions for the event counts, and propose a modification of one of these methods. The new interval estimator has asymptotically nominal coverage for the rate difference with an arbitrary distribution of event counts, and good finite sample properties, avoiding substantial undercoverage with small samples, rare events, or over-dispersed data. The proposed method can handle covariate adjustment and can be implemented with commonly available software. The method is illustrated using real data on adverse events in a clinical trial.

Survival analysis for AdVerse events with VarYing follow-up times (SAVVY)-estimation of adverse event risks

Background

The SAVVY project aims to improve the analyses of adverse events (AEs), whether prespecified or emerging, in clinical trials through the use of survival techniques appropriately dealing with varying follow-up times and competing events (CEs). Although statistical methodologies have advanced, in AE analyses, often the incidence proportion, the incidence density, or a non-parametric Kaplan-Meier estimator are used, which ignore either censoring or CEs. In an empirical study including randomized clinical trials from several sponsor organizations, these potential sources of bias are investigated. The main purpose is to compare the estimators that are typically used to quantify AE risk within trial arms to the non-parametric Aalen-Johansen estimator as the gold-standard for estimating cumulative AE probabilities. A follow-up paper will consider consequences when comparing safety between treatment groups.

Methods

Estimators are compared with descriptive statistics, graphical displays, and a more formal assessment using a random effects meta-analysis. The influence of different factors on the size of deviations from the gold-standard is investigated in a meta-regression. Comparisons are conducted at the maximum follow-up time and at earlier evaluation times. CEs definition does not only include death before AE but also end of follow-up for AEs due to events related to the disease course or safety of the treatment.

Results

Ten sponsor organizations provided 17 clinical trials including 186 types of investigated AEs. The one minus Kaplan-Meier estimator was on average about 1.2-fold larger than the Aalen-Johansen estimator and the probability transform of the incidence density ignoring CEs was even 2-fold larger. The average bias using the incidence proportion was less than 5%. Assuming constant hazards using incidence densities was hardly an issue provided that CEs were accounted for. The meta-regression showed that the bias depended mainly on the amount of censoring and on the amount of CEs.

Conclusions

The choice of the estimator of the cumulative AE probability and the definition of CEs are crucial. We recommend using the Aalen-Johansen estimator with an appropriate definition of CEs whenever the risk for AEs is to be quantified and to change the guidelines accordingly.

Metabolic disease and adverse events from immune checkpoint inhibitors

OBJECTIVE

Obese and overweight body mass index (BMI) categories have been associated with increased immune-related adverse events (irAEs) in patients with cancer receiving immune checkpoint inhibitors (ICIs); however, the impact of being overweight in conjunction with related metabolic syndrome-associated factors on irAEs have not been investigated. We aimed to evaluate the impact of overweight and obese BMI according to metabolic disease burden on the development of irAEs.

DESIGN AND METHODS

We conducted a retrospective observational study of patients receiving ICIs at a cancer center. Our main study outcome was development of ≥grade 2 (moderate) irAEs. Our main predictor was weight/metabolic disease risk category: (1) normal weight (BMI 18.5-24.9 kg/m²)/low metabolic risk (<2 metabolic diseases [diabetes, dyslipidemia, hypertension]), (2) normal weight/high metabolic risk (≥2 metabolic diseases), (3) overweight (BMI ≥25 kg/m²)/low metabolic risk, and (4) overweight/high metabolic risk.

RESULTS

Of 411 patients in our cohort, 374 were eligible for analysis. Overall, 111 (30%) patients developed ≥grade 2 irAEs. In Cox analysis, overweight/low metabolic risk was significantly associated with ≥grade 2 irAEs (hazard ratio [HR]: 2.0, 95% confidence interval [95% CI]: 1.2-3.4) when compared to normal weight/low metabolic risk, while overweight/high metabolic risk (HR: 1.3, 95% CI: 0.7-2.2) and normal weight/high metabolic risk (HR: 1.5, 95% CI: 0.7-3.0) were not.

CONCLUSIONS

Overweight patients with fewer metabolic comorbidities were at increased risk for irAEs. This study provides an important insight that BMI should be evaluated in the context of associated metabolic comorbidities in assessing risk of irAE development and ICI immune response.

Estimating and comparing adverse event probabilities in the presence of varying follow-up times and competing events

Safety analyses of adverse events (AEs) are important in assessing benefit-risk of therapies but are often rather simplistic compared to efficacy analyses. AE probabilities are typically estimated by incidence proportions, sometimes incidence densities or Kaplan-Meier estimation are proposed. These analyses either do not account for censoring, rely on a too restrictive parametric model, or ignore competing events. With the non-parametric Aalen-Johansen estimator as the "gold standard", that is, reference estimator, potential sources of bias are investigated in an example from oncology and in simulations, for both one-sample and two-sample scenarios. The Aalen-Johansen estimator serves as a reference, because it is the proper non-parametric generalization of the Kaplan-Meier estimator to multiple outcomes. Because of potential large variances at the end of follow-up, comparisons also consider further quantiles of the observed times. To date, consequences for safety comparisons have hardly been investigated, the impact of using different estimators for group comparisons being unclear. For example, the ratio of two both underestimating or overestimating estimators may not be comparable to the ratio of the reference, and our investigation also considers the ratio of AE probabilities. We find that ignoring competing events is more of a problem than falsely assuming constant hazards by the use of the incidence density and that the choice of the AE probability estimator is crucial for group comparisons.

Aggregate Safety Assessment Planning for the Drug Development Life-Cycle

The Program Safety Analysis Plan (PSAP) was proposed previously as a tool to proactively plan for integrated analyses of product safety data. Building on the PSAP and taking into consideration the evolving regulatory landscape, the Drug Information Association-American Statistical Association (DIA-ASA) Interdisciplinary Safety Evaluation scientific working group herein proposes the Aggregate Safety Assessment Plan (ASAP) process. The ASAP evolves over a product's life-cycle and promotes interdisciplinary, systematic safety planning as well as ongoing data review and characterization of the emerging product safety profile. Objectives include alignment on the safety topics of interest, identification of safety knowledge gaps, planning for aggregate safety evaluation of the clinical trial data and preparing for safety communications. The ASAP seeks to tailor the analyses for a drug development program while standardizing the analyses across studies within the program. The document is intended to be modular and flexible in nature, depending on the program complexity, phase of development and existing sponsor processes. Implementation of the ASAP process will facilitate early safety signal detection, improve characterization of product risks, harmonize safety messaging, and inform program decision-making.

Statistical methods for the analysis of adverse event data in randomised controlled trials: a scoping review and taxonomy

BACKGROUND

Statistical methods for the analysis of harm outcomes in randomised controlled trials (RCTs) are rarely used, and there is a reliance on simple approaches to display information such as in frequency tables. We aimed to identify whether any statistical methods had been specifically developed to analyse prespecified secondary harm outcomes and non-specific emerging adverse events (AEs).

METHODS

A scoping review was undertaken to identify articles that proposed original methods or the original application of existing methods for the analysis of AEs that aimed to detect potential adverse drug reactions (ADRs) in phase II-IV parallel controlled group trials. Methods where harm outcomes were the (co)-primary outcome were excluded. Information was extracted on methodological characteristics such as: whether the method required the event to be prespecified or could be used to screen emerging events; and whether it was applied to individual events or the overall AE profile. Each statistical method was appraised and a taxonomy was developed for classification.

RESULTS

Forty-four eligible articles proposing 73 individual methods were included. A taxonomy was developed and articles were categorised as: visual summary methods (8 articles proposing 20 methods); hypothesis testing methods (11 articles proposing 16 methods); estimation methods (15 articles proposing 24 methods); or methods that provide decision-making probabilities (10 articles proposing 13 methods). Methods were further classified according to whether they required a prespecified event (9 articles proposing 12 methods), or could be applied to emerging events (35 articles proposing 61 methods); and if they were (group) sequential methods (10 articles proposing 12 methods) or methods to perform final/one analyses (34 articles proposing 61 methods).

CONCLUSIONS

This review highlighted that a broad range of methods exist for AE analysis. Immediate implementation of some of these could lead to improved inference for AE data in RCTs. For example, a well-designed graphic can be an effective means to communicate complex AE data and methods appropriate for counts, time-to-event data and that avoid dichotomising continuous outcomes can improve efficiencies in analysis. Previous research has shown that adoption of such methods in the scientific press is limited and that strategies to support change are needed.

TRIAL REGISTRATION

PROSPERO registration: https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=97442.

Survival analysis for AdVerse events with VarYing follow-up times (SAVVY): Rationale and statistical concept of a meta-analytic study

The assessment of safety is an important aspect of the evaluation of new therapies in clinical trials, with analyses of adverse events being an essential part of this. Standard methods for the analysis of adverse events such as the incidence proportion, that is the number of patients with a specific adverse event out of all patients in the treatment groups, do not account for both varying follow-up times and competing risks. Alternative approaches such as the Aalen-Johansen estimator of the cumulative incidence function have been suggested. Theoretical arguments and numerical evaluations support the application of these more advanced methodology, but as yet there is to our knowledge only insufficient empirical evidence whether these methods would lead to different conclusions in safety evaluations. The Survival analysis for AdVerse events with VarYing follow-up times (SAVVY) project strives to close this gap in evidence by conducting a meta-analytical study to assess the impact of the methodology on the conclusion of the safety assessment empirically. Here we present the rationale and statistical concept of the empirical study conducted as part of the SAVVY project. The statistical methods are presented in unified notation, and examples of their implementation in R and SAS are provided.

Data monitoring committees for clinical trials evaluating treatments of COVID-19

The first cases of coronavirus disease 2019 (COVID-19) were reported in December 2019 and the outbreak of SARS-CoV-2 was declared a pandemic in March 2020 by the World Health Organization. This sparked a plethora of investigations into diagnostics and vaccination for SARS-CoV-2, as well as treatments for COVID-19. Since COVID-19 is a severe disease associated with a high mortality, clinical trials in this disease should be monitored by a data monitoring committee (DMC), also known as data safety monitoring board (DSMB). DMCs in this indication face a number of challenges including fast recruitment requiring an unusually high frequency of safety reviews, more frequent use of complex designs and virtually no prior experience with the disease. In this paper, we provide a perspective on the work of DMCs for clinical trials of treatments for COVID-19. More specifically, we discuss organizational aspects of setting up and running DMCs for COVID-19 trials, in particular for trials with more complex designs such as platform trials or adaptive designs. Furthermore, statistical aspects of monitoring clinical trials of treatments for COVID-19 are considered. Some recommendations are made regarding the presentation of the data, stopping rules for safety monitoring and the use of external data. The proposed stopping boundaries are assessed in a simulation study motivated by clinical trials in COVID-19.

Clinical Trial Drug Safety Assessment for Studies and Submissions Impacted by COVID-19

Abstract-In this article, we provide guidance on how safety analyses and reporting of clinical trial safety data may need to be modified, given potential impact from the COVID-19 pandemic. Impact could include missed visits, alternative methods for assessments (such as virtual visits), alternative locations for assessments (such as local labs), and study drug interruptions. Starting from the safety analyses typically included in Clinical Study Reports for Phase 2-4 clinical trials and integrated submission documents, we assess what modifications might be needed. If the impact from COVID-19 affects treatment arms equally, analyses of adverse events from controlled data can, to a large extent, remain unchanged. However, interpretation of summaries from uncontrolled data (summaries that include open-label extension data) will require even more caution than usual. Special consideration will be needed for safety topics of interest, especially events expected to have a higher incidence due to a COVID-19 infection or due to quarantine or travel restrictions (e.g., depression). Analyses of laboratory measurements may need to be modified to account for the combination of measurements from local and central laboratories.

Assessing the Impact of COVID-19 on the Clinical Trial Objective and Analysis of Oncology Clinical Trials-Application of the Estimand Framework

Abstract-Coronavirus disease 2019 (COVID-19) outbreak has rapidly evolved into a global pandemic. The impact of COVID-19 on patient journeys in oncology represents a new risk to interpretation of trial results and its broad applicability for future clinical practice. We identify key intercurrent events (ICEs) that may occur due to COVID-19 in oncology clinical trials with a focus on time-to-event endpoints and discuss considerations pertaining to the other estimand attributes introduced in the ICH E9 addendum. We propose strategies to handle COVID-19 related ICEs, depending on their relationship with malignancy and treatment and the interpretability of data after them. We argue that the clinical trial objective from a world without COVID-19 pandemic remains valid. The estimand framework provides a common language to discuss the impact of COVID-19 in a structured and transparent manner. This demonstrates that the applicability of the framework may even go beyond what it was initially intended for.

Statistical Issues and Recommendations for Clinical Trials Conducted During the COVID-19 Pandemic

Abstract-The COVID-19 pandemic has had and continues to have major impacts on planned and ongoing clinical trials. Its effects on trial data create multiple potential statistical issues. The scale of impact is unprecedented, but when viewed individually, many of the issues are well defined and feasible to address. A number of strategies and recommendations are put forward to assess and address issues related to estimands, missing data, validity and modifications of statistical analysis methods, need for additional analyses, ability to meet objectives and overall trial interpretability.

A Framework for Safety Evaluation Throughout the Product Development Life-Cycle

Evaluation of the safety profile of medicines is moving from a more reactive approach, where safety experts and statisticians have been primarily focusing on the review of clinical trial data and spontaneous reports, to a more proactive endeavor with cross-functional teams strategically evolving their understanding of the safety profile. They do this by anticipating the ultimate benefit-risk profile and its related risk management implications from the start of development. The proposed approach is based on assessments of integrated program-level safety data. These data stem from multiple sources such as preclinical information; clinical and spontaneous adverse event reports; epidemiological, real-world, and registry data; as well as, potentially, data from social media. Blended qualitative and quantitative evaluations allow integration of data from diverse sources. Adding to this, a collaborative multidisciplinary view, which is focused on continuous learning and decision-making via diverse safety management teams, ensures that companies look at their growing safety database and associated risk management implications from every relevant perspective. This multifaceted and iterative approach starts early in the development of a new medicine, continues into the post-marketing setting, and wanes as the product matures and the safety profile becomes more well understood. Not only does this satisfy regulatory requirements but, crucially, it provides the healthcare system and treated patients with a better understanding of the drug's safety profile.

Understanding current practice, identifying barriers and exploring priorities for adverse event analysis in randomised controlled trials: an online, cross-sectional survey of statisticians from academia and industry

OBJECTIVES

To gain a better understanding of current adverse event (AE) analysis practices and the reasons for the lack of use of sophisticated statistical methods for AE data analysis in randomised controlled trials (RCTs), with the aim of identifying priorities and solutions to improve practice.

DESIGN

A cross-sectional, online survey of statisticians working in clinical trials, followed up with a workshop of senior statisticians working across the UK.

PARTICIPANTS

We aimed to recruit into the survey a minimum of one statistician from each of the 51 UK Clinical Research Collaboration registered clinical trial units (CTUs) and industry statisticians from both pharmaceuticals and clinical research organisations.

OUTCOMES

To gain a better understanding of current AE analysis practices, measure awareness of specialist methods for AE analysis and explore priorities, concerns and barriers when analysing AEs.

RESULTS

Thirty-eight (38/51; 75%) CTUs, 5 (5/7; 71%) industry and 21 attendees at the 2019 Promoting Statistical Insights Conference participated in the survey. Of the 64 participants that took part, 46 participants were classified as public sector participants and 18 as industry participants. Participants indicated that they predominantly (80%) rely on subjective comparisons when comparing AEs between treatment groups. Thirty-eight per cent were aware of specialist methods for AE analysis, but only 13% had undertaken such analyses. All participants believed guidance on appropriate AE analysis and 97% thought training specifically for AE analysis is needed. These were both endorsed as solutions by workshop participants.

CONCLUSIONS

This research supports our earlier work that identified suboptimal AE analysis practices in RCTs and confirms the underuse of more sophisticated AE analysis approaches. Improvements are needed, and further research in this area is required to identify appropriate statistical methods. This research provides a unanimous call for the development of guidance, as well as training on suitable methods for AE analysis to support change.

Dynamic inference in general nested case-control designs

Nested case-control designs are attractive in studies with a time-to-event endpoint if the outcome is rare or if interest lies in evaluating expensive covariates. The appeal is that these designs restrict to small subsets of all patients at risk just prior to the observed event times. Only these small subsets need to be evaluated. Typically, the controls are selected at random and methods for time-simultaneous inference have been proposed in the literature. However, the martingale structure behind nested case-control designs allows for more powerful and flexible non-standard sampling designs. We exploit that structure to find simultaneous confidence bands based on wild bootstrap resampling procedures within this general class of designs. We show in a simulation study that the intended coverage probability is obtained for confidence bands for cumulative baseline hazard functions. We apply our methods to observational data about hospital-acquired infections.

Relapse- and Immunosuppression-Free Survival after Hematopoietic Stem Cell Transplantation: How Can We Assess Treatment Success for Complex Time-to-Event Endpoints?

In most clinical oncology trials, time-to-first-event analyses are used for efficacy assessment, which often do not capture the entire disease process. Instead, the focus may be on more complex time-to-event endpoints, such as the course of disease after the first event or endpoints occurring after randomization. We propose "relapse- and immunosuppression-free survival" (RIFS) as an innovative and clinically relevant outcome measure for assessing treatment success after hematopoietic stem cell transplant (SCT). To capture the time-dynamic relationship of multiple episodes of immunosuppressive therapy during follow-up, relapse, and nonrelapse mortality, a multistate model was developed. The statistical complexity is that the probability of RIFS is nonmonotonic over time; thus, standard time-to-first-event methodology is inappropriate for formal treatment comparisons. Instead, a generalization of the Kaplan-Meier method was used for probability estimation, and simulation-based resampling was suggested as a strategy for statistical inference. We reanalyzed data from a recently published phase III trial in 201 leukemia patients after SCT. The study evaluated long-term treatment success of standard graft-versus-host disease prophylaxis plus a pretransplant antihuman T-lymphocyte immunoglobulin compared with standard prophylaxis alone. Results suggested that treatment increased the long-term probability of RIFS by approximately 30% during the entire follow-up period, which complements the original findings. This article highlights the importance of complex endpoints in oncology, which provide deeper insight into the treatment and disease process over time. Multistate models combined with resampling are highlighted as a promising tool to evaluate treatment success beyond standard endpoints. An example code is provided in the Supplementary Materials.

Quantitative assessment of adverse events in clinical trials: Comparison of methods at an interim and the final analysis

In clinical study reports (CSRs), adverse events (AEs) are commonly summarized using the incidence proportion (IP). IPs can be calculated for all types of AEs and are often interpreted as the probability that a treated patient experiences specific AEs. Exposure time can be taken into account with time-to-event methods. Using one minus Kaplan-Meier (1-KM) is known to overestimate the AE probability in the presence of competing events (CEs). The use of a nonparametric estimator of the cumulative incidence function (CIF) has therefore been advocated as more appropriate. In this paper, we compare different methods to estimate the probability of one selected AE. In particular, we investigate whether the proposed methods provide a reasonable estimate of the AE probability at an interim analysis (IA). The characteristics of the methods in the presence of a CE are illustrated using data from a breast cancer study and we quantify the potential bias in a simulation study. At the final analysis performed for the CSR, 1-KM systematically overestimates and in most cases IP slightly underestimates the given AE probability. CIF has the lowest bias in most simulation scenarios. All methods might lead to biased estimates at the IA except for AEs with early onset. The magnitude of the bias varies with the time-to-AE and/or CE occurrence, the selection of event-specific hazards and the amount of censoring. In general, reporting AE probabilities for prespecified fixed time points is recommended.

Bootstrapping complex time-to-event data without individual patient data, with a view toward time-dependent exposures

We consider nonparametric and semiparametric resampling of multistate event histories by simulating multistate trajectories from an empirical multivariate hazard measure. One advantage of our approach is that it does not necessarily require individual patient data, but may be based on published information. This is also attractive for both study planning and simulating realistic real-world event history data in general. The concept extends to left-truncation and right-censoring mechanisms, nondegenerate initial distributions, and nonproportional as well as non-Markov settings. A special focus is on its connection to simulating survival data with time-dependent covariates. For the case of qualitative time-dependent exposures, we demonstrate that our proposal gives a more natural interpretation of how such data evolve over the course of time than many of the competing approaches. The multistate perspective avoids any latent failure time structure and sampling spaces impossible in real life, whereas its parsimony follows the principle of Occam's razor. We also suggest empirical simulation as a novel bootstrap procedure to assess estimation uncertainty in the absence of individual patient data. This is not possible for established procedures such as Efron's bootstrap. A simulation study investigating the effect of liver functionality on survival in patients with liver cirrhosis serves as a proof of concept. Example code is provided.

Limitations of the incidence density ratio as approximation of the hazard ratio

Background

Incidence density ratios (IDRs) are frequently used to account for varying follow-up times when comparing the risks of adverse events in two treatment groups. The validity of the IDR as approximation of the hazard ratio (HR) is unknown in the situation of differential average follow up by treatment group and non-constant hazard functions. Thus, the use of the IDR when individual patient data are not available might be questionable.

Methods

A simulation study was performed using various survival-time distributions with increasing and decreasing hazard functions and various situations of differential follow up by treatment group. HRs and IDRs were estimated from the simulated survival times and compared with the true HR. A rule of thumb was derived to decide in which data situations the IDR can be used as approximation of the HR.

Results

The results show that the validity of the IDR depends on the survival-time distribution, the difference between the average follow-up durations, the baseline risk, and the sample size. For non-constant hazard functions, the IDR is only an adequate approximation of the HR if the average follow-up durations of the groups are equal and the baseline risk is not larger than 25%. In the case of large differences in the average follow-up durations between the groups and non-constant hazard functions, the IDR represents no valid approximation of the HR.

Conclusions

The proposed rule of thumb allows the use of the IDR as approximation of the HR in specific data situations, when it is not possible to estimate the HR by means of adequate survival-time methods because the required individual patient data are not available. However, in general, adequate survival-time methods should be used to analyze adverse events rather than the simple IDR.

On estimands and the analysis of adverse events in the presence of varying follow-up times within the benefit assessment of therapies

The analysis of adverse events (AEs) is a key component in the assessment of a drug's safety profile. Inappropriate analysis methods may result in misleading conclusions about a therapy's safety and consequently its benefit-risk ratio. The statistical analysis of AEs is complicated by the fact that the follow-up times can vary between the patients included in a clinical trial. This paper takes as its focus the analysis of AE data in the presence of varying follow-up times within the benefit assessment of therapeutic interventions. Instead of approaching this issue directly and solely from an analysis point of view, we first discuss what should be estimated in the context of safety data, leading to the concept of estimands. Although the current discussion on estimands is mainly related to efficacy evaluation, the concept is applicable to safety endpoints as well. Within the framework of estimands, we present statistical methods for analysing AEs with the focus being on the time to the occurrence of the first AE of a specific type. We give recommendations which estimators should be used for the estimands described. Furthermore, we state practical implications of the analysis of AEs in clinical trials and give an overview of examples across different indications. We also provide a review of current practices of health technology assessment (HTA) agencies with respect to the evaluation of safety data. Finally, we describe problems with meta-analyses of AE data and sketch possible solutions.