Statistical analyses of adverse event data from clinical trials. Special emphasis on serious events

Survival analysis for AdVerse events with VarYing follow-up times (SAVVY): summary of findings and assessment of existing guidelines

BACKGROUND

The SAVVY project aims to improve the analyses of adverse events (AEs) in clinical trials through the use of survival techniques appropriately dealing with varying follow-up times and competing events (CEs). This paper summarizes key features and conclusions from the various SAVVY papers.

METHODS

Summarizing several papers reporting theoretical investigations using simulations and an empirical study including randomized clinical trials from several sponsor organizations, biases from ignoring varying follow-up times or CEs are investigated. The bias of commonly used estimators of the absolute (incidence proportion and one minus Kaplan-Meier) and relative (risk and hazard ratio) AE risk is quantified. Furthermore, we provide a cursory assessment of how pertinent guidelines for the analysis of safety data deal with the features of varying follow-up time and CEs.

RESULTS

SAVVY finds that for both, avoiding bias and categorization of evidence with respect to treatment effect on AE risk into categories, the choice of the estimator is key and more important than features of the underlying data such as percentage of censoring, CEs, amount of follow-up, or value of the gold-standard.

CONCLUSIONS

The choice of the estimator of the cumulative AE probability and the definition of CEs are crucial. Whenever varying follow-up times and/or CEs are present in the assessment of AEs, SAVVY recommends using the Aalen-Johansen estimator (AJE) with an appropriate definition of CEs to quantify AE risk. There is an urgent need to improve pertinent clinical trial reporting guidelines for reporting AEs so that incidence proportions or one minus Kaplan-Meier estimators are finally replaced by the AJE with appropriate definition of CEs.

Tree-temporal scan statistics for safety signal detection in vaccine clinical trials

The evaluation of safety is critical in all clinical trials. However, the quantitative analysis of safety data in clinical trials poses statistical difficulties because of multiple potentially overlapping endpoints. Tree-temporal scan statistic approaches address this issue and have been widely employed in other data sources, but not to date in clinical trials. We evaluated the performance of three complementary scan statistical methods for routine quantitative safety signal detection: the self-controlled tree-temporal scan (SCTTS), a tree-temporal scan based on group comparison (BGTTS), and a log-rank based tree-temporal scan (LgRTTS). Each method was evaluated using data from two phase III clinical trials, and simulated data (simulation study). In the case study, the reference set was adverse events (AEs) in the Reference Safety Information of the evaluated vaccine. The SCTTS method had higher sensitivity than other methods, and after dose 1 detected 80 true positives (TP) with a positive predictive value (PPV) of 60%. The LgRTTS detected 49 TPs with 69% PPV. The BGTTS had 90% of PPV with 38 TPs. In the simulation study, with simulated reference sets of AEs, the SCTTS method had good sensitivity to detect transient effects. The LgRTTS method showed the best performance for the detection of persistent effects, with high sensitivity and expected probability of type I error. These three methods provide complementary approaches to safety signal detection in clinical trials or across clinical development programmes. All three methods formally adjust for multiple testing of large numbers of overlapping endpoints without being excessively conservative.

An Overview of Regression Models for Adverse Events Analysis

Over the last few years, several review articles described the adverse events analysis as sub-optimal in clinical trials. Indeed, the context surrounding adverse events analyses often imply an overwhelming number of events, a lack of power to find associations, but also a lack of specific training regarding those complex data. In randomized controlled trials or in observational studies, comparing the occurrence of adverse events according to a covariable of interest (e.g., treatment) is a recurrent question in the analysis of drug safety data, and adjusting other important factors is often relevant. This article is an overview of the existing regression models that may be considered to compare adverse events and to discuss model choice regarding the characteristics of the adverse events of interest. Many dimensions may be relevant to compare the adverse events between patients, (e.g., timing, recurrence, and severity). Recent efforts have been made to cover all of them. For chronic treatments, the occurrence of intercurrent events during the patient follow-up usually needs the modeling approach to be adapted (at least with regard to their interpretation). Moreover, analysis based on regression models should not be limited to the estimation of relative effects. Indeed, absolute risks stemming from the model should be presented systematically to help the interpretation, to validate the model, and to encourage comparison of studies.

Survival analysis for AdVerse events with VarYing follow-up times (SAVVY)-estimation of adverse event risks

Background

The SAVVY project aims to improve the analyses of adverse events (AEs), whether prespecified or emerging, in clinical trials through the use of survival techniques appropriately dealing with varying follow-up times and competing events (CEs). Although statistical methodologies have advanced, in AE analyses, often the incidence proportion, the incidence density, or a non-parametric Kaplan-Meier estimator are used, which ignore either censoring or CEs. In an empirical study including randomized clinical trials from several sponsor organizations, these potential sources of bias are investigated. The main purpose is to compare the estimators that are typically used to quantify AE risk within trial arms to the non-parametric Aalen-Johansen estimator as the gold-standard for estimating cumulative AE probabilities. A follow-up paper will consider consequences when comparing safety between treatment groups.

Methods

Estimators are compared with descriptive statistics, graphical displays, and a more formal assessment using a random effects meta-analysis. The influence of different factors on the size of deviations from the gold-standard is investigated in a meta-regression. Comparisons are conducted at the maximum follow-up time and at earlier evaluation times. CEs definition does not only include death before AE but also end of follow-up for AEs due to events related to the disease course or safety of the treatment.

Results

Ten sponsor organizations provided 17 clinical trials including 186 types of investigated AEs. The one minus Kaplan-Meier estimator was on average about 1.2-fold larger than the Aalen-Johansen estimator and the probability transform of the incidence density ignoring CEs was even 2-fold larger. The average bias using the incidence proportion was less than 5%. Assuming constant hazards using incidence densities was hardly an issue provided that CEs were accounted for. The meta-regression showed that the bias depended mainly on the amount of censoring and on the amount of CEs.

Conclusions

The choice of the estimator of the cumulative AE probability and the definition of CEs are crucial. We recommend using the Aalen-Johansen estimator with an appropriate definition of CEs whenever the risk for AEs is to be quantified and to change the guidelines accordingly.

Estimating and comparing adverse event probabilities in the presence of varying follow-up times and competing events

Safety analyses of adverse events (AEs) are important in assessing benefit-risk of therapies but are often rather simplistic compared to efficacy analyses. AE probabilities are typically estimated by incidence proportions, sometimes incidence densities or Kaplan-Meier estimation are proposed. These analyses either do not account for censoring, rely on a too restrictive parametric model, or ignore competing events. With the non-parametric Aalen-Johansen estimator as the "gold standard", that is, reference estimator, potential sources of bias are investigated in an example from oncology and in simulations, for both one-sample and two-sample scenarios. The Aalen-Johansen estimator serves as a reference, because it is the proper non-parametric generalization of the Kaplan-Meier estimator to multiple outcomes. Because of potential large variances at the end of follow-up, comparisons also consider further quantiles of the observed times. To date, consequences for safety comparisons have hardly been investigated, the impact of using different estimators for group comparisons being unclear. For example, the ratio of two both underestimating or overestimating estimators may not be comparable to the ratio of the reference, and our investigation also considers the ratio of AE probabilities. We find that ignoring competing events is more of a problem than falsely assuming constant hazards by the use of the incidence density and that the choice of the AE probability estimator is crucial for group comparisons.

Survival analysis for AdVerse events with VarYing follow-up times (SAVVY): Rationale and statistical concept of a meta-analytic study

The assessment of safety is an important aspect of the evaluation of new therapies in clinical trials, with analyses of adverse events being an essential part of this. Standard methods for the analysis of adverse events such as the incidence proportion, that is the number of patients with a specific adverse event out of all patients in the treatment groups, do not account for both varying follow-up times and competing risks. Alternative approaches such as the Aalen-Johansen estimator of the cumulative incidence function have been suggested. Theoretical arguments and numerical evaluations support the application of these more advanced methodology, but as yet there is to our knowledge only insufficient empirical evidence whether these methods would lead to different conclusions in safety evaluations. The Survival analysis for AdVerse events with VarYing follow-up times (SAVVY) project strives to close this gap in evidence by conducting a meta-analytical study to assess the impact of the methodology on the conclusion of the safety assessment empirically. Here we present the rationale and statistical concept of the empirical study conducted as part of the SAVVY project. The statistical methods are presented in unified notation, and examples of their implementation in R and SAS are provided.

On estimands and the analysis of adverse events in the presence of varying follow-up times within the benefit assessment of therapies

The analysis of adverse events (AEs) is a key component in the assessment of a drug's safety profile. Inappropriate analysis methods may result in misleading conclusions about a therapy's safety and consequently its benefit-risk ratio. The statistical analysis of AEs is complicated by the fact that the follow-up times can vary between the patients included in a clinical trial. This paper takes as its focus the analysis of AE data in the presence of varying follow-up times within the benefit assessment of therapeutic interventions. Instead of approaching this issue directly and solely from an analysis point of view, we first discuss what should be estimated in the context of safety data, leading to the concept of estimands. Although the current discussion on estimands is mainly related to efficacy evaluation, the concept is applicable to safety endpoints as well. Within the framework of estimands, we present statistical methods for analysing AEs with the focus being on the time to the occurrence of the first AE of a specific type. We give recommendations which estimators should be used for the estimands described. Furthermore, we state practical implications of the analysis of AEs in clinical trials and give an overview of examples across different indications. We also provide a review of current practices of health technology assessment (HTA) agencies with respect to the evaluation of safety data. Finally, we describe problems with meta-analyses of AE data and sketch possible solutions.

Reporting Adverse Drug Reactions in Product Labels

Product labels are intended to provide health care professionals with clear and concise prescribing information that will enhance the safe and effective use of drug products. In this manuscript, we offer suggestions to improve product labels. First, we recommend that product labels that include comparator data be changed to include adjusted incidence proportions (or adjusted incidence rates when needed and appropriate) for adverse drug reactions that are somewhat common. Second, we believe that including comparator incidence in product labels is a good practice, as it gives health care providers and patients appropriate information to put the absolute risks in perspective. Finally, we recommend changing the practice of reporting extremely rare events based on the "Rule of 3" in the Summary of Product Characteristics in Europe. We recommend that these adverse drug reactions be put in a separate table from other adverse drug reactions with a note that it is difficult to reliably estimate their incidences. In exceptional circumstances, it may be possible to present an estimate of their incidence based on postmarketing data. We believe the proposed changes could help product labels to better reflect the risk of a drug relative to a comparator.

Safety data from randomized controlled trials: applying models for recurrent events

Simple descriptive listings and inference statistics based on 2×2 tables are still the most common way of summarizing and reporting adverse events data from randomized controlled trials, although these methods do not account for differences in observation times between treatment groups. Using standard methods from survival analysis such as the Cox model or Kaplan-Meier estimates would overcome this problem but limit the analysis to the first safety-related event of each subject. As an alternative, we discuss two models for recurrent events data-the Andersen-Gill and Prentice-Williams-Peterson model-regarding their applicability to safety data from randomized controlled trials. We argue that these models can be used to estimate two different quantities: a direct treatment effect on the risk of an event (Prentice-Williams-Peterson) and a total treatment effect as sum of the direct effect and the treatment's indirect effect via the event history (Anderson-Gill). Using simulated data, we illustrate the difference between these treatment effects and analyze the performance of both models in different scenarios. Because both models are limited to the analysis of cause-specific hazards if competing risks are present, we suggest to incorporate estimates of the mean frequency of events in the analysis to additionally allow the comparison of treatment effects on absolute event probabilities. We demonstrate the application of both models and the mean frequency function to safety endpoints with an illustrative analysis of data from a randomized phase-III study. Copyright © 2016 John Wiley & Sons, Ltd.

Statistical issues in the analysis of adverse events in time-to-event data

The aim of this work is to shed some light on common issues in the statistical analysis of adverse events (AEs) in clinical trials, when the main outcome is a time-to-event endpoint. To begin, we show that AEs are always subject to competing risks. That is, the occurrence of a certain AE may be precluded by occurrence of the main time-to-event outcome or by occurrence of another (fatal) AE. This has raised concerns on 'informative' censoring. We show that, in general, neither simple proportions nor Kaplan-Meier estimates of AE occurrence should be used, but common survival techniques for hazards that censor the competing event are still valid, but incomplete analyses. They must be complemented by an analogous analysis of the competing event for inference on the cumulative AE probability. The commonly used incidence rate (or incidence density) is a valid estimator of the AE hazard assuming it to be time constant. An estimator of the cumulative AE probability can be derived if the incidence rate of AE is combined with an estimator of the competing hazard. We discuss less restrictive analyses using non-parametric and semi-parametric approaches. We first consider time-to-first-AE analyses and then briefly discuss how they can be extended to the analysis of recurrent AEs. We will give a practical presentation with illustration of the methods by a simple example. Copyright © 2016 John Wiley & Sons, Ltd.

The whey fermentation product malleable protein matrix decreases TAG concentrations in patients with the metabolic syndrome: a randomised placebo-controlled trial

Animal and human studies suggest that a malleable protein matrix (MPM) from whey decreases plasma lipid concentrations and may positively influence other components of the metabolic syndrome such as glucose metabolism and blood pressure (BP). The primary objective of this double-blind, multi-centre trial was to investigate the effects of a low-fat yoghurt supplemented with whey MPM on fasting TAG concentrations in patients with the metabolic syndrome. A total of 197 patients were randomised to receive MPM or a matching placebo yoghurt identical in protein content (15 g/d). Patients were treated during 3 months with two daily servings of 150 g yoghurt each to compare changes from baseline in efficacy variables. MPM treatment resulted in a significantly larger reduction of TAG concentrations in comparison to placebo (relative change -16%, P=0·004). The difference was even more pronounced in subjects with elevated fasting TAG (≥200 mg/dl) at baseline (-18%, P=0·005). The relative treatment difference in fasting plasma glucose was -7·1 mg/dl (P=0·089). This effect was also more pronounced in subjects with impaired fasting glucose at baseline (-11 mg/dl, P=0·03). In patients with hypertension, the relative treatment difference in systolic BP reached -5·9 mmHg (P=0·054). The relative treatment difference in body weight was -1·7 kg (P=0·015). The most common adverse events were gastrointestinal in nature. Conclusions from the present study are that consumption of a low-fat yoghurt supplemented with whey MPM twice a day over 3 months significantly reduces fasting TAG concentrations in patients with the metabolic syndrome and improves multiple other cardiovascular risk factors.

Statistical methods to analyze adverse events data of randomized clinical trials

The adverse events data of randomized clinical trials are often analyzed based on either crude incidence rates or exposure-adjusted incidence rates. These rates do not adequately account for an individual patient's profile of adverse events over the study period when an individual may remain in the trial after experiencing one or more events (i.e., occurrence of multiple events of the same kind or different kinds). Moreover, the required statistical assumptions (e.g., constant hazard rate over time) for valid estimates of incidence rates are not likely to be met in practice by adverse events data of clinical trials. A nonparametric approach called the mean cumulative function (MCF) provides a valid statistical inference on recurrent adverse event profiles of drugs in randomized clinical trials. The estimate involves no assumptions about the form of MCF. To demonstrate the applicability and utility of the MCF approach in clinical trial datasets, an adverse event dataset obtained from a clinical trial is analyzed in this article. As compared to the crude or exposure-adjusted incidence rates of adverse events, the MCF estimates facilitate more understanding of safety profiles of a drug in a randomized clinical trial.

A note on post-marketing safety study design to characterise time-dependent adverse events

PURPOSE

To investigate a suitable post-marketing safety study design, that is number of patients and duration of exposure, to well characterise adverse events (AEs) profiles under limited resources, fixed patient-months.

METHODS

A simulation study is conducted to investigate a suitable study design that can appropriately characterise the shape of the hazard function of AEs using the Weibull model. The reliability of the estimates is evaluated by referring their bias and mean squared error (MSE).

RESULTS

In general, patients should be followed for a longer period even if the number of patients is relatively small for characterising delayed AEs. Patients' drop-out affects the estimation and deteriorates its reliability. For AEs that are likely to occur soon after the exposure, a study with relatively shorter duration and large number of patients is preferable.

CONCLUSIONS

It is important to evaluate statistically the appropriate study design in planning a safety study so that a good estimate of the hazard function would be obtained.

Multivariate time-to-event analysis of multiple adverse events of drugs in integrated analyses

In each clinical trial the statistical evaluation of adverse events (AEs) is a major part of standard safety analyses. However, the analyses of AEs usually lack from adequately accounting for the occurrence of multiple, different AEs. Furthermore, predictive variables other than treatment such as age, sex and concomitant medication are often ignored. These issues can be addressed by the Cox regression as introduced by Andersen and Gill and Wei et al. A further issue arises from the fact that an ordered programme of studies is conducted during clinical testing of pharmaceutical drugs. In this paper, we therefore discuss a stratified multivariate Cox regression model that can be used in integrated summaries of safety. We derive partial maximum likelihood estimators of the model parameters which can be shown to be consistent and asymptotically normally distributed. Mainly based on a sandwich estimator of their covariance matrix several test statistics are proposed that can be used to test various null hypotheses on the underlying parameters. Their asymptotic null distributions are given. The benefit of this survival time approach for analysing AEs is illustrated by evaluating symptoms of common cold from the database of a clinical development project.

Observation of Time-Dependent Adverse Events and the Influence of Drop-Out Thereon in Long-Term Safety Studies—Simulation Study Under the Current Practice of Post-marketing Safety Evaluation in Japan

Safety assessment of a new drug should be continuously carried out in the premarketing phase as well as in the postmarketing phase. Considering the actual conditions and problems of postmarketing safety studies in Japan, i.e., the lack of attention to the extent of patients' exposure to the drug (duration and the number of patients), we simulated the number of adverse events to be observed after specified intervals of exposure. This was done by applying different sets of hazard functions for a Weibull distribution under the circumstances that a certain number of patients has dropped out, focusing on rare and delayed adverse events associated with chronically used drugs. By using the result of these simulations, we point out potential problems of underestimating adverse event rates in situations where the hazard rate of the event escalates over time. Patients drop-out from the study also deteriorates the ability to observe such time-dependent adverse events. The simulation can also serve as a useful tool to examine the necessary sample size and the duration of exposure in order to observe and characterize potentially expected adverse events. It is important to take the two key factors into consideration: the change of hazard function over time and the effect of drop-out in designing, analyzing, and evaluating safety studies for new drugs.

Respiratory adverse event profiles in cystic fibrosis placebo subjects in short- and long-term inhaled therapy trials

The frequency and nature of adverse events (AEs) are important safety endpoints in clinical trials of therapies for cystic fibrosis (CF) subjects, yet published tables of background AE rates in the CF population are not readily available. Our objective in this study was to produce tables of respiratory AE rates for placebo subjects (pediatric and adult) for inhaled therapy trials in CF subjects. Respiratory AE rates in inhaled therapy trials were computed by combining data on placebo subjects from early-phase dosing studies and middle/late-phase studies, where placebo consisted of 4 or 5 mL of inhaled saline solution. AE rates were computed as number of events divided by number of placebo-subject days of observation, and 95% confidence intervals were computed based on a Poisson model. AEs were categorized as both broad (e.g., respiratory, reactive airway disease) and specific (e.g., cough, chest tightness, hemoptysis). In short-term studies, respiratory AE rates (95% confidence interval) were 1.1(0.7, 1.6)/person-week and 1.0(0.7, 1.4)/person-week in pediatric and adult subjects, respectively. In long-term studies, respiratory AE rates were 1.7(1.6, 1.8)/person-month and 2.2(2.1, 2.3)/person-month in pediatric and adult subjects, respectively. Stepwise Poisson models were fit to determine if baseline covariates were important in predicting AE rates. Forced expiratory volume in one second (FEV(1)) percent of predicted and age in short-term studies, and FEV(1) percent predicted and gender in long-term studies were statistically important in predicting respiratory AE rates. Although these variables were statistically significant, the models' predictive abilities were low, with adjusted R(2)'s of 0.06 and 0.12 in the short- and long-term studies, respectively. Combining placebo-subject AE data recorded from multiple CF clinical trials yields better estimates of true rates of occurrence in the CF population. The tables published from this study can be used to assist those charged with safety monitoring in CF clinical trials.

Risk of serious upper gastrointestinal and cardiovascular thromboembolic complications with meloxicam

PURPOSE

To assess the risk of serious gastrointestinal and thromboembolic complications with approved doses of meloxicam.

METHODS

We pooled data from clinical trials of meloxicam at doses of 7.5 or 15 mg/d. A blinded gastrointestinal adjudication committee used prespecified criteria to identify gastric or duodenal perforation, gastric outlet obstruction, or hemodynamically important upper gastrointestinal bleeding. For analysis of thromboembolic complications, investigator-reported events were analyzed without adjudication.

RESULTS

We analyzed data from 24,196 patients from 28 trials, most of whom had been followed for up to 60 days. Of these patients, 13,118 received meloxicam (10,158 received a daily dose of 7.5 mg and 2960 received 15 mg), 5283 were treated with diclofenac 100 mg, 181 received diclofenac 150 mg, 5371 were treated with piroxicam 20 mg, and 243 received naproxen 500 mg twice daily. Patients who received 7.5 mg of meloxicam daily had a 0.03% risk of serious upper gastrointestinal events, which was significantly lower than the risk in those who received diclofenac, naproxen, or piroxicam (P <0.02). With the 15 mg daily dose of meloxicam, this risk was significantly different only when compared with piroxicam (P = 0.03). The risk of thromboembolic events in patients treated with meloxicam at either dose was lower than with diclofenac, but similar to that observed with piroxicam and naproxen.

CONCLUSION

This pooled analysis of 24,196 patients demonstrates that meloxicam has a favorable gastrointestinal and thromboembolic safety profile. However, only a small number of patients were followed for more than 60 days, and meaningful comparisons were not possible in this subgroup.

Prescription-event monitoring: methodology and recent progress

Event monitoring was first suggested 25 years ago as a way of detecting adverse reactions to drugs. Prescription-event monitoring (PEM), which has been developed by the Drug Safety Research Unit, is the first large-scale systematic post-marketing surveillance method to use event monitoring in the U.K. PEM identifies patients, who have been prescribed a particular drug, and their doctors from photocopies of National Health Service prescriptions which are processed centrally in England. A personalized follow-up questionnaire ("green form") is mailed to each patient's general practitioner, usually on the first anniversary of the initial prescription, asking for information about the patient, especially any "events" that he or she may have experienced since beginning treatment with the drug. The methodology of PEM is presented, together with examples of analyses that can be performed using results from recent studies. The problems and benefits of PEM are discussed.