Use of the false discovery rate for evaluating clinical safety data

Tree-temporal scan statistics for safety signal detection in vaccine clinical trials

The evaluation of safety is critical in all clinical trials. However, the quantitative analysis of safety data in clinical trials poses statistical difficulties because of multiple potentially overlapping endpoints. Tree-temporal scan statistic approaches address this issue and have been widely employed in other data sources, but not to date in clinical trials. We evaluated the performance of three complementary scan statistical methods for routine quantitative safety signal detection: the self-controlled tree-temporal scan (SCTTS), a tree-temporal scan based on group comparison (BGTTS), and a log-rank based tree-temporal scan (LgRTTS). Each method was evaluated using data from two phase III clinical trials, and simulated data (simulation study). In the case study, the reference set was adverse events (AEs) in the Reference Safety Information of the evaluated vaccine. The SCTTS method had higher sensitivity than other methods, and after dose 1 detected 80 true positives (TP) with a positive predictive value (PPV) of 60%. The LgRTTS detected 49 TPs with 69% PPV. The BGTTS had 90% of PPV with 38 TPs. In the simulation study, with simulated reference sets of AEs, the SCTTS method had good sensitivity to detect transient effects. The LgRTTS method showed the best performance for the detection of persistent effects, with high sensitivity and expected probability of type I error. These three methods provide complementary approaches to safety signal detection in clinical trials or across clinical development programmes. All three methods formally adjust for multiple testing of large numbers of overlapping endpoints without being excessively conservative.

Assessing the incidence and severity of drug adverse events: a Bayesian hierarchical cumulative logit model

Detection of safety signals based on multiple comparisons of adverse events (AEs) between two treatments in a clinical trial involves evaluations requiring multiplicity adjustment. A Bayesian hierarchical mixture model is a good solution to this problem as it borrows information across AEs within the same System Organ Class (SOC) and modulates extremes due merely to chance. However, the hierarchical model compares only the incidence rates of AEs, regardless of severity. In this article, we propose a three-level Bayesian hierarchical non-proportional odds cumulative logit model. Our model allows for testing the equality of incidence rate and severity for AEs between the control arm and the treatment arm while addressing multiplicities. We conduct simulation study to investigate the operating characteristics of the proposed hierarchical model. The simulation study demonstrates that the proposed method could be implemented as an extension of the Bayesian hierarchical mixture model in detecting AEs with elevated incidence rate and/or elevated severity. To illustrate, we apply our proposed method using the safety data from a phase III, two-arm randomized trial.

Safety of 9-valent human papillomavirus vaccine administered to males and females in routine use

BACKGROUND

The nine-valent human papillomavirus vaccine (HPV9, Gardasil®9) was licensed in the USA in December 2014. This study was a multiyear post-licensure study to assess HPV9 safety following routine administration.

METHODS

This retrospective cohort study compared the risk of emergency department visits and hospitalizations during the interval soon after vaccination with risk during a later interval. Kaiser Permanente Northern California (KPNC) members aged ≥ 9 years who received ≥ 1 HPV9 dose between 10/1/2015-9/30/2017 were included. Outcomes were grouped into predefined diagnostic categories. We compared the odds of events in postvaccination risk intervals (days 0-14, days 1-60) with odds of events during control intervals (days 61-75, days 61-120) using conditional logistic regression. We characterized prespecified events on the day of vaccination (allergic reaction and syncope) and all deaths in the study period.

RESULTS

The study included 215,965 individuals receiving ≥ 1 dose of HPV9, of whom 140,628 had no prior HPV vaccination. We observed similar numbers of males and females and racial/ethnic diversity consistent with the underlying population. At first dose median age was 12-13 years and 77% received ≥ 1 concomitant vaccine. Eighteen event categories were significantly elevated, including skin disorders (odds ratio [OR] 1.88, 95% confidence interval [CI] 1.00, 3.53) and ill-defined conditions (OR 1.36, 95% CI 1.13, 1.64; category includes abdominal pain, allergic reactions, syncope, etc.). On review, most findings were previously known, preceded vaccination, or had other causes. Allergic reactions and syncope at vaccination were infrequent but many were potentially related. No deaths (n = 37) were considered related to HPV9 and were consistent with the background rate.

CONCLUSIONS

We did not identify new safety concerns related to HPV9. The results are consistent with the HPV9 safety profile as established from previous studies/surveillance.

REGISTRATION

European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (EUPAS13151, protocol V503-028).

Application of Bayesian approaches in drug development: starting a virtuous cycle

The pharmaceutical industry and its global regulators have routinely used frequentist statistical methods, such as null hypothesis significance testing and p values, for evaluation and approval of new treatments. The clinical drug development process, however, with its accumulation of data over time, can be well suited for the use of Bayesian statistical approaches that explicitly incorporate existing data into clinical trial design, analysis and decision-making. Such approaches, if used appropriately, have the potential to substantially reduce the time and cost of bringing innovative medicines to patients, as well as to reduce the exposure of patients in clinical trials to ineffective or unsafe treatment regimens. Nevertheless, despite advances in Bayesian methodology, the availability of the necessary computational power and growing amounts of relevant existing data that could be used, Bayesian methods remain underused in the clinical development and regulatory review of new therapies. Here, we highlight the value of Bayesian methods in drug development, discuss barriers to their application and recommend approaches to address them. Our aim is to engage stakeholders in the process of considering when the use of existing data is appropriate and how Bayesian methods can be implemented more routinely as an effective tool for doing so.

Application of multiple testing procedures for identifying relevant comorbidities, from a large set, in traumatic brain injury for research applications utilizing big health-administrative data

Background

Multiple testing procedures (MTP) are gaining increasing popularity in various fields of biostatistics, especially in statistical genetics. However, in injury surveillance research utilizing the growing amount and complexity of health-administrative data encoded in the International Statistical Classification of Diseases and Related Health Problems 10th Revision (ICD-10), few studies involve MTP and discuss their applications and challenges.

Objective

We aimed to apply MTP in the population-wide context of comorbidity preceding traumatic brain injury (TBI), one of the most disabling injuries, to find a subset of comorbidity that can be targeted in primary injury prevention.

Methods

In total, 2,600 ICD-10 codes were used to assess the associations between TBI and comorbidity, with 235,003 TBI patients, on a matched data set of patients without TBI. McNemar tests were conducted on each 2,600 ICD-10 code, and appropriate multiple testing adjustments were applied using the Benjamini-Yekutieli procedure. To study the magnitude and direction of associations, odds ratios with 95% confidence intervals were constructed.

Results

Benjamini-Yekutieli procedure captured 684 ICD-10 codes, out of 2,600, as codes positively associated with a TBI event, reducing the effective number of codes for subsequent analysis and comprehension.

Conclusion

Our results illustrate the utility of MTP for data mining and dimension reduction in TBI research utilizing big health-administrative data to support injury surveillance research and generate ideas for injury prevention.

BAHAMA: A Bayesian Hierarchical Model for the Detection of MedDRA®-Coded Adverse Events in Randomized Controlled Trials

INTRODUCTION

Patients participating in randomized controlled trials (RCTs) are susceptible to a wide range of different adverse events (AE) during the RCT. MedDRA® is a hierarchical standardization terminology to structure the AEs reported in an RCT. The lowest level in the MedDRA hierarchy is a single medical event, and every higher level is the aggregation of the lower levels.

METHOD

We propose a multi-stage Bayesian hierarchical Poisson model for estimating MedDRA-coded AE rate ratios (RRs). To deal with rare AEs, we introduce data aggregation at a higher level within the MedDRA structure and based on thresholds on incidence and MedDRA structure.

RESULTS

With simulations, we showed the effects of this data aggregation process and the method's performance. Furthermore, an application to a real example is provided and compared with other methods.

CONCLUSION

We showed the benefit of using the full MedDRA structure and using aggregated data. The proposed model, as well as the pre-processing, is implemented in an R-package: BAHAMA.

Safety of 4-valent human papillomavirus vaccine in males: a large observational post-marketing study

The 4-valent human papillomavirus (HPV) vaccine (4vHPV vaccine), Gardasil®, is indicated for the prevention of several HPV-related diseases. The objective was to assess the safety of 4vHPV vaccine administered to males as part of routine care. The study used a US health insurance claims database, and included males, age 9 to 26 years, who initiated 4vHPV between October 2009 and December 2016. General safety outcomes were identified using ICD diagnosis codes associated with emergency room visits and hospitalizations in the claims database in risk periods (Days 1–60 and Days 1–14 following vaccine administration) and self-comparison periods (Days 91–150 and 91–104 for the Days 1–60 and Days 1–14 analysis, respectively). Incidence rates (IRs) and relative rates (RRs) with 95% confidence intervals (CIs) were calculated comparing the risk and self-comparison periods. In this study, 114,035 males initiated 4vHPV vaccine and received 202,737 doses. Using the 60-day time window, 5 outcomes had significantly elevated RRs after accounting for multiple comparisons: ear conditions (RR 1.28, 95% CI 1.03–1.59); otitis media and related conditions (RR 1.65, 95% CI 1.09–2.54); cellulitis and abscess of arm (RR 2.17, 95% CI 1.06–4.72); intracranial injury (RR 1.23, 95% CI 1.01–1.50); and concussion (RR 1.29, 95% CI 1.05–1.59). A higher rate of allergic reactions was noted on the day of 4vHPV vaccine receipt compared to other vaccines (21.07 events per 10,000 doses, 95% CI 18.89–23.44 versus 11.44 per 10,000 doses, 95% CI 9.84–13.22). A higher incidence rate of VTE was observed following vaccination but this association was not significant (RR 2.17, 95% CI 0.35–22.74). The 4vHPV vaccine was associated with same-day allergic reactions as well as ear infections, intracranial injury, cellulitis, and concussion within 2 months after vaccination. While allergic reaction and cellulitis are consistent with the known safety profile of 4vHPV vaccine, the association of the other outcomes were determined by an independent Safety Review Committee to be most likely a result of activities common in adolescent males that coincide with the timing of vaccination and not directly related to vaccination itself.

Implications and Contributions: The study results support the general safety of routine immunization with 4vHPV vaccine among males to prevent HPV-related diseases and cancers.

Restoring invisible and abandoned trials of gabapentin for neuropathic pain: a clinical and methodological investigation

INTRODUCTION

Gabapentin (Neurontin) is prescribed widely for conditions for which it has not been approved by regulators, including certain neuropathic pain conditions. There is limited evidence that gabapentin is safe and effective for the treatment of neuropathic pain. Published trial reports, and systematic reviews based on published trial reports, mislead patients and providers because information about gabapentin's harms has been published only partly. We confirmed that trials conducted by the drug developer have been abandoned, and we plan to conduct a restoration with support from the Restoring Invisible and Abandoned Trials Support Centre (https://restoringtrials.org/).

METHODS AND ANALYSIS

In this study, we will analyse and report the harms that were observed in six trials of gabapentin, which have not been reported publicly (eg, in journal articles). We will use clinical study reports and individual participant data to identify and report the harms observed in each individual trial and to summarise the harms observed across all six trials. We will report all adverse events observed in the included trials by sharing deidentified data and summary tables on the Open Science Framework (https://osf.io/w8puv/). Additionally, we will produce a summary report that describes differences between the randomised groups in each trial and across trials for prespecified harms outcomes.

ETHICS AND DISSEMINATION

We will use secondary data. This study was determined to be exempt from Institutional Review Board (IRB) review (protocol #1910607198).

Signal Detection in EUROmediCAT: Identification and Evaluation of Medication-Congenital Anomaly Associations and Use of VigiBase as a Complementary Source of Reference

INTRODUCTION

Knowledge on the safety of medication use during pregnancy is often sparse. Pregnant women are generally excluded from clinical trials, and there is a dependence on post-marketing surveillance to identify teratogenic medications.

AIMS

This study aimed to identify signals of potentially teratogenic medications using EUROmediCAT registry data on medication exposure in pregnancies with a congenital anomaly, and to investigate the use of VigiBase reports of adverse events of medications in the evaluation of these signals.

METHODS

Signals of medication-congenital anomaly associations were identified in EUROmediCAT (21,636 congenital anomaly cases with 32,619 medication exposures), then investigated in a subset of VigiBase (45,749 cases and 165,121 exposures), by reviewing statistical reporting patterns and VigiBase case reports. Evidence from the literature and quantitative and qualitative aspects of both datasets were considered before recommending signals as warranting further independent investigation.

RESULTS

EUROmediCAT analysis identified 49 signals of medication-congenital anomaly associations. Incorporating investigation in VigiBase and the literature, these were categorised as follows: four non-specific medications; 11 likely due to maternal disease; 11 well-established teratogens; two reviewed in previous EUROmediCAT studies with limited additional evidence; and 13 with insufficient basis for recommending follow-up. Independent investigations are recommended for eight signals: pregnen (4) derivatives with limb reduction; nitrofuran derivatives with cleft palate and patent ductus arteriosus; salicylic acid and derivatives with atresia or stenosis of other parts of the small intestine and tetralogy of Fallot; carbamazepine with atrioventricular septal defect and severe congenital heart defect; and selective beta-2-adrenoreceptor agonists with posterior urethral valve and/or prune belly.

CONCLUSION

EUROmediCAT data should continue to be used for signal detection, accompanied by information from VigiBase and review of the existing literature to prioritise signals for further independent evaluation.

Interdisciplinary Safety Evaluation for Learning and Decision-Making

The FDA IND safety reporting Final Rule (21CFR 312.32) applies to all human drugs and biological products being studied under an Investigational New Drug (IND). A sponsor must file an IND safety report for any serious unexpected suspected adverse reaction (SUSAR) of a medicinal product being investigated. Some events may be obviously drug-related (e.g., agranulocytosis, anaphylactic reaction, drug-induced hepatic injury, Stevens-Johnson Syndrome). For serious adverse events that are not interpretable as individual occurrences, additional processes and procedures need to be employed for identifying and assessing risks in the accumulating safety data. The approaches shared in this manuscript apply principally to safety reporting of events that are anticipated to occur in the patient population-regardless of study participation. For these events, the study sponsor should periodically review the data in the aggregate and make a judgment as to whether there is a reasonable possibility of an event having been caused by the study drug rather than the underlying condition of the patient or a concomitant therapy. Factors cited for consideration are the size and consistency of the difference in event frequency between the test and control groups, supportive preclinical findings, evidence of a dose response relationship, plausible mechanism of action, known class effect and occurrence of other related adverse events. Examples are provided that demonstrate the flexibility sponsors have in meeting the spirit of the Final Rule; some combination and variation of methods from the examples could be employed. The important thing, as expressed by Jacqueline Corrigan-Curay (Director of the Office of Medical Policy, Center for Drug Evaluation and Research, FDA), is to have a thoughtful process; a system in place to look for clinically important imbalances, applying the best clinical and quantitative judgment, while maintaining trial integrity (Ball et al. in Interdisciplinary aggregate assessments for IND safety reporting: a dialogue among colleagues from industry, Academia and the FDA. ASA biopharmaceutical section regulatory-industry statistics workshop, 2018).

Statistical methods for the analysis of adverse event data in randomised controlled trials: a scoping review and taxonomy

BACKGROUND

Statistical methods for the analysis of harm outcomes in randomised controlled trials (RCTs) are rarely used, and there is a reliance on simple approaches to display information such as in frequency tables. We aimed to identify whether any statistical methods had been specifically developed to analyse prespecified secondary harm outcomes and non-specific emerging adverse events (AEs).

METHODS

A scoping review was undertaken to identify articles that proposed original methods or the original application of existing methods for the analysis of AEs that aimed to detect potential adverse drug reactions (ADRs) in phase II-IV parallel controlled group trials. Methods where harm outcomes were the (co)-primary outcome were excluded. Information was extracted on methodological characteristics such as: whether the method required the event to be prespecified or could be used to screen emerging events; and whether it was applied to individual events or the overall AE profile. Each statistical method was appraised and a taxonomy was developed for classification.

RESULTS

Forty-four eligible articles proposing 73 individual methods were included. A taxonomy was developed and articles were categorised as: visual summary methods (8 articles proposing 20 methods); hypothesis testing methods (11 articles proposing 16 methods); estimation methods (15 articles proposing 24 methods); or methods that provide decision-making probabilities (10 articles proposing 13 methods). Methods were further classified according to whether they required a prespecified event (9 articles proposing 12 methods), or could be applied to emerging events (35 articles proposing 61 methods); and if they were (group) sequential methods (10 articles proposing 12 methods) or methods to perform final/one analyses (34 articles proposing 61 methods).

CONCLUSIONS

This review highlighted that a broad range of methods exist for AE analysis. Immediate implementation of some of these could lead to improved inference for AE data in RCTs. For example, a well-designed graphic can be an effective means to communicate complex AE data and methods appropriate for counts, time-to-event data and that avoid dichotomising continuous outcomes can improve efficiencies in analysis. Previous research has shown that adoption of such methods in the scientific press is limited and that strategies to support change are needed.

TRIAL REGISTRATION

PROSPERO registration: https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=97442.

Innovative trial designs and analyses for vaccine clinical development

In the past decades, the world has experienced several major virus outbreaks, e.g. West African Ebola outbreak, Zika virus in South America and most recently global coronavirus (COVID-19) pandemic. Many vaccines have been developed to prevent a variety of infectious diseases successfully. However, several infections have not been preventable so far, like COVID-19, which induces an immediate urgent need for effective vaccines. These emerging infectious diseases often pose unprecedent challenges for the global heath community as well as the conventional vaccine development paradigm. With a long and costly traditional vaccine development process, there are extensive needs in innovative vaccine trial designs and analyses, which aim to design more efficient vaccines trials. Featured with reduced development timeline, less resource consuming or improved estimate for the endpoints of interests, these more efficient trials bring effective medicine to target population in a faster and less costly way. In this paper, we will review a few vaccine trials equipped with adaptive design features, Bayesian designs that accommodate historical data borrowing, the master protocol strategy emerging during COVID-19 vaccine development, Real-World-Data (RWD) embedded trials and the correlate of protection framework and relevant research works. We will also discuss some statistical methodologies that improve the vaccine efficacy, safety and immunogenicity analyses. Innovative clinical trial designs and analyses, together with advanced research technologies and deeper understanding of the human immune system, are paving the way for the efficient development of new vaccines in the future.

Two-stage Bayesian hierarchical modeling for blinded and unblinded safety monitoring in randomized clinical trials

Background

Monitoring and reporting of drug safety during a clinical trial is essential to its success. More recent attention to drug safety has encouraged statistical methods development for monitoring and detecting potential safety signals. This paper investigates the potential impact of the process of the blinded investigator identifying a potential safety signal, which should be further investigated by the Data and Safety Monitoring Board with an unblinded safety data analysis.

Methods

In this paper, two-stage Bayesian hierarchical models are proposed for safety signal detection following a pre-specified set of interim analyses that are applied to efficacy. At stage 1, a hierarchical blinded model uses blinded safety data to detect a potential safety signal and at stage 2, a hierarchical logistic model is applied to confirm the signal with unblinded safety data.

Results

Any interim safety monitoring analysis is usually scheduled via negotiation between the trial sponsor and the Data and Safety Monitoring Board. The proposed safety monitoring process starts once 53 subjects have been enrolled into an eight-arm phase II clinical trial for the first interim analysis. Operating characteristics describing the performance of this proposed workflow are investigated using simulations based on the different scenarios.

Conclusions

The two-stage Bayesian safety procedure in this paper provides a statistical view to monitor safety during the clinical trials. The proposed two-stage monitoring model has an excellent accuracy of detecting and flagging a potential safety signal at stage 1, and with the most important feature that further action at stage 2 could confirm the safety issue.

A hierarchical testing approach for detecting safety signals in clinical trials

Detecting safety signals in clinical trial safety data is known to be challenging due to high dimensionality, rare occurrence, weak signal, and complex dependence. We propose a new hierarchical testing approach for analyzing safety data from a typical randomized clinical trial. This approach accounts for the hierarchical structure of adverse events (AEs), that is, AEs are categorized by system organ class (SOC). Our approach contains two steps: the first step tests, for each SOC, whether any AEs within this SOC are differently distributed between treatment arms; and the second step identifies signal AEs from SOCs passing the first step tests. We show the superiority, in terms of power of detecting safety signals given controlled false discovery rate, of the new approach comparing with currently available approaches through simulation studies. We also demonstrate this approach with two real data examples.

Quantitative Safety Monitoring in Clinical Trials: Application of Multiple Statistical Methodologies for Infrequent Events

BACKGROUND

There are limited quantitative approaches for evaluating rare safety outcomes from controlled clinical trials in either a blinded or unblinded setting. This manuscript demonstrates an application of three statistical methods for quantitative safety monitoring that can be implemented during any phase of a clinical trial, including open-label extension studies.

METHODS

An interactive safety monitoring (iSM) tool was developed using R language in the publicly available R-Shiny app and was implemented for three statistical methods of quantitative safety monitoring. These methods are sequential probability ratio test (SPRT), maximized SPRT (MaxSPRT), and Bayesian posterior probability threshold (BPPT). The iSM tool evaluated specific safety signals that incorporated pre-specified background rates or reference risk ratios.

RESULTS

Two sets of blinded clinical trial data were used for case studies to demonstrate the use the iSM tool. Two particular adverse events, myocardial infarction (MI) and serious infection, were monitored. Monte Carlo simulation was conducted to evaluate the operating characteristics of pre-specified parameters. It showed that after adjusting for exposure, the BPPT and MaxSPRT yielded similar results in identifying a pre-specified signals while the SPRT method failed to detect such signals.

CONCLUSION

Statistical methods shown for the case studies, as well as the application of the user-friendly iSM tool, greatly enhance the quantitative monitoring of safety events of interest in ongoing clinical trials The BPPT and MaxSPRT methods seem more sensitive in picking-up early signals than the SPRT method when the number of safety events is small.

False discovery rate control for multiple testing based on discrete p-values

For multiple testing based on discrete p-values, we propose a false discovery rate (FDR) procedure "BH+" with proven conservativeness. BH+ is at least as powerful as the BH (i.e., Benjamini-Hochberg) procedure when they are applied to superuniform p-values. Further, when applied to mid-p-values, BH+ can be more powerful than it is applied to conventional p-values. An easily verifiable necessary and sufficient condition for this is provided. BH+ is perhaps the first conservative FDR procedure applicable to mid-p-values and to p-values with general distributions. It is applied to multiple testing based on discrete p-values in a methylation study, an HIV study and a clinical safety study, where it makes considerably more discoveries than the BH procedure. In addition, we propose an adaptive version of the BH+ procedure, prove its conservativeness under certain conditions, and provide evidence on its excellent performance via simulation studies.

Bayesian hierarchical methods in the detection of potentially teratogenic first-trimester medications

PURPOSE

Bayesian hierarchical models (BHMs) have been used to identify adverse drug reactions, allowing information sharing amongst adverse reactions and drugs expected to have similar properties. This study evaluated the use of BHMs in the routine signal detection analyses of potential first-trimester teratogens, where these models have not previously been applied.

METHODS

Data on 15 058 malformed foetuses exposed to first trimester medications (1995-2011) from 13 European congenital anomaly (CA) registries were analysed. The proportion of each CA in women taking a specific medication was compared with the proportion of that CA in all other women in the dataset (55 CAs × 523 medications). BHMs were grouped by either medications or CAs or by both simultaneously, and the results compared with analysing each medication-CA combination separately and adjusting for multiplicity using a double false discovery rate (FDR) procedure. The proportions of "high-risk" medications (medications which have been shown to carry a moderate to high risk of foetal malformations) identified as potential signals were compared, as well as the total number of potential signals requiring follow up (the effective workload).

RESULTS

BHMs identified more high-risk medications than the double FDR method, but the effective workload was larger. A BHM grouping both medications and CAs, for example, identified 23% of high-risk medications compared with 14% by the double FDR; however, there was an increase from 16 to 71 potential signals requiring follow up.

CONCLUSION

For comparable effective workloads, BHMs did not outperform the double FDR, which is comparatively straightforward to implement and is therefore recommended for continued use in teratogenic signal detection analyses.

Controlling false discovery proportion in identification of drug-related adverse events from multiple system organ classes

Analyzing safety data from clinical trials to detect safety signals worth further examination involves testing multiple hypotheses, one for each observed adverse event (AE) type. There exists certain hierarchical structure for these hypotheses due to the classification of the AEs into system organ classes, and these AEs are also likely correlated. Many approaches have been proposed to identify safety signals under the multiple testing framework and tried to achieve control of false discovery rate (FDR). The FDR control concerns the expectation of the false discovery proportion (FDP). In practice, the control of the actual random variable FDP could be more relevant and has recently drawn much attention. In this paper, we proposed a two-stage procedure for safety signal detection with direct control of FDP, through a permutation-based approach for screening groups of AEs and a permutation-based approach of constructing simultaneous upper bounds for false discovery proportion. Our simulation studies showed that this new approach has controlled FDP. We demonstrate our approach using data sets derived from a drug clinical trial.

Bridging blinded and unblinded analysis for ongoing safety monitoring and evaluation

In order to better characterize the safety profile of investigational new drugs (INDs) during clinical development, more interest and attention have been paid to ongoing safety monitoring and evaluation. The 2015 US FDA IND safety reporting draft guidance compels sponsors to periodically evaluate unblinded safety data. However, maintaining the trial blind is necessary to avoid jeopardizing the validity of study findings. In this article, we propose an innovative new approach which includes analyzing both blinded and unblinded data. The proposed two-stage framework incorporates periodic analyses of blinded safety data to detect and flag adverse events that may have potential risk elevation related to experimental treatment, as well as planned unblinded analyses to quantify associations between the drug and adverse events, and to determine thresholds for referring adverse events for medical review and safety reporting.

Efficient methods for signal detection from correlated adverse events in clinical trials

It is an important and yet challenging task to identify true signals from many adverse events that may be reported during the course of a clinical trial. One unique feature of drug safety data from clinical trials, unlike data from post-marketing spontaneous reporting, is that many types of adverse events are reported by only very few patients leading to rare events. Due to the limited study size, the p-values of testing whether the rate is higher in the treatment group across all types of adverse events are in general not uniformly distributed under the null hypothesis that there is no difference between the treatment group and the placebo group. A consequence is that typically fewer than 100 α percent of the hypotheses are rejected under the null at the nominal significance level of α . The other challenge is multiplicity control. Adverse events from the same body system may be correlated. There may also be correlations between adverse events from different body systems. To tackle these challenging issues, we develop Monte-Carlo-based methods for the signal identification from patient-reported adverse events in clinical trials. The proposed methodologies account for the rare events and arbitrary correlation structures among adverse events within and/or between body systems. Extensive simulation studies demonstrate that the proposed method can accurately control the family-wise error rate and is more powerful than existing methods under many practical situations. Application to two real examples is provided.

Identifying signals of potentially harmful medications in pregnancy: use of the double false discovery rate method to adjust for multiple testing

AIMS

Surveillance of medication use in pregnancy is essential to identify associations between first trimester medications and congenital anomalies (CAs). Medications in the same Anatomical Therapeutic Chemical classes may have similar effects. We aimed to use this information to improve the detection of potential teratogens in CA surveillance data.

METHODS

Data on 15 058 malformed fetuses with first trimester medication exposures from 1995-2011 were available from EUROmediCAT, a network of European CA registries. For each medication-CA combination, the proportion of the CA in fetuses with the medication was compared to the proportion of the CA in all other fetuses in the dataset. The Australian classification system was used to identify high-risk medications in order to compare two methods of controlling the false discovery rate (FDR): a single FDR applied across all combinations, and a double FDR incorporating groupings of medications.

RESULTS

There were 28 765 potential combinations (523 medications × 55 CAs) for analysis. An FDR cut-off of 50% resulted in a reasonable effective workload, for which single FDR gave rise to eight medication signals (three high-risk medications) and double FDR 50% identified 16 signals (six high-risk). Over a range of FDR cut-offs, double FDR identified more high-risk medications as signals, for comparable effective workloads.

CONCLUSIONS

The double FDR method appears to improve the detection of potential teratogens in comparison to the single FDR, while maintaining a low risk of false positives. Use of double FDR is recommended in routine signal detection analyses of CA data.

Rethinking the Clinically Based Thresholds of TransCelerate BioPharma for Risk-Based Monitoring

BACKGROUND

The quality of data from clinical trials has received a great deal of attention in recent years. Of central importance is the need to protect the well-being of study participants and maintain the integrity of final analysis results. However, traditional approaches to assess data quality have come under increased scrutiny as providing little benefit for the substantial cost. Numerous regulatory guidance documents and industry position papers have described risk-based approaches to identify quality and safety issues. In particular, the position paper of TransCelerate BioPharma recommends defining risk thresholds to assess safety and quality risks based on past clinical experience. This exercise can be extremely time-consuming, and the resulting thresholds may only be relevant to a particular therapeutic area, patient or clinical site population. In addition, predefined thresholds cannot account for safety or quality issues where the underlying rate of observing a particular problem may change over the course of a clinical trial, and often do not consider varying patient exposure.

METHODS

In this manuscript, we appropriate rules commonly utilized for funnel plots to define a traffic-light system for risk indicators based on statistical criteria that consider the duration of patient follow-up. Further, we describe how these methods can be adapted to assess changing risk over time. Finally, we illustrate numerous graphical approaches to summarize and communicate risk, and discuss hybrid clinical-statistical approaches to allow for the assessment of risk at sites with low patient enrollment.

RESULTS

We illustrate the aforementioned methodologies for a clinical trial in patients with schizophrenia.

CONCLUSIONS

Funnel plots are a flexible graphical technique that can form the basis for a risk-based strategy to assess data integrity, while considering site sample size, patient exposure, and changing risk across time.

Cardiovascular safety of vildagliptin in patients with type 2 diabetes: A European multi-database, non-interventional post-authorization safety study

The aim of this non-interventional, multi-database, analytical cohort study was to assess the cardiovascular (CV) safety of vildagliptin vs other non-insulin antidiabetic drugs (NIADs) using real-world data from 5 European electronic healthcare databases. Patients with type 2 diabetes aged ≥18 years on NIAD treatment were enrolled. Adjusted incidence rate ratios (IRRs) and 95% confidence intervals (CIs) for the outcomes of interest (myocardial infarction [MI], acute coronary syndrome [ACS], stroke, congestive heart failure [CHF], individually and as a composite) were estimated using negative binomial regression. Approximately 2.8% of the enrolled patients (n = 738 054) used vildagliptin at any time during the study, with an average follow-up time of 1.4 years, resulting in a cumulative current vildagliptin exposure of 28 330 person-years. The adjusted IRRs (vildagliptin [±other NIADs] vs other NIADs) were in the range of 0.61 to 0.97 (MI), 0.55 to 1.60 (ACS), 0.02 to 0.77 (stroke), 0.49 to 1.03 (CHF), and 0.22 to 1.02 (composite CV outcomes). The IRRs and their 95% CIs were close to 1, demonstrating no increased risk of adverse CV events, including the risk of CHF, with vildagliptin vs other NIADs in real-world conditions.

Protocol for analyses of adverse event data from randomized controlled trials of statin therapy

The Cholesterol Treatment Trialists' (CTT) Collaboration was originally established to conduct individual participant data meta-analyses of major vascular events, cause-specific mortality, and site-specific cancers in large, long-term, randomized trials of statin therapy (and other cholesterol-modifying treatments). The results of the trials of statin therapy and their associated meta-analyses have shown that statins significantly reduce the risk of major vascular events without any increase in the risk of nonvascular causes of death or of site-specific cancer, but do produce small increases in the incidence of myopathy, diabetes, and, probably, hemorrhagic stroke. The CTT Collaboration has not previously sought data on other outcomes, and so a comprehensive meta-analysis of all adverse events recorded in each of the eligible trials has not been conducted. This protocol prospectively describes plans to extend the CTT meta-analysis data set so as to provide a more complete understanding of the nature and magnitude of any other effects of statin therapy.

Overall conceptual framework for studying the genetics of autoimmune diseases following vaccination: a regulatory perspective

The US Vaccine Adverse Event Reporting System contains case reports of autoimmune diseases (ADs) occurring following vaccinations. ADs are rare and occur in unvaccinated people, making the potential association between vaccines and ADs challenging to evaluate. Developing mechanistic pathways that link genes, immune mediators, vaccine components and ADs would be helpful for hypothesis generation, enhancing theories of biologic plausibility and grouping rare autoimmune adverse events to increase the ability to detect and evaluate safety signals. Here, we propose a conceptual framework for investigating the genetics of ADs as safety signals following vaccination, potentially contributing to the identification of relevant biomarkers. We also discuss a study design that incorporates genetic information into postmarket clinical evaluation of autoimmune adverse events following vaccination.

Anacetrapib lowers LDL by increasing ApoB clearance in mildly hypercholesterolemic subjects

BACKGROUND

Individuals treated with the cholesteryl ester transfer protein (CETP) inhibitor anacetrapib exhibit a reduction in both LDL cholesterol and apolipoprotein B (ApoB) in response to monotherapy or combination therapy with a statin. It is not clear how anacetrapib exerts these effects; therefore, the goal of this study was to determine the kinetic mechanism responsible for the reduction in LDL and ApoB in response to anacetrapib.

METHODS

We performed a trial of the effects of anacetrapib on ApoB kinetics. Mildly hypercholesterolemic subjects were randomized to background treatment of either placebo (n = 10) or 20 mg atorvastatin (ATV) (n = 29) for 4 weeks. All subjects then added 100 mg anacetrapib to background treatment for 8 weeks. Following each study period, subjects underwent a metabolic study to determine the LDL-ApoB-100 and proprotein convertase subtilisin/kexin type 9 (PCSK9) production rate (PR) and fractional catabolic rate (FCR).

RESULTS

Anacetrapib markedly reduced the LDL-ApoB-100 pool size (PS) in both the placebo and ATV groups. These changes in PS resulted from substantial increases in LDL-ApoB-100 FCRs in both groups. Anacetrapib had no effect on LDL-ApoB-100 PRs in either treatment group. Moreover, there were no changes in the PCSK9 PS, FCR, or PR in either group. Anacetrapib treatment was associated with considerable increases in the LDL triglyceride/cholesterol ratio and LDL size by NMR.

CONCLUSION

These data indicate that anacetrapib, given alone or in combination with a statin, reduces LDL-ApoB-100 levels by increasing the rate of ApoB-100 fractional clearance.

TRIAL REGISTRATION

ClinicalTrials.gov NCT00990808.

FUNDING

Merck & Co. Inc., Kenilworth, New Jersey, USA. Additional support for instrumentation was obtained from the National Center for Advancing Translational Sciences (UL1TR000003 and UL1TR000040).

Statistical Evaluation of Drug Safety Data

There has been growing awareness of the importance of the statistical evaluation of drug safety data both in the premarketing and postmarketing settings. Careful and comprehensive approaches are warranted in safety evaluation. This paper offers a high-level review of some key issues and emerging statistical methodological developments. Specifically, the following topics are discussed: prospective program-level safety planning, evaluation, and reporting; the impact of adverse event grouping on statistical analysis; the applications of Bayesian methods in safety signal detection; meta-analysis for analyzing safety data; and safety graphics. Aspects related to benefit-risk assessments are also covered.

Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods

BACKGROUND

Thresholds for statistical significance when assessing meta-analysis results are being insufficiently demonstrated by traditional 95% confidence intervals and P-values. Assessment of intervention effects in systematic reviews with meta-analysis deserves greater rigour.

METHODS

Methodologies for assessing statistical and clinical significance of intervention effects in systematic reviews were considered. Balancing simplicity and comprehensiveness, an operational procedure was developed, based mainly on The Cochrane Collaboration methodology and the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guidelines.

RESULTS

We propose an eight-step procedure for better validation of meta-analytic results in systematic reviews (1) Obtain the 95% confidence intervals and the P-values from both fixed-effect and random-effects meta-analyses and report the most conservative results as the main results. (2) Explore the reasons behind substantial statistical heterogeneity using subgroup and sensitivity analyses (see step 6). (3) To take account of problems with multiplicity adjust the thresholds for significance according to the number of primary outcomes. (4) Calculate required information sizes (≈ the a priori required number of participants for a meta-analysis to be conclusive) for all outcomes and analyse each outcome with trial sequential analysis. Report whether the trial sequential monitoring boundaries for benefit, harm, or futility are crossed. (5) Calculate Bayes factors for all primary outcomes. (6) Use subgroup analyses and sensitivity analyses to assess the potential impact of bias on the review results. (7) Assess the risk of publication bias. (8) Assess the clinical significance of the statistically significant review results.

CONCLUSIONS

If followed, the proposed eight-step procedure will increase the validity of assessments of intervention effects in systematic reviews of randomised clinical trials.

Evaluation of statistical methods for safety signal detection: a simulation study

Proactive evaluation of drug safety with systematic screening and detection is critical to protect patients' safety and important in regulatory approval of new drug indications and postmarketing communications and label renewals. In recent years, quite a few statistical methodologies have been developed to better evaluate drug safety through the life cycle of the product development. The statistical methods for flagging safety signals have been developed in two major areas - one for data collected from spontaneous reporting system, mostly in the postmarketing area, and the other for data from clinical trials. To our knowledge, the methods developed for one area have not been applied to the other one so far. In this article, we propose to utilize all such methods for flagging safety signals in both areas regardless of which specific area they were originally developed for. Therefore, we selected eight typical methods, that is, proportional reporting ratios, reporting odds ratios, the maximum likelihood ratio test, Bayesian confidence propagation neural network method, chi-square test for rates comparison, Benjamini and Hochberg procedure, new double false discovery rate control procedure, and Bayesian hierarchical mixture model for systematic comparison through simulations. The Benjamini and Hochberg procedure and new double false discovery rate control procedure perform best overall in terms of sensitivity and false discovery rate. The likelihood ratio test also performs well when the sample sizes are large.

Changes in LDL particle concentrations after treatment with the cholesteryl ester transfer protein inhibitor anacetrapib alone or in combination with atorvastatin

OBJECTIVES

Our aim was to assess the effects of the cholesteryl ester transfer protein (CETP) inhibitor anacetrapib and atorvastatin, both as monotherapy and in combination, on particle concentrations of low-density lipoproteins (LDL), very low-density lipoproteins (VLDL), and intermediate-density lipoproteins in dyslipidemic patients.

BACKGROUND

Although increases in high-density lipoproteins with CETP inhibition are well-documented, effects on atherogenic lipoprotein particle subclasses in dyslipidemic patients have not been extensively characterized.

METHODS

Ion mobility was performed on stored plasma samples collected from patients before and after treatment with anacetrapib alone (150 and 300 mg/d) or in combination with atorvastatin (20 mg/d) in a previously conducted 8-week phase IIb study.

RESULTS

Anacetrapib produced significant placebo-adjusted reductions of total LDL particles and all subfractions except for increases in very small LDL 4a and 4b. Atorvastatin reduced all LDL subfractions except LDL 4b. Results were generally additive for anacetrapib + atorvastatin. For patients treated with anacetrapib, the placebo-adjusted reduction in LDL 3a was attenuated and there was an increase in LDL 3b and 4a for those with low vs high triglyceride (TG) levels. For the atorvastatin alone vs placebo treatment comparison, there were small reductions in LDL 3a, 3b, and 4a for those with low vs high TG levels.

CONCLUSIONS

Anacetrapib and atorvastatin produced similar reductions from baseline in total LDL particles, but did not have comparable effects on all LDL particle subfractions, and neither drug reduced the smallest LDL 4b particles. The clinical significance of these changes and the differential effects on very small LDL 4a in patients with higher vs lower TG remain to be determined (clinicaltrials.gov, NCT00325455).

False discovery rate control is a recommended alternative to Bonferroni-type adjustments in health studies

OBJECTIVES

Procedures for controlling the false positive rate when performing many hypothesis tests are commonplace in health and medical studies. Such procedures, most notably the Bonferroni adjustment, suffer from the problem that error rate control cannot be localized to individual tests, and that these procedures do not distinguish between exploratory and/or data-driven testing vs. hypothesis-driven testing. Instead, procedures derived from limiting false discovery rates may be a more appealing method to control error rates in multiple tests.

STUDY DESIGN AND SETTING

Controlling the false positive rate can lead to philosophical inconsistencies that can negatively impact the practice of reporting statistically significant findings. We demonstrate that the false discovery rate approach can overcome these inconsistencies and illustrate its benefit through an application to two recent health studies.

RESULTS

The false discovery rate approach is more powerful than methods like the Bonferroni procedure that control false positive rates. Controlling the false discovery rate in a study that arguably consisted of scientifically driven hypotheses found nearly as many significant results as without any adjustment, whereas the Bonferroni procedure found no significant results.

CONCLUSION

Although still unfamiliar to many health researchers, the use of false discovery rate control in the context of multiple testing can provide a solid basis for drawing conclusions about statistical significance.

Detecting potential safety issues in large clinical or observational trials by Bayesian screening when event counts arise from poisson distributions

Patients in large clinical trials and in studies employing large observational databases report many different adverse events, most of which will not have been anticipated at the outset. Conventional hypothesis testing of between group differences for each adverse event can be problematic: Lack of significance does not mean lack of risk, the tests usually are not adjusted for multiplicity, and the data determine which hypotheses are tested. This article describes a Bayesian screening approach that does not test hypotheses, is self-adjusting for multiplicity, provides a direct assessment of the likelihood of no material drug-event association, and quantifies the strength of the observed association. The criteria for assessing drug-event associations can be determined by clinical or regulatory considerations. In contrast to conventional approaches, the diagnostic properties of this new approach can be evaluated analytically. Application of the method to findings from a vaccine trial yields results similar to those found by methods using a false discovery rate argument or a hierarchical Bayes approach. [Supplemental materials are available for this article. Go to the publisher's online edition of Journal of Biopharmaceutical Statistics for the following free supplemental resource: Appendix R: Code for calculations.].

Summarizing the incidence of adverse events using volcano plots and time intervals

BACKGROUND

Adverse event incidence analyses are a critical component for describing the safety profile of any new intervention. The results typically are presented in lengthy summary tables. For therapeutic areas where patients have frequent adverse events, analysis and interpretation are made more difficult by the sheer number and variety of events that occur. Understanding the risk in these instances becomes even more crucial.

PURPOSE

We describe a space-saving graphical summary that overcomes the limitations of traditional presentations of adverse events and improves interpretability of the safety profile.

METHODS

We present incidence analyses of adverse events graphically using volcano plots to highlight treatment differences. Data from a clinical trial of patients experiencing an aneurysmal subarachnoid hemorrhage are used for illustration. Adjustments for multiplicity are illustrated.

RESULTS

Color is used to indicate the treatment with higher incidence; bubble size represents the total number of events that occur in the treatment arms combined. Adjustments for multiple comparisons are displayed in a manner to indicate clearly those events for which the difference between treatment arms is statistically significant. Furthermore, adverse events can be displayed by time intervals, with multiple volcano plots or animation to appreciate changes in adverse event risk over time. Such presentations can emphasize early differences across treatments that may resolve later or highlight events for which treatment differences may become more substantial with longer follow-up.

LIMITATIONS

Treatment arms are compared in a pairwise fashion.

CONCLUSIONS

Volcano plots are space-saving tools that emphasize important differences between the adverse event profiles of two treatment arms. They can incorporate multiplicity adjustments in a manner that is straightforward to interpret and, by using time intervals, can illustrate how adverse event risk changes over the course of a clinical trial.

Predictors and moderators of treatment response in childhood anxiety disorders: results from the CAMS trial

OBJECTIVE

We sought to examine predictors and moderators of treatment outcomes among 488 youths ages 7-17 years (50% female; 74% ≤ 12 years) meeting Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; American Psychiatric Association, 2000) criteria for diagnoses of separation anxiety disorder, social phobia, or generalized anxiety disorder who were randomly assigned to receive either cognitive behavioral therapy (CBT), sertraline (SRT), their combination (COMB), or medication management with pill placebo (PBO) in the Child/Adolescent Anxiety Multimodal Study (CAMS).

METHOD

Six classes of predictor and moderator variables (22 variables) were identified from the literature and examined using continuous (Pediatric Anxiety Ratings Scale; PARS) and categorical (Clinical Global Impression Scale-Improvement; CGI-I) outcome measures.

RESULTS

Three baseline variables predicted better outcomes (independent of treatment condition) on the PARS, including low anxiety severity (as measured by parents and independent evaluators) and caregiver strain. No baseline variables were found to predict Week 12 responder status (CGI-I). Participants' principal diagnosis moderated treatment outcomes but only on the PARS. No baseline variables were found to moderate treatment outcomes on Week 12 responder status (CGI-I).

DISCUSSION

Overall, anxious children responded favorably to CAMS treatments. However, having more severe and impairing anxiety, greater caregiver strain, and a principal diagnosis of social phobia were associated with less favorable outcomes. Clinical implications of these findings are discussed.

Bioinformatic analysis of HIV-1 entry and pathogenesis

The evolution of human immunodeficiency virus type 1 (HIV-1) with respect to co-receptor utilization has been shown to be relevant to HIV-1 pathogenesis and disease. The CCR5-utilizing (R5) virus has been shown to be important in the very early stages of transmission and highly prevalent during asymptomatic infection and chronic disease. In addition, the R5 virus has been proposed to be involved in neuroinvasion and central nervous system (CNS) disease. In contrast, the CXCR4-utilizing (X4) virus is more prevalent during the course of disease progression and concurrent with the loss of CD4(+) T cells. The dual-tropic virus is able to utilize both co-receptors (CXCR4 and CCR5) and has been thought to represent an intermediate transitional virus that possesses properties of both X4 and R5 viruses that can be encountered at many stages of disease. The use of computational tools and bioinformatic approaches in the prediction of HIV-1 co-receptor usage has been growing in importance with respect to understanding HIV-1 pathogenesis and disease, developing diagnostic tools, and improving the efficacy of therapeutic strategies focused on blocking viral entry. Current strategies have enhanced the sensitivity, specificity, and reproducibility relative to the prediction of co-receptor use; however, these technologies need to be improved with respect to their efficient and accurate use across the HIV-1 subtypes. The most effective approach may center on the combined use of different algorithms involving sequences within and outside of the env-V3 loop. This review focuses on the HIV-1 entry process and on co-receptor utilization, including bioinformatic tools utilized in the prediction of co-receptor usage. It also provides novel preliminary analyses for enabling identification of linkages between amino acids in V3 with other components of the HIV-1 genome and demonstrates that these linkages are different between X4 and R5 viruses.

The practice of pre-marketing safety assessment in drug development

The last 15 years have seen a substantial increase in efforts devoted to safety assessment by statisticians in the pharmaceutical industry. While some of these efforts were driven by regulations and public demand for safer products, much of the motivation came from the realization that there is a strong need for a systematic approach to safety planning, evaluation, and reporting at the program level throughout the drug development life cycle. An efficient process can help us identify safety signals early and afford us the opportunity to develop effective risk minimization plan early in the development cycle. This awareness has led many pharmaceutical sponsors to set up internal systems and structures to effectively conduct safety assessment at all levels (patient, study, and program). In addition to process, tools have emerged that are designed to enhance data review and pattern recognition. In this paper, we describe advancements in the practice of safety assessment during the premarketing phase of drug development. In particular, we share examples of safety assessment practice at our respective companies, some of which are based on recommendations from industry-initiated working groups on best practice in recent years.