Minimizing signal detection time in postmarket sequential analysis: balancing positive predictive value and sensitivity

Serially Combining Epidemiological Designs Does Not Improve Overall Signal Detection in Vaccine Safety Surveillance

INTRODUCTION

Vaccine safety surveillance commonly includes a serial testing approach with a sensitive method for 'signal generation' and specific method for 'signal validation.' The extent to which serial testing in real-world studies improves or hinders overall performance in terms of sensitivity and specificity remains unknown.

METHODS

We assessed the overall performance of serial testing using three administrative claims and one electronic health record database. We compared type I and II errors before and after empirical calibration for historical comparator, self-controlled case series (SCCS), and the serial combination of those designs against six vaccine exposure groups with 93 negative control and 279 imputed positive control outcomes.

RESULTS

The historical comparator design mostly had fewer type II errors than SCCS. SCCS had fewer type I errors than the historical comparator. Before empirical calibration, the serial combination increased specificity and decreased sensitivity. Type II errors mostly exceeded 50%. After empirical calibration, type I errors returned to nominal; sensitivity was lowest when the methods were combined.

CONCLUSION

While serial combination produced fewer false-positive signals compared with the most specific method, it generated more false-negative signals compared with the most sensitive method. Using a historical comparator design followed by an SCCS analysis yielded decreased sensitivity in evaluating safety signals relative to a one-stage SCCS approach. While the current use of serial testing in vaccine surveillance may provide a practical paradigm for signal identification and triage, single epidemiological designs should be explored as valuable approaches to detecting signals.

Monitoring Drug Safety in Pregnancy with Scan Statistics: A Comparison of Two Study Designs

BACKGROUND

Traditional surveillance of adverse infant outcomes following maternal medication exposures relies on pregnancy exposure registries, which are often underpowered. We characterize the statistical power of TreeScan, a data mining tool, to identify potential signals in the setting of perinatal medication exposures and infant outcomes.

METHODS

We used empirical data to inform background incidence of major congenital malformations and other birth conditions. Statistical power was calculated using two probability models compatible with TreeScan, Bernoulli and Poisson, while varying the sample size, magnitude of the risk increase, and incidence of a specified outcome. We also simulated larger referent to exposure matching ratios when using the Bernoulli model in the setting of fixed N:1 propensity score matching. Finally, we assessed the impact of outcome misclassification on power.

RESULTS

The Poisson model demonstrated greater power to detect signals than the Bernoulli model across all scenarios and suggested a sample size of 4,000 exposed pregnancies is needed to detect a twofold increase in risk of a common outcome (approximately 8 per 1,000) with 85% power. Increasing the fixed matching ratio with the Bernoulli model did not reliably increase power. An outcome definition with high sensitivity is expected to have somewhat greater power to detect signals than an outcome definition with high positive predictive value.

CONCLUSIONS

Use of the Poisson model with an outcome definition that prioritizes sensitivity may be optimal for signal detection. TreeScan is a viable method for surveillance of adverse infant outcomes following maternal medication use.

Comparison of laboratory threshold criteria in drug-induced liver injury detection algorithms for use in pharmacovigilance

PURPOSE

For the purpose of pharmacovigilance, we sought to determine the best performing laboratory threshold criteria to detect drug-induced liver injury (DILI) in the electronic medical records (EMR).

METHODS

We compared three commonly used liver chemistry criteria from the DILI expert working group (DEWG), DILI network (DILIN), and Council for International Organizations of Medical Sciences (CIOMS), based on hospital EMR for years 2010 and 2011 (42 176 admissions), using independent medical record review. The performance characteristics were compared in terms of sensitivity, specificity, positive predictive value (PPV), negative predictive value, accuracy, F-measure, and area under the receiver operating characteristic curve (AUROC).

RESULTS

DEWG had the highest PPV (5.5%, 95% CI: 4.1%-7.2%), specificity (97.0%, 95% CI: 96.8%-97.2%), accuracy (96.8%, 95% CI: 96.6%-97.0%) and F-measure (0.099). CIOMS had the highest sensitivity (74.0%, 95% CI: 64.3%-82.3%) and AUROC (85.2%, 95% CI: 80.8%-89.7%). Besides the laboratory criteria, including additional keywords in the classification algorithm improved the PPV and F-measure to a maximum of 29.0% (95% CI: 22.3%-36.5%) and 0.379, respectively.

CONCLUSIONS

More stringent criteria (DEWG and DILIN) performed better in terms of PPV, specificity, accuracy and F-measure. CIOMS performed better in terms of sensitivity. An algorithm with high sensitivity is useful in pharmacovigilance for detecting rare events and to avoid missing cases. Requiring at least two abnormal liver chemistries during hospitalization and text-word searching in the discharge summaries decreased false positives without loss in sensitivity.

Comparison of alpha-spending plans for near real-time monitoring for Guillain-Barré Syndrome after influenza vaccination during the 2010/11 influenza season

BACKGROUND

Near real-time surveillance of the influenza vaccine, which is administered to a large proportion of the US population every year, is essential to ensure safety of the vaccine. For efficient near real-time surveillance, it is key to select appropriate parameters such as monitoring start date, number of interim tests and a scheme for spending a pre-defined total alpha across the entire influenza season. Guillain-Barré Syndrome, shown to be associated with the 1976 influenza vaccine, is used to evaluate how choices of these parameters can affect whether or not a signal is detected and the time to signal. FDA has been monitoring for the risk of GBS after influenza vaccination for every influenza season since 2008.

METHODS

Using Medicare administrative data and the Updating Sequential Probability Ratio Test methodology to account for claims delay, we evaluated a number of different alpha-spending plans by varying several parameters.

RESULTS

For relative risks of 5 or greater, almost all alpha-spending plans have 100% power; however, for relative risks of 1.5 or lower, the constant and O'Brien-Fleming plans have increasingly more power. For RRs of 1.5 and greater, the Pocock plan signals earliest but would not signal at a RR of 1.25, as observed in prior influenza seasons. There were no remarkable differences across the different plans in regards to monitoring start dates defined by the number of vaccinations; reducing the number of interim tests improves performance only marginally.

CONCLUSIONS

A constant alpha-spending plan appears to be robust, in terms of power and time to detect a signal, across a range of these parameters, including alternate monitoring start dates based on either cumulative vaccinations or GBS claims observed, frequency of monitoring, hypothetical relative risks, and vaccine uptake patterns.

An Implementation and Visualization of the Tree-Based Scan Statistic for Safety Event Monitoring in Longitudinal Electronic Health Data

INTRODUCTION

Longitudinal electronic healthcare data hold great potential for drug safety surveillance. The tree-based scan statistic (TBSS), as implemented by the TreeScan^® software, allows for hypothesis-free signal detection in longitudinal data by grouping safety events according to branching, hierarchical data coding systems, and then identifying signals of disproportionate recording (SDRs) among the singular events or event groups.

OBJECTIVE

The objective of this analysis was to identify and visualize SDRs with the TBSS in historical data from patients using two antifungal drugs, itraconazole or terbinafine. By examining patients who used either itraconazole or terbinafine, we provide a conceptual replication of a previous TBSS analyses by varying methodological choices and using a data source that had not been previously used with the TBSS, i.e., the Optum Clinformatics™ claims database. With this analysis, we aimed to test a parsimonious design that could be the basis of a broadly applicable method for multiple drug and safety event pairs.

METHODS

The TBSS analysis was used to examine incident events and any itraconazole or terbinafine use among US-based patients from 2002 through 2007. Event frequencies before and after the first day of drug exposure were compared over 14- and 56-day periods of observation in a Bernoulli model with a self-controlled design. Safety events were classified into a hierarchical tree structure using the Clinical Classifications Software (CCS) which mapped International Classification of Diseases, 9th Revision (ICD-9) codes to 879 diagnostic groups. Using the TBSS, the log likelihood ratio of observed versus expected events in all groups along the CCS hierarchy were compared, and groups of events that occurred at disproportionally high frequencies were identified as potential SDRs; p-values for the potential SDRs were estimated with Monte-Carlo permutation based methods. Output from TreeScan^® was visualized and plotted as a network which followed the CCS tree structure.

RESULTS

Terbinafine use (n = 223,968) was associated with SDRs for diseases of the circulatory system (14- and 56-day p = 0.001) and heart (14-day p = 0.026 and 56-day p = 0.001) as well as coronary atherosclerosis and other heart disease (14-day p = 0.003 and 56-day p = 0.004). For itraconazole use (n = 36,025), the TBSS identified SDRs for coronary atherosclerosis and other heart disease (p = 0.002) and complications of an implanted or grafted device (14-day p = 0.001 and 56-day p < 0.05). Use of both drugs was associated with SDRs for diseases of the digestive system at 14 days (p < 0.05) and this SDR had been observed among terbinafine users in a previous TBSS analysis with a different data source. The TreeScan^® visualization facilitated the identification of the atherosclerosis and other heart disease SDRs as well as highlighting the consistency of the SDR for diseases of the digestive system across drugs and data sources.

CONCLUSION

With the TBSS, we identified potential SDRs related to the circulatory system that may reflect the cardiac risk that was described in the itraconazole product label. SDRs for diseases of the digestive system among terbinafine users were also reported in a previous signal detection analysis, although other SDRs from the previous publications were not replicated. The TBSS visualizations aided in the understanding and interpretation of the TBSS output, including the comparisons to the previous publications. In this conceptual replication, differences in the results observed in our analysis and the previous analyses could be attributable to variation in modeling and design choices as well as factors that were intrinsic to the underlying data sources. The broad consistency, but far from perfect concordance, of our results with the known safety profile of these antifungals including the risks from the itraconazole product label supports the rationale for continued investigations of signal detection methods across differing data sources and populations.

Machine learning model combining features from algorithms with different analytical methodologies to detect laboratory-event-related adverse drug reaction signals

Background

The importance of identifying and evaluating adverse drug reactions (ADRs) has been widely recognized. Many studies have developed algorithms for ADR signal detection using electronic health record (EHR) data. In this study, we propose a machine learning (ML) model that enables accurate ADR signal detection by integrating features from existing algorithms based on inpatient EHR laboratory results.

Materials and methods

To construct an ADR reference dataset, we extracted known drug–laboratory event pairs represented by a laboratory test from the EU-SPC and SIDER databases. All possible drug–laboratory event pairs, except known ones, are considered unknown. To detect a known drug–laboratory event pair, three existing algorithms—CERT, CLEAR, and PACE—were applied to 21-year inpatient EHR data. We also constructed ML models (based on random forest, L1 regularized logistic regression, support vector machine, and a neural network) that use the intermediate products of the CERT, CLEAR, and PACE algorithms as inputs and determine whether a drug–laboratory event pair is associated. For performance comparison, we evaluated the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F1-measure, and area under receiver operating characteristic (AUROC).

Results

All measures of ML models outperformed those of existing algorithms with sensitivity of 0.593–0.793, specificity of 0.619–0.796, NPV of 0.645–0.727, PPV of 0.680–0.777, F1-measure of 0.629–0.709, and AUROC of 0.737–0.816. Features related to change or distribution of shape were considered important for detecting ADR signals.

Conclusions

Improved performance of ML models indicated that applying our model to EHR data is feasible and promising for detecting more accurate and comprehensive ADR signals.

Study protocol for the dabigatran, apixaban, rivaroxaban, edoxaban, warfarin comparative effectiveness research study

Since 2010, four oral anticoagulants have been approved for marketing in addition to warfarin for treatment of thromboembolic disease. Limited head-to-head data exist comparing these treatments, leaving patients and clinicians with little guidance for selecting a strategy that balances recurrence reduction with bleeding risk. In the dabigatran, apixaban, rivaroxban, edoxaban and warfarin comparative effectiveness research study, we compare all five currently available oral anticoagulant agents for the extended treatment of deep venous thrombosis and pulmonary embolism, as well as no extended treatment, and evaluate whether results differ in specific sub-populations. As our population includes Medicare novel anticoagulant users and large numbers of commercially insured and Medicaid patients, our results will likely be transportable to the majority of US patients experiencing a DVT or pulmonary embolism. Clinical Trials registration: NCT03271450.

Continuous Post-Market Sequential Safety Surveillance with Minimum Events to Signal

The CDC Vaccine Safety Datalink project has pioneered the use of near real-time post-market vaccine safety surveillance for the rapid detection of adverse events. Doing weekly analyses, continuous sequential methods are used, allowing investigators to evaluate the data near-continuously while still maintaining the correct overall alpha level. With continuous sequential monitoring, the null hypothesis may be rejected after only one or two adverse events are observed. In this paper, we explore continuous sequential monitoring when we do not allow the null to be rejected until a minimum number of observed events have occurred. We also evaluate continuous sequential analysis with a delayed start until a certain sample size has been attained. Tables with exact critical values, statistical power and the average times to signal are provided. We show that, with the first option, it is possible to both increase the power and reduce the expected time to signal, while keeping the alpha level the same. The second option is only useful if the start of the surveillance is delayed for logistical reasons, when there is a group of data available at the first analysis, followed by continuous or near-continuous monitoring thereafter.

Assessing recording delays in general practice records to inform near real-time vaccine safety surveillance using the Clinical Practice Research Datalink (CPRD)

Purpose

Near real‐time vaccine safety surveillance (NRTVSS) is an option for post‐licensure vaccine safety assessment. NRTVSS requires timely recording of outcomes in the database used. Our main objective was to examine recording delays in the Clinical Practice Research Datalink (CPRD) for outcomes of interest for vaccine safety to inform the feasibility of NRTVSS using these data. We also evaluated completeness of recording and further assessed reporting delays for hospitalized events in CPRD.

Methods

We selected Guillain–Barré syndrome (GBS), Bell's palsy (BP), optic neuritis (ON) and febrile seizures (FS), from January 2005 to June 2014. We assessed recording delays (e.g. due to feedback from specialist referral) in stand‐alone CPRD by comparing the event and system dates and excluding delays >1 year. We used linked CPRD‐hospitalization data to further evaluate delays and completeness of recording in CPRD.

Results

Among 51 220 patients for the stand‐alone CPRD analysis (GBS: n = 830; BP: n = 12 602; ON: n = 1720; and FS: n = 36 236), most had a record entered within 1 month of the event date (GBS: 73.6%; BP: 93.4%; ON: 76.2%; and FS: 85.6%). A total of 13 482 patients, with a first record in hospital, were included for the analysis of linked data (GBS: n = 678; BP: n = 4060; ON: n = 485; and FS: n = 8321). Of these, <50% had a record in CPRD after 1 year (GBS: 41.3%; BP: 22.1%; ON: 22.4%; and FS: 41.8%).

Conclusion

This work shows that most diagnoses in CPRD for the conditions examined were recorded with delays of ≤30 days, making NRTVSS possible. The pattern of delays was condition‐specific and could be used to adjust for delays in the NRTVSS analysis. Despite low sensitivity of recording, implementing NRTVSS in CPRD is worthwhile and could be carried out, at least on a trial basis, for events of interest. © 2017 The Authors. Pharmacoepidemiology & Drug Safety Published by John Wiley & Sons Ltd.

A Synthesis of Current Surveillance Planning Methods for the Sequential Monitoring of Drug and Vaccine Adverse Effects Using Electronic Health Care Data

INTRODUCTION

The large-scale assembly of electronic health care data combined with the use of sequential monitoring has made proactive postmarket drug- and vaccine-safety surveillance possible. Although sequential designs have been used extensively in randomized trials, less attention has been given to methods for applying them in observational electronic health care database settings.

EXISTING METHODS

We review current sequential-surveillance planning methods from randomized trials, and the Vaccine Safety Datalink (VSD) and Mini-Sentinel Pilot projects-two national observational electronic health care database safety monitoring programs.

FUTURE SURVEILLANCE PLANNING

Based on this examination, we suggest three steps for future surveillance planning in health care databases: (1) prespecify the sequential design and analysis plan, using available feasibility data to reduce assumptions and minimize later changes to initial plans; (2) assess existing drug or vaccine uptake, to determine if there is adequate information to proceed with surveillance, before conducting more resource-intensive planning; and (3) statistically evaluate and clearly communicate the sequential design with all those designing and interpreting the safety-surveillance results prior to implementation. Plans should also be flexible enough to accommodate dynamic and often unpredictable changes to the database information made by the health plans for administrative purposes.

CONCLUSIONS

This paper is intended to encourage dialogue about establishing a more systematic, scalable, and transparent sequential design-planning process for medical-product safety-surveillance systems utilizing observational electronic health care databases. Creating such a framework could yield improvements over existing practices, such as designs with increased power to assess serious adverse events.

Near real-time vaccine safety surveillance using electronic health records-a systematic review of the application of statistical methods

PURPOSE

Pre-licensure studies have limited ability to detect rare adverse events (AEs) to vaccines, requiring timely post-licensure studies. With the increasing availability of electronic health records (EHR) near real-time vaccine safety surveillance using these data has emerged as an option. We reviewed methods currently used to inform development of similar systems for countries considering their introduction.

METHODS

Medline, EMBASE and Web of Science were searched, with additional searches of conference abstract books. Questionnaires were sent to organizations worldwide to ascertain unpublished studies. Eligible studies used EHR and regularly assessed pre-specified AE to vaccine(s). Key features of studies were compared descriptively.

RESULTS

From 2779 studies, 31 were included from the USA (23), UK (6), and Taiwan and New Zealand (1 each). These were published/conducted between May 2005 and April 2015. Thirty-eight different vaccines were studied, focusing mainly on influenza (47.4%), especially 2009 H1N1 vaccines. Forty-six analytic approaches were used, reflecting frequency of EHR updates and the AE studied. Poisson-based maximized sequential probability ratio test was the most common (43.5%), followed by its binomial (23.9%) and conditional versions (10.9%). Thirty-seven of 49 analyses (75.5%) mentioned control for confounding, using an adjusted expected rate (51.4% of those adjusting), stratification (16.2%) or a combination of a self-controlled design and stratification (13.5%). Guillain-Barré syndrome (11.9%), meningitis/encephalitis/myelitis (11.9%) and seizures (10.8%) were studied most often.

CONCLUSIONS

Near real-time vaccine safety surveillance using EHR has developed over the past decade but is not yet widely used. As more countries have access to EHR, it will be important that appropriate methods are selected, considering the data available and AE of interest.

Data Extraction and Management in Networks of Observational Health Care Databases for Scientific Research: A Comparison of EU-ADR, OMOP, Mini-Sentinel and MATRICE Strategies

Introduction:

We see increased use of existing observational data in order to achieve fast and transparent production of empirical evidence in health care research. Multiple databases are often used to increase power, to assess rare exposures or outcomes, or to study diverse populations. For privacy and sociological reasons, original data on individual subjects can’t be shared, requiring a distributed network approach where data processing is performed prior to data sharing.

Case Descriptions and Variation Among Sites:

We created a conceptual framework distinguishing three steps in local data processing: (1) data reorganization into a data structure common across the network; (2) derivation of study variables not present in original data; and (3) application of study design to transform longitudinal data into aggregated data sets for statistical analysis. We applied this framework to four case studies to identify similarities and differences in the United States and Europe: Exploring and Understanding Adverse Drug Reactions by Integrative Mining of Clinical Records and Biomedical Knowledge (EU-ADR), Observational Medical Outcomes Partnership (OMOP), the Food and Drug Administration’s (FDA’s) Mini-Sentinel, and the Italian network—the Integration of Content Management Information on the Territory of Patients with Complex Diseases or with Chronic Conditions (MATRICE).

Findings:

National networks (OMOP, Mini-Sentinel, MATRICE) all adopted shared procedures for local data reorganization. The multinational EU-ADR network needed locally defined procedures to reorganize its heterogeneous data into a common structure. Derivation of new data elements was centrally defined in all networks but the procedure was not shared in EU-ADR. Application of study design was a common and shared procedure in all the case studies. Computer procedures were embodied in different programming languages, including SAS, R, SQL, Java, and C++.

Conclusion:

Using our conceptual framework we found several areas that would benefit from research to identify optimal standards for production of empirical knowledge from existing databases.an opportunity to advance evidence-based care management. In addition, formalized CM outcomes assessment methodologies will enable us to compare CM effectiveness across health delivery settings.

Prospective influenza vaccine safety surveillance using fresh data in the Sentinel System

Purpose

To develop the infrastructure to conduct timely active surveillance for safety of influenza vaccines and other medical countermeasures in the Sentinel System (formerly the Mini‐Sentinel Pilot), a Food and Drug Administration‐sponsored national surveillance system that typically relies on data that are mature, settled, and updated quarterly.

Methods

Three Data Partners provided their earliest available (“fresh”) cumulative claims data on influenza vaccination and health outcomes 3–4 times on a staggered basis during the 2013–2014 influenza season, collectively producing 10 data updates. We monitored anaphylaxis in the entire population using a cohort design and seizures in children ≤4 years of age using both a self‐controlled risk interval design (primary) and a cohort design (secondary). After each data update, we conducted sequential analysis for inactivated (IIV) and live (LAIV) influenza vaccines using the Maximized Sequential Probability Ratio Test, adjusting for data‐lag.

Results

Most of the 10 sequential analyses were conducted within 6 weeks of the last care‐date in the cumulative dataset. A total of 6 682 336 doses of IIV and 782 125 doses of LAIV were captured. The primary analyses did not identify any statistical signals following IIV or LAIV. In secondary analysis, the risk of seizures was higher following concomitant IIV and PCV13 than historically after IIV in 6‐ to 23‐month‐olds (relative risk = 2.7), which requires further investigation.

Conclusions

The Sentinel System can implement a sequential analysis system that uses fresh data for medical product safety surveillance. Active surveillance using sequential analysis of fresh data holds promise for detecting clinically significant health risks early. Limitations of employing fresh data for surveillance include cost and the need for careful scrutiny of signals. © 2015 The Authors. Pharmacoepidemiology and Drug Safety Published by John Wiley & Sons Ltd.