Serially Combining Epidemiological Designs Does Not Improve Overall Signal Detection in Vaccine Safety Surveillance

A rapid cycle analytics framework for vaccine safety surveillance within a real-world data network: Experience with enhanced surveillance of the Janssen COVID-19 vaccine

OBJECTIVE

To complement and support routine pharmacovigilance, Janssen conducted rapid real-world data analyses for near real-time safety monitoring of the Janssen COVID-19 vaccine and to contextualize potential safety signals.

METHODS

Analyses were performed in four U.S. healthcare claims databases (February 2022-May 2023) using standardized algorithms for three vaccine exposures, 56 outcomes, and 93 negative controls. Three self-controlled case series and two comparative cohort variants were conducted, each with consideration of multiple at-risk periods following vaccination. Only results that passed pre-determined, standardized diagnostics were unblinded. Two evidence interpretation strategies were employed: 1) Discovery: aimed to support discovering potentially unknown associations for further investigation, correcting for multiple testing and sequential looks over time. 2) Estimation: aimed to quantify the strength of association for specific exposure-outcome pairs and assess statistical uncertainty.

RESULTS

A total of 13 outcomes of interest showed results exceeding the prespecified Discovery threshold. Guillain-Barré Syndrome (GBS) and Bell's palsy had the most consistent signaling over time, analytic methods, and data sources. GBS, an adverse drug reaction that was added to the product information in August 2021, is used as the example to demonstrate the aspects of this rapid analytic framework. Estimation results for GBS were consistent, with effect estimates in the 1-28 day risk window ranging from an incidence rate ratio of 4.0 (95 % confidence interval: 2.1-7.7) in a self-controlled design to a hazard ratio of 6.3 (3.0-13.0) in a cohort design.

CONCLUSIONS

This work demonstrates the value and feasibility of conducting rapid cycle analysis across numerous outcomes in multiple databases employing complementary methodologies over successive time points while maintaining scientific integrity. The scalability of the approach is facilitated by the a priori specification of analytic diagnostics and corresponding thresholds, which excludes analyses likely to yield unreliable results, thereby minimizing subjective interpretation and post-hoc rationalization of failed diagnostic tests.