Detecting Signals of Disproportionate Reporting from Singapore's Spontaneous Adverse Event Reporting System: An Application of the Sequential Probability Ratio Test

Analyzing adverse drug reaction using statistical and machine learning methods: A systematic review

BACKGROUND

Adverse drug reactions (ADRs) are unintended negative drug-induced responses. Determining the association between drugs and ADRs is crucial, and several methods have been proposed to demonstrate this association. This systematic review aimed to examine the analytical tools by considering original articles that utilized statistical and machine learning methods for detecting ADRs.

METHODS

A systematic literature review was conducted based on articles published between 2015 and 2020. The keywords used were statistical, machine learning, and deep learning methods for detecting ADR signals. The study was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement (PRISMA) guidelines.

RESULTS

We reviewed 72 articles, of which 51 and 21 addressed statistical and machine learning methods, respectively. Electronic medical record (EMR) data were exclusively analyzed using the regression method. For FDA Adverse Event Reporting System (FAERS) data, components of the disproportionality method were preferable. DrugBank was the most used database for machine learning. Other methods accounted for the highest and supervised methods accounted for the second highest.

CONCLUSIONS

Using the 72 main articles, this review provides guidelines on which databases are frequently utilized and which analysis methods can be connected. For statistical analysis, >90% of the cases were analyzed by disproportionate or regression analysis with each spontaneous reporting system (SRS) data or electronic medical record (EMR) data; for machine learning research, however, there was a strong tendency to analyze various data combinations. Only half of the DrugBank database was occupied, and the k-nearest neighbor method accounted for the greatest proportion.

Comparison of Data Mining Methods for the Signal Detection of Adverse Drug Events with a Hierarchical Structure in Postmarketing Surveillance

There are several different proposed data mining methods for the postmarketing surveillance of drug safety. Adverse events are often classified into a hierarchical structure. Our objective was to compare the performance of several of these different data mining methods for adverse drug events data with a hierarchical structure. We generated datasets based on the World Health Organization's Adverse Reaction Terminology (WHO-ART) hierarchical structure. We evaluated different data mining methods for signal detection, including several frequentist methods such as reporting odds ratio (ROR), proportional reporting ratio (PRR), information component (IC), the likelihood ratio test-based method (LRT), and Bayesian methods such as gamma Poisson shrinker (GPS), Bayesian confidence propagating neural network (BCPNN), the new IC method, and the simplified Bayesian method (sB), as well as the tree-based scan statistic through an extensive simulation study. We also applied the methods to real data on two diabetes drugs, voglibose and acarbose, from the Korea Adverse event reporting system. Only the tree-based scan statistic method maintained the type I error rate at the desired level. Likelihood ratio test-based methods and Bayesian methods tended to be more conservative than other methods in the simulation study and detected fewer signals in the real data example. No method was superior to the others in terms of the statistical power and sensitivity of detecting true signals. It is recommended that those conducting drug‒adverse event surveillance use not just one method, but make a decision based on several methods.

Quantitative data mining in signal detection: the Singapore experience

Background: In Singapore, the Health Sciences Authority (HSA) reviews an average of 20,000 spontaneous adverse event (AE) reports yearly. Potential safety signals are identified manually and discussed on a weekly basis. In this study, we compared the use of four quantitative data mining (QDM) methods with weekly manual review to determine if signals of disproportionate reporting (SDRs) can improve the efficiency of manual reviews and thereby enhance drug safety signal detection.Methods: We formulated a QDM triage strategy to reduce the number of SDRs for weekly review and compared the results against those derived from manual reviews alone for the same 6-month period. We then incorporated QDM triage into the manual review workflow for the subsequent two 6-month periods and made further comparisons against QDM triage alone.Results: The incorporation of QDM triage into routine manual reviews resulted in a reduction of 20% to 30% in the number of drug-AE pairs identified for further evaluation. Sequential Probability Ratio Test (SPRT) detected more signals that mirror human manual signal detection than the other three methods.Conclusions: The adoption of QDM triage into our manual reviews is a more efficient way forward in signal detection, avoiding missing important drug safety signals.

An algorithm to detect unexpected increases in frequency of reports of adverse events in EudraVigilance

Purpose

The European Medicines Agency developed an algorithm to detect unexpected increases in frequency of reports, to enhance the ability to detect adverse events that manifest as increases in frequency, in particular quality defects, medication errors, and cases of abuse or misuse.

Methods

An algorithm based on a negative binomial time‐series regression model run on 6 sequential observations prior to the monitored period was developed to forecast monthly counts of reports. A heuristic model to capture increases in counts when the previous 4 observations were null supplemented the regression. Count data were determined at drug‐event combination. Sensitivity analyses were run to determine the effect of different methods of pooling or stratifying count data. Positive retrospective detections and positive predictive values (PPVs) were determined.

Results

The algorithm detected 8 of the 13 historical concerns, including all concerns of quality defects. The highest PPV (1.29%) resulted from increasing the lower count threshold from 3 to 5 and including literature reports in the counts. Both the regression model and the heuristic model components to the algorithm contributed to the detection of concerns. Sensitivity analysis indicates that stratification by commercial product reduces the PPV but suggests that pooling counts of related events may improve it.

Conclusion

The results are encouraging and suggest that the algorithm could be useful for the detection of concerns that manifest as changes in frequency of reporting; however, further testing, including in prospective use, is warranted.