New adaptive lasso approaches for variable selection in automated pharmacovigilance signal detection

Predicting the Addition of Information Regarding Clinically Significant Adverse Drug Reactions to Japanese Drug Package Inserts Using a Machine-Learning Model

PURPOSE

To develop a machine learning (ML)-based model for predicting the addition of clinically significant adverse reaction (CSAR)-associated information to drug package inserts (PIs) based on information of adverse drug reaction (ADR) cases during the post-marketing stage in Japan.

METHODS

We collected data on CSARs added to PIs from August 2011 to March 2020. ADR cases that led to CSARs resulting in PI revisions were considered as a positive case, and ML was used to construct a binary classification model to predict the PI revisions. We selected 34 features based on the ADR aggregate data collected 6 months before PI revisions. Prediction performance was evaluated using the Matthews correlation coefficient (MCC).

RESULTS

We found CSAR information added to PIs in 617 cases, 334 of which were due to the accumulation of domestic cases, and used only domestic case data for the prediction model. Among prediction models developed using several kinds of algorithms, the support vector machine with the radial basis function kernel with feature selection showed the highest predictive performance, having an MCC of 0.938 for the cross-validation and 0.922 for the test dataset. The feature with the highest importance in the model was the "average number of patients reported per quarter."

CONCLUSION

Our model accurately predicted PI revisions using information on ADR cases that occurred 6 months before. This is the first ML model that can predict the necessary safety measures and is an efficient method for guiding the decision to adopt additional safety measures early.

On the use of cross-validation for the calibration of the adaptive lasso

Cross-validation is the standard method for hyperparameter tuning, or calibration, of machine learning algorithms. The adaptive lasso is a popular class of penalized approaches based on weighted L1 -norm penalties, with weights derived from an initial estimate of the model parameter. Although it violates the paramount principle of cross-validation, according to which no information from the hold-out test set should be used when constructing the model on the training set, a "naive" cross-validation scheme is often implemented for the calibration of the adaptive lasso. The unsuitability of this naive cross-validation scheme in this context has not been well documented in the literature. In this work, we recall why the naive scheme is theoretically unsuitable and how proper cross-validation should be implemented in this particular context. Using both synthetic and real-world examples and considering several versions of the adaptive lasso, we illustrate the flaws of the naive scheme in practice. In particular, we show that it can lead to the selection of adaptive lasso estimates that perform substantially worse than those selected via a proper scheme in terms of both support recovery and prediction error. In other words, our results show that the theoretical unsuitability of the naive scheme translates into suboptimality in practice, and call for abandoning it.