Development and application of two semi-automated tools for targeted medical product surveillance in a distributed data network

A methodological review of the high-dimensional propensity score in comparative-effectiveness and safety-of-interventions research finds incomplete reporting relative to algorithm development and robustness

OBJECTIVES

The use of secondary databases has become popular for evaluating the effectiveness and safety of interventions in real-life settings. However, the absence of important confounders in these databases is challenging. To address this issue, the high-dimensional propensity score (hdPS) algorithm was developed in 2009. This algorithm uses proxy variables for mitigating confounding by combining information available across several healthcare dimensions. This study assessed the methodology and reporting of the hdPS in comparative effectiveness and safety research.

STUDY DESIGN AND SETTING

In this methodological review, we searched PubMed and Google Scholar from July 2009 to May 2022 for studies that used the hdPS for evaluating the effectiveness or safety of healthcare interventions. Two reviewers independently extracted study characteristics and assessed how the hdPS was applied and reported. Risk of bias was evaluated with the Risk Of Bias In Non-randomised Studies - of Interventions (ROBINS-I) tool.

RESULTS

In total, 136 studies met the inclusion criteria; the median publication year was 2018 (Q1-Q3 2016-2020). The studies included 192 datasets, mostly North American databases (n = 132, 69%). The hdPS was used in primary analysis in 120 studies (88%). Dimensions were defined in 101 studies (74%), with a median of 5 (Q1-Q3 4-6) dimensions included. A median of 500 (Q1-Q3 200-500) empirically identified covariates were selected. Regarding hdPS reporting, only 11 studies (8%) reported all recommended items. Most studies (n = 81, 60%) had a moderate overall risk of bias.

CONCLUSION

There is room for improvement in the reporting of hdPS studies, especially regarding the transparency of methodological choices that underpin the construction of the hdPS.

Improving Methods of Identifying Anaphylaxis for Medical Product Safety Surveillance Using Natural Language Processing and Machine Learning

We sought to determine whether machine learning and natural language processing (NLP) applied to electronic medical records could improve performance of automated health-care claims-based algorithms to identify anaphylaxis events using data on 516 patients with outpatient, emergency department, or inpatient anaphylaxis diagnosis codes during 2015-2019 in 2 integrated health-care institutions in the Northwest United States. We used one site's manually reviewed gold-standard outcomes data for model development and the other's for external validation based on cross-validated area under the receiver operating characteristic curve (AUC), positive predictive value (PPV), and sensitivity. In the development site 154 (64%) of 239 potential events met adjudication criteria for anaphylaxis compared with 180 (65%) of 277 in the validation site. Logistic regression models using only structured claims data achieved a cross-validated AUC of 0.58 (95% CI: 0.54, 0.63). Machine learning improved cross-validated AUC to 0.62 (0.58, 0.66); incorporating NLP-derived covariates further increased cross-validated AUCs to 0.70 (0.66, 0.75) in development and 0.67 (0.63, 0.71) in external validation data. A classification threshold with cross-validated PPV of 79% and cross-validated sensitivity of 66% in development data had cross-validated PPV of 78% and cross-validated sensitivity of 56% in external data. Machine learning and NLP-derived data improved identification of validated anaphylaxis events.

Risk of Severe Abnormal Uterine Bleeding Associated with Rivaroxaban Compared with Apixaban, Dabigatran and Warfarin

INTRODUCTION

There have been reports of clinically relevant uterine bleeding events among women of reproductive age exposed to rivaroxaban.

OBJECTIVE

The aim of this study was to compare the risk of severe abnormal uterine bleeding (SAUB) resulting in transfusion or surgical intervention among women on rivaroxaban versus apixaban, dabigatran and warfarin.

METHODS

We conducted a retrospective cohort study in the FDA's Sentinel System (10/2010-09/2015) among females aged 18+ years with venous thromboembolism (VTE), or atrial flutter/fibrillation (AF) who newly initiated a direct oral anticoagulant (DOAC; rivaroxaban, apixaban, dabigatran) or warfarin. We followed women from dispensing date until the earliest of transfusion or surgery following vaginal bleeding, disenrollment, exposure or study end date, or recorded death. We estimated hazard ratios (HRs) using Cox proportional hazards regression via propensity score stratification. Four pairwise comparisons were conducted for each intervention.

RESULTS

Overall, there was an increased risk of surgical intervention with rivaroxaban when compared with dabigatran (HR 1.19; 95% CI 1.03-1.38), apixaban (1.23; 1.04-1.47), and warfarin (1.34; 1.22-1.47). No difference in risk for surgical intervention was observed for dabigatran-apixaban comparisons. Increased risk of transfusion was observed for rivaroxaban compared with dabigatran (1.49; 1.03-2.17) only. For patients with no gynecological history, rivaroxaban was associated with risk of surgical intervention compared with dabigatran (1.22; 1.05-1.42), apixaban (1.25; 1.04-1.49), and warfarin (1.36; 1.23-1.50).

CONCLUSION

Our study found increased SAUB risk with rivaroxaban use compared with other DOACs or warfarin. Increased risk with rivaroxaban was present among women without underlying gynecological conditions. Women on anticoagulant therapy should be aware of a risk of SAUB.

Understanding when real world data can be used to replicate a clinical trial: A cross-sectional study of medications approved in 2011

PURPOSE

To determine how commonly pre-approval clinical trials could potentially be replicated using real-world data from insurance claims databases.

METHODS

We conducted a cross-sectional study of medications approved by the FDA in 2011. For each medication, we reviewed the drug's label and the details of the pivotal clinical trials supporting its approval. We assessed whether each clinical trial could be replicated using an insurance claims databases by determining whether the following pivotal trial features could be reliably captured in claims data: study outcome, inclusion criteria, exclusion criteria, and the presence of an appropriate active comparator.

RESULTS

In 2011, 28 new medications were approved. The most common disease areas were oncology (N = 8, 29%), infectious disease (N = 5, 18%), and neurology (N = 4, 14%). The primary outcome of pre-approval clinical trials was identifiable in claims databases for six (21%) of the medications. Two (ticagrelor and linagliptin) had at least 80% of inclusion and exclusion criteria that could be identified in claims databases and had an available active comparator. The non-identifiable primary outcomes were related to patient-reported symptoms (N = 9, 32%), imaging findings (N = 5, 18%), laboratory values (N = 5, 18%), or other measurements (eg, blood pressure) not typically available in insurance claims databases (N = 4, 14%).

CONCLUSIONS

Among drugs FDA-approved in 2011, two (7%) had a pre-approval trial that could be replicated using insurance claims databases. In such qualifying trials, replication using claims databases could be useful in assessing whether they provide concordant results.

Analytic and Data Sharing Options in Real-World Multidatabase Studies of Comparative Effectiveness and Safety of Medical Products

A wide range of analytic and data sharing options are available in nonexperimental multidatabase studies designed to assess the real-world benefits and risks of medical products. Researchers often consider six scientific domains when choosing among these options-study design, exposure type, outcome type, covariate summarization technique, covariate adjustment method, and data sharing approach. This article reviews available analytic and data sharing options and discusses key scientific and practical considerations when choosing among these options in multidatabase studies of comparative effectiveness and safety of medical products. The scientific considerations must be balanced against what the data-contributing sites are able or willing to share. While pooling of person-level data sets remains the most familiar and analytically flexible approach, newer analytic and data sharing approaches that share less granular summary-level information may be equally valid and preferred in some multidatabase studies, especially when sharing of person-level data is challenging or infeasible.

Reproducing Protocol-Based Studies Using Parameterizable Tools-Comparison of Analytic Approaches Used by Two Medical Product Surveillance Networks

The US Sentinel System and the Canadian Network for Observational Drug Effect Studies (CNODES) are two medical product safety surveillance networks. Using Sentinel's preprogrammed, parameterizable analytic tools, we reproduced two protocol-based studies conducted by CNODES to assess the risks of acute pancreatitis and heart failure (HF) associated with the use of incretin-based drugs, compared with use of ≥ 2 oral hypoglycemic agents. Results from the replication new-user cohort analyses aligned with those from the CNODES nested case-control studies. The adjusted hazard ratios were 0.95 (0.81-1.12; vs. 1.03 (0.87-1.22) in CNODES) for acute pancreatitis and 0.91 (0.84-1.00; vs. 0.82 (0.67-1.00) in CNODES) for HF among patients without HF history. The CNODES's common protocol approach allows studies tailored to specific safety questions, whereas the Sentinel's common data model plus pretested program approach enables more rapid analysis. Despite these differences, it is possible to obtain comparable results using both approaches.

Real-World Data Approaches for Early Detection of Potential Safety and Effectiveness Signals for Generic Substitution: A Metoprolol Extended-Release Case Study

Real-world spontaneous adverse event reports and administrative health care data were utilized as one part of a multipronged approach to verify surveillance signals related to generic drug formulations. This study used metoprolol succinate extended release as a historic case example from which to develop an analytic framework. The US Food and Drug Administration Adverse Event Reporting System was utilized for disproportionality analyses and to identify outcomes of interest. Claims data were analyzed for generic uptake, proportion of prescriptions with "dispense as written" orders, time to discontinuation or switching, and relative rates of clinical events. Adverse Event Reporting System data showed that the Medical Dictionary for Regulatory Activities terms for product quality were higher for generic metoprolol cases and that a number of clinical events were increased that could be side effects of high or low variability in drug levels. Compared to amlodipine-benazepril, which also had a first-approved generic at the same time, market share data showed that metoprolol succinate had lower utilization and more prescriptions written as dispense as written. Switching and discontinuation were generally higher for metoprolol users compared to amlodipine-benazepril users. Finally, clinical event rates were generally higher for generic versus brand metoprolol but lower for the same comparison for amlodipine-benazepril users. In the claims-based analyses, the 90-day period immediately after generic entry provided stronger signal capture than using the entire study period. This analytic framework can be implemented to actively monitor new generic formulations for potential bioequivalence failures. Signals from these analyses require confirmation (eg, via pharmacometric analyses) to be informative for regulatory action.

The Devil's in the details: Reports on reproducibility in pharmacoepidemiologic studies

PURPOSE

The U.S. Food and Drug Administration's Sentinel Initiative "modular programs" have been shown to replicate findings from conventional protocol-driven, custom-programmed studies. One such parallel assessment-dabigatran and warfarin and selected outcomes-produced concordant findings for three of four study outcomes. The effect estimates and confidence intervals for the fourth-acute myocardial infarction-had more variability as compared with other outcomes. This paper evaluates the potential sources of that variability that led to unexpected divergence in findings.

METHODS

We systematically compared the two studies and evaluated programming differences and their potential impact using a different dataset that allowed more granular data access for investigation. We reviewed the output at each of five main processing steps common in both study programs: cohort identification, propensity score estimation, propensity score matching, patient follow-up, and risk estimation.

RESULTS

Our findings point to several design features that warrant greater investigator attention when performing observational database studies: (a) treatment of recorded events (eg, diagnoses, procedures, and dispensings) co-occurring on the index date of study drug dispensing in cohort eligibility criteria and propensity score estimation and (b) construction of treatment episodes for study drugs of interest that have more complex dispensing patterns.

CONCLUSIONS

More precise and unambiguous operational definitions of all study parameters will increase transparency and reproducibility in observational database studies.

Assessment of Use of Combined Dextromethorphan and Quinidine in Patients With Dementia or Parkinson Disease After US Food and Drug Administration Approval for Pseudobulbar Affect

IMPORTANCE

In 2010, the US Food and Drug Administration (FDA) approved a combination of dextromethorphan hydrobromide and quinidine sulfate for the treatment of pseudobulbar affect after studies in patients with amyotrophic lateral sclerosis (ALS) or multiple sclerosis (MS). This medication, however, may be commonly prescribed in patients with dementia and/or Parkinson disease (PD).

OBJECTIVE

To investigate the prescribing patterns of dextromethorphan-quinidine, including trends in associated costs.

DESIGN, SETTING, AND PARTICIPANTS

This population-based cohort study of patients prescribed dextromethorphan-quinidine used data from 2 commercial insurance databases, Optum Clinformatics Data Mart and Truven Health MarketScan. The Medicare Part D Prescription Drug Program data set was used to evaluate numbers of prescriptions and total reported spending by the Centers for Medicare & Medicaid Services. Patients were included if they were prescribed dextromethorphan-quinidine from October 29, 2010, when the drug was approved, through March 1, 2017, for Optum and December 31, 2015, for Truven. Data were analyzed from December 1, 2017, through August 1, 2018.

MAIN OUTCOMES AND MEASURES

The proportion of patients prescribed dextromethorphan-quinidine with a diagnosis of MS, ALS, or dementia and/or PD, as well as the number of patients with a history of heart failure (a contraindication for the drug).

RESULTS

In the commercial health care databases, 12 858 patients filled a prescription for dextromethorphan-quinidine during the study period. Mean (SD) age was 66.0 (18.5) years, 66.7% were women, and 13.3% had a history of heart failure. Combining results from both databases, few patients had a diagnosis of MS (8.4%) or ALS (6.8%); most (57.0%) had a diagnosis of dementia and/or PD. In the Medicare Part D database, the number of patients prescribed dextromethorphan-quinidine increased 15.3-fold, from 3296 in 2011 to 50 402 in 2016. Reported spending by Centers for Medicare & Medicaid Services on this medication increased from $3.9 million in 2011 to $200.4 million in 2016.

CONCLUSIONS AND RELEVANCE

Despite approval by the FDA for pseudobulbar affect based on studies of patients with ALS or MS, dextromethorphan-quinidine appears to be primarily prescribed for patients with dementia and/or PD.

How pharmacoepidemiology networks can manage distributed analyses to improve replicability and transparency and minimize bias

Several pharmacoepidemiology networks have been developed over the past decade that use a distributed approach, implementing the same analysis at multiple data sites, to preserve privacy and minimize data sharing. Distributed networks are efficient, by interrogating data on very large populations. The structure of these networks can also be leveraged to improve replicability, increase transparency, and reduce bias. We describe some features of distributed networks using, as examples, the Canadian Network for Observational Drug Effect Studies, the Sentinel System in the USA, and the European Research Network of Pharmacovigilance and Pharmacoepidemiology. Common protocols, analysis plans, and data models, with policies on amendments and protocol violations, are key features. These tools ensure that studies can be audited and repeated as necessary. Blinding and strict conflict of interest policies reduce the potential for bias in analyses and interpretation. These developments should improve the timeliness and accuracy of information used to support both clinical and regulatory decisions.

Automated data-adaptive analytics for electronic healthcare data to study causal treatment effects

BACKGROUND

Decision makers in health care increasingly rely on nonrandomized database analyses to assess the effectiveness, safety, and value of medical products. Health care data scientists use data-adaptive approaches that automatically optimize confounding control to study causal treatment effects. This article summarizes relevant experiences and extensions.

METHODS

The literature was reviewed on the uses of high-dimensional propensity score (HDPS) and related approaches for health care database analyses, including methodological articles on their performance and improvement. Articles were grouped into applications, comparative performance studies, and statistical simulation experiments.

RESULTS

The HDPS algorithm has been referenced frequently with a variety of clinical applications and data sources from around the world. The appeal of HDPS for database research rests in 1) its superior performance in situations of unobserved confounding through proxy adjustment, 2) its predictable efficiency in extracting confounding information from a given data source, 3) its ability to automate estimation of causal treatment effects to the extent achievable in a given data source, and 4) its independence of data source and coding system. Extensions of the HDPS approach have focused on improving variable selection when exposure is sparse, using free text information and time-varying confounding adjustment.

CONCLUSION

Semiautomated and optimized confounding adjustment in health care database analyses has proven successful across a wide range of settings. Machine-learning extensions further automate its use in estimating causal treatment effects across a range of data scenarios.