Validation to correct for outcome misclassification bias

Validation of new medication use algorithms as proxies for worsening disease activity in patients with juvenile idiopathic arthritis

PURPOSE

To facilitate claims-based research on populations with juvenile idiopathic arthritis (JIA), we sought to validate an algorithm of new medication use as a proxy for worsening JIA disease activity.

METHODS

Using electronic health record data from three pediatric centers, we defined new JIA medication use as (re)initiation of disease-modifying antirheumatic drugs or glucocorticoids (oral or intra-articular). Data were collected from 201 randomly selected subjects with (101) or without (100) new medication use. We assessed the positive predictive value (PPV) and negative predictive value (NPV) based on a reference standard of documented worsening of JIA disease activity. The algorithm was refined to optimize test characteristics.

RESULTS

Overall, the medication-based algorithm had suboptimal performance in representing worsening JIA disease activity (PPV 69.3%, NPV 77.1%). However, algorithm performance improved for definitions specifying longer times after JIA diagnosis (≥1-year post-diagnosis: PPV 82.9%, NPV 80.0%) or after initiation of prior JIA treatment (≥1-year post-treatment: PPV 89.7%, NPV 80.0%).

CONCLUSION

An algorithm for new JIA medication use appears to be a reasonable proxy for worsening JIA disease activity, particularly when specifying new use ≥1 year since initiating a prior JIA medication. This algorithm will be valuable for conducting research on JIA populations within administrative claims databases.

Detection Algorithms for Gastrointestinal Perforation Cases in the Medical Information Database Network (MID-NET®) in Japan

BACKGROUND

The Medical Information Database Network (MID-NET®) in Japan is a vast repository providing an essential pharmacovigilance tool. Gastrointestinal perforation (GIP) is a critical adverse drug event, yet no well-established GIP identification algorithm exists in MID-NET®.

METHODS

This study evaluated 12 identification algorithms by combining ICD-10 codes with GIP therapeutic procedures. Two sites contributed 200 inpatients with GIP-suggestive ICD-10 codes (100 inpatients each), while a third site contributed 165 inpatients with GIP-suggestive ICD-10 codes and antimicrobial prescriptions. The positive predictive values (PPVs) of the algorithms were determined, and the relative sensitivity (rSn) among the 165 inpatients at the third institution was evaluated.

RESULTS

A trade-off between PPV and rSn was observed. For instance, ICD-10 code-based definitions yielded PPVs of 59.5%, whereas ICD-10 codes with CT scan and antimicrobial information gave PPVs of 56.0% and an rSn of 97.0%, and ICD-10 codes with CT scan and antimicrobial information as well as three types of operation codes produced PPVs of 84.2% and an rSn of 24.2%. The same algorithms produced statistically significant differences in PPVs among the three institutions. Combining diagnostic and procedure codes improved the PPVs. The algorithm combining ICD-10 codes with CT scan and antimicrobial information and 80 different operation codes offered the optimal balance (PPV: 61.6%, rSn: 92.4%).

CONCLUSION

This study developed valuable GIP identification algorithms for MID-NET®, revealing the trade-offs between accuracy and sensitivity. The algorithm with the most reasonable balance was determined. These findings enhance pharmacovigilance efforts and facilitate further research to optimize adverse event detection algorithms.

Risk of Anaphylaxis Among New Users of GLP-1 Receptor Agonists: A Cohort Study

OBJECTIVE

To assess risk of anaphylaxis among patients with type 2 diabetes mellitus who are initiating therapy with a glucagon-like peptide 1 receptor agonist (GLP-1 RA), with a focus on those starting lixisenatide therapy.

RESEARCH DESIGN AND METHODS

A cohort study was conducted in three large, U.S. claims databases (2017-2021). Adult (aged ≥18 years) new users of a GLP-1 RA who had type 2 diabetes mellitus and ≥6 months enrollment in the database before GLP-1 RA initiation (start of follow-up) were included. GLP-1 RAs evaluated were lixisenatide, an insulin glargine/lixisenatide fixed-ratio combination (FRC), exenatide, liraglutide or insulin degludec/liraglutide FRC, dulaglutide, and semaglutide (injectable and oral). The first anaphylaxis event during follow-up was identified using a validated algorithm. Incidence rates (IRs) and 95% CIs were calculated within each medication cohort. The unadjusted IR ratio (IRR) comparing anaphylaxis rates in the lixisenatide cohort with all other GLP-1 RAs combined was analyzed post hoc.

RESULTS

There were 696,089 new users with 456,612 person-years of exposure to GLP-1 RAs. Baseline demographics, comorbidities, and use of other prescription medications in the 6 months before the index date were similar across medication cohorts. IRs (95% CIs) per 10,000 person-years were 1.0 (0.0-5.6) for lixisenatide, 6.0 (3.6-9.4) for exenatide, 5.1 (3.7-7.0) for liraglutide, 3.9 (3.1-4.8) for dulaglutide, and 3.6 (2.6-4.9) for semaglutide. The IRR (95% CI) for the anaphylaxis rate for the lixisenatide cohort compared with the pooled other GLP-1 RA cohort was 0.24 (0.01-1.35).

CONCLUSIONS

Anaphylaxis is rare with GLP-1 RAs. Lixisenatide is unlikely to confer higher risk of anaphylaxis than other GLP-1 RAs.

Validation and impact of algorithms for identifying variables in observational studies of routinely collected data

BACKGROUND AND OBJECTIVES

Among observational studies of routinely collected health data (RCD) for exploring treatment effects, algorithms are used to identify study variables. However, the extent to which algorithms are reliable and impact the credibility of effect estimates is far from clear. This study aimed to investigate the validation of algorithms for identifying study variables from RCD, and examine the impact of alternative algorithms on treatment effects.

METHODS

We searched PubMed for observational studies published in 2018 that used RCD to explore drug treatment effects. Information regarding the reporting, validation, and interpretation of algorithms was extracted. We summarized the reporting and methodological characteristics of algorithms and validation. We also assessed the divergence in effect estimates given alternative algorithms by calculating the ratio of estimates of the primary vs. alternative analyses.

RESULTS

A total of 222 studies were included, of which 93 (41.9%) provided a complete list of algorithms for identifying participants, 36 (16.2%) for exposure, and 132 (59.5%) for outcomes, and 15 (6.8%) for all study variables including population, exposure, and outcomes. Fifty-nine (26.6%) studies stated that the algorithms were validated, and 54 (24.3%) studies reported methodological characteristics of 66 validations, among which 61 validations in 49 studies were from the cross-referenced validation studies. Of those 66 validations, 22 (33.3%) reported sensitivity and 16 (24.2%) reported specificity. A total of 63.6% of studies reporting sensitivity and 56.3% reporting specificity used test-result-based sampling, an approach that potentially biases effect estimates. Twenty-eight (12.6%) studies used alternative algorithms to identify study variables, and 24 reported the effects estimated by primary analyses and sensitivity analyses. Of these, 20% had differential effect estimates when using alternative algorithms for identifying population, 18.2% for identifying exposure, and 45.5% for classifying outcomes. Only 32 (14.4%) studies discussed how the algorithms may affect treatment estimates.

CONCLUSION

In observational studies of RCD, the algorithms for variable identification were not regularly validated, and-even if validated-the methodological approach and performance of the validation were often poor. More seriously, different algorithms may yield differential treatment effects, but their impact is often ignored by researchers. Strong efforts, including recommendations, are warranted to improve good practice.