PheValuator 2.0: Methodological improvements for the PheValuator approach to semi-automated phenotype algorithm evaluation

Semaglutide and Nonarteritic Anterior Ischemic Optic Neuropathy

Importance

Semaglutide, a glucagonlike peptide-1 receptor agonist (GLP-1RA), has recently been implicated in cases of nonarteritic anterior ischemic optic neuropathy (NAION), raising safety concerns in the treatment of type 2 diabetes (T2D).

Objective

To investigate the potential association between semaglutide and NAION in the Observational Health Data Sciences and Informatics (OHDSI) network.

Design, Setting, and Participants

This was a retrospective study across 14 databases (6 administrative claims and 8 electronic health records). Included were adults with T2D taking semaglutide, other GLP-1RA (dulaglutide, exenatide), or non-GLP-1RA medications (empagliflozin, sitagliptin, glipizide) from December 1, 2017, to December 31, 2023. The incidence proportion and rate of NAION were calculated. Association between semaglutide and NAION was assessed using 2 approaches: an active-comparator cohort design comparing new users of semaglutide with those taking other GLP-1RAs and non-GLP-1RA drugs, and a self-controlled case-series (SCCS) analysis to compare individuals' risks during exposure and nonexposure periods for each drug. The cohort design used propensity score-adjusted Cox proportional hazards models to estimate hazard ratios (HRs). The SCCS used conditional Poisson regression models to estimate incidence rate ratios (IRRs). Network-wide HR and IRR estimates were generated using a random-effects meta-analysis model.

Exposures

GLP-1RA and non-GLP-1RAs.

Main Outcomes and Measures

NAION under 2 alternative definitions based on diagnosis codes: one more inclusive and sensitive, the other more restrictive and specific.

Results

The study included 37.1 million individuals with T2D, including 810 390 new semaglutide users. Of the 43 620 new users of semaglutide in the Optum's deidentified Clinformatics Data Mart Database, 24 473 (56%) were aged 50 to 69 years, and 26 699 (61%) were female. The incidence rate of NAION was 14.5 per 100 000 person-years among semaglutide users. The HR for NAION among new users of semaglutide was not different compared with that of the non-GLP-1RAs using the sensitive NAION definition-empagliflozin (HR, 1.44; 95% CI, 0.78-2.68; P = .12), sitagliptin (HR, 1.30; 95% CI, 0.56-3.01; P = .27), and glipizide (HR, 1.23; 95% CI, 0.66-2.28; P = .25). The risk was higher only compared with patients taking empagliflozin (HR, 2.27; 95% CI, 1.16-4.46; P = .02) using the specific definition. SCCS analysis of semaglutide exposure showed an increased risk of NAION (meta-analysis IRR, 1.32; 95% CI, 1.14-1.54; P < .001).

Conclusions and Relevance

Results of this study suggest a modest increase in the risk of NAION among individuals with T2D associated with semaglutide use, smaller than that previously reported, and warranting further investigation into the clinical implications of this association.

A rapid cycle analytics framework for vaccine safety surveillance within a real-world data network: Experience with enhanced surveillance of the Janssen COVID-19 vaccine

OBJECTIVE

To complement and support routine pharmacovigilance, Janssen conducted rapid real-world data analyses for near real-time safety monitoring of the Janssen COVID-19 vaccine and to contextualize potential safety signals.

METHODS

Analyses were performed in four U.S. healthcare claims databases (February 2022-May 2023) using standardized algorithms for three vaccine exposures, 56 outcomes, and 93 negative controls. Three self-controlled case series and two comparative cohort variants were conducted, each with consideration of multiple at-risk periods following vaccination. Only results that passed pre-determined, standardized diagnostics were unblinded. Two evidence interpretation strategies were employed: 1) Discovery: aimed to support discovering potentially unknown associations for further investigation, correcting for multiple testing and sequential looks over time. 2) Estimation: aimed to quantify the strength of association for specific exposure-outcome pairs and assess statistical uncertainty.

RESULTS

A total of 13 outcomes of interest showed results exceeding the prespecified Discovery threshold. Guillain-Barré Syndrome (GBS) and Bell's palsy had the most consistent signaling over time, analytic methods, and data sources. GBS, an adverse drug reaction that was added to the product information in August 2021, is used as the example to demonstrate the aspects of this rapid analytic framework. Estimation results for GBS were consistent, with effect estimates in the 1-28 day risk window ranging from an incidence rate ratio of 4.0 (95 % confidence interval: 2.1-7.7) in a self-controlled design to a hazard ratio of 6.3 (3.0-13.0) in a cohort design.

CONCLUSIONS

This work demonstrates the value and feasibility of conducting rapid cycle analysis across numerous outcomes in multiple databases employing complementary methodologies over successive time points while maintaining scientific integrity. The scalability of the approach is facilitated by the a priori specification of analytic diagnostics and corresponding thresholds, which excludes analyses likely to yield unreliable results, thereby minimizing subjective interpretation and post-hoc rationalization of failed diagnostic tests.

Objective study validity diagnostics: a framework requiring pre-specified, empirical verification to increase trust in the reliability of real-world evidence

OBJECTIVE

Propose a framework to empirically evaluate and report validity of findings from observational studies using pre-specified objective diagnostics, increasing trust in real-world evidence (RWE).

MATERIALS AND METHODS

The framework employs objective diagnostic measures to assess the appropriateness of study designs, analytic assumptions, and threats to validity in generating reliable evidence addressing causal questions. Diagnostic evaluations should be interpreted before the unblinding of study results or, alternatively, only unblind results from analyses that pass pre-specified thresholds. We provide a conceptual overview of objective diagnostic measures and demonstrate their impact on the validity of RWE from a large-scale comparative new-user study of various antihypertensive medications. We evaluated expected absolute systematic error (EASE) before and after applying diagnostic thresholds, using a large set of negative control outcomes.

RESULTS

Applying objective diagnostics reduces bias and improves evidence reliability in observational studies. Among 11 716 analyses (EASE = 0.38), 13.9% met pre-specified diagnostic thresholds which reduced EASE to zero. Objective diagnostics provide a comprehensive and empirical set of tests that increase confidence when passed and raise doubts when failed.

DISCUSSION

The increasing use of real-world data presents a scientific opportunity; however, the complexity of the evidence generation process poses challenges for understanding study validity and trusting RWE. Deploying objective diagnostics is crucial to reducing bias and improving reliability in RWE generation. Under ideal conditions, multiple study designs pass diagnostics and generate consistent results, deepening understanding of causal relationships. Open-source, standardized programs can facilitate implementation of diagnostic analyses.

CONCLUSION

Objective diagnostics are a valuable addition to the RWE generation process.

CohortDiagnostics: Phenotype evaluation across a network of observational data sources using population-level characterization

OBJECTIVE

This paper introduces a novel framework for evaluating phenotype algorithms (PAs) using the open-source tool, Cohort Diagnostics.

MATERIALS AND METHODS

The method is based on several diagnostic criteria to evaluate a patient cohort returned by a PA. Diagnostics include estimates of incidence rate, index date entry code breakdown, and prevalence of all observed clinical events prior to, on, and after index date. We test our framework by evaluating one PA for systemic lupus erythematosus (SLE) and two PAs for Alzheimer's disease (AD) across 10 different observational data sources.

RESULTS

By utilizing CohortDiagnostics, we found that the population-level characteristics of individuals in the cohort of SLE closely matched the disease's anticipated clinical profile. Specifically, the incidence rate of SLE was consistently higher in occurrence among females. Moreover, expected clinical events like laboratory tests, treatments, and repeated diagnoses were also observed. For AD, although one PA identified considerably fewer patients, absence of notable differences in clinical characteristics between the two cohorts suggested similar specificity.

DISCUSSION

We provide a practical and data-driven approach to evaluate PAs, using two clinical diseases as examples, across a network of OMOP data sources. Cohort Diagnostics can ensure the subjects identified by a specific PA align with those intended for inclusion in a research study.

CONCLUSION

Diagnostics based on large-scale population-level characterization can offer insights into the misclassification errors of PAs.

The necessity of validity diagnostics when drawing causal inferences from observational data: lessons from a multi-database evaluation of the risk of non-infectious uveitis among patients exposed to Remicade®

BACKGROUND

Autoimmune disorders have primary manifestations such as joint pain and bowel inflammation but can also have secondary manifestations such as non-infectious uveitis (NIU). A regulatory health authority raised concerns after receiving spontaneous reports for NIU following exposure to Remicade®, a biologic therapy with multiple indications for which alternative therapies are available. In assessment of this clinical question, we applied validity diagnostics to support observational data causal inferences.

METHODS

We assessed the risk of NIU among patients exposed to Remicade® compared to alternative biologics. Five databases, four study populations, and four analysis methodologies were used to estimate 80 potential treatment effects, with 20 pre-specified as primary. The study populations included inflammatory bowel conditions Crohn's disease or ulcerative colitis (IBD), ankylosing spondylitis (AS), psoriatic conditions plaque psoriasis or psoriatic arthritis (PsO/PsA), and rheumatoid arthritis (RA). We conducted four analysis strategies intended to address limitations of causal estimation using observational data and applied four diagnostics with pre-specified quantitative rules to evaluate threats to validity from observed and unobserved confounding. We also qualitatively assessed post-propensity score matching representativeness, and bias susceptibility from outcome misclassification. We fit Cox proportional-hazards models, conditioned on propensity score-matched sets, to estimate the on-treatment risk of NIU among Remicade® initiators versus alternatives. Estimates from analyses that passed four validity tests were assessed.

RESULTS

Of the 80 total analyses and the 20 analyses pre-specified as primary, 24% and 20% passed diagnostics, respectively. Among patients with IBD, we observed no evidence of increased risk for NIU relative to other similarly indicated biologics (pooled hazard ratio [HR] 0.75, 95% confidence interval [CI] 0.38-1.40). For patients with RA, we observed no increased risk relative to similarly indicated biologics, although results were imprecise (HR: 1.23, 95% CI 0.14-10.47).

CONCLUSIONS

We applied validity diagnostics on a heterogenous, observational setting to answer a specific research question. The results indicated that safety effect estimates from many analyses would be inappropriate to interpret as causal, given the data available and methods employed. Validity diagnostics should always be used to determine if the design and analysis are of sufficient quality to support causal inferences. The clinical implications of our findings on IBD suggests that, if an increased risk exists, it is unlikely to be greater than 40% given the 1.40 upper bound of the pooled HR confidence interval.

Semi-supervised ROC analysis for reliable and streamlined evaluation of phenotyping algorithms

OBJECTIVE

High-throughput phenotyping will accelerate the use of electronic health records (EHRs) for translational research. A critical roadblock is the extensive medical supervision required for phenotyping algorithm (PA) estimation and evaluation. To address this challenge, numerous weakly-supervised learning methods have been proposed. However, there is a paucity of methods for reliably evaluating the predictive performance of PAs when a very small proportion of the data is labeled. To fill this gap, we introduce a semi-supervised approach (ssROC) for estimation of the receiver operating characteristic (ROC) parameters of PAs (eg, sensitivity, specificity).

MATERIALS AND METHODS

ssROC uses a small labeled dataset to nonparametrically impute missing labels. The imputations are then used for ROC parameter estimation to yield more precise estimates of PA performance relative to classical supervised ROC analysis (supROC) using only labeled data. We evaluated ssROC with synthetic, semi-synthetic, and EHR data from Mass General Brigham (MGB).

RESULTS

ssROC produced ROC parameter estimates with minimal bias and significantly lower variance than supROC in the simulated and semi-synthetic data. For the 5 PAs from MGB, the estimates from ssROC are 30% to 60% less variable than supROC on average.

DISCUSSION

ssROC enables precise evaluation of PA performance without demanding large volumes of labeled data. ssROC is also easily implementable in open-source R software.

CONCLUSION

When used in conjunction with weakly-supervised PAs, ssROC facilitates the reliable and streamlined phenotyping necessary for EHR-based research.

Comparing broad and narrow phenotype algorithms: differences in performance characteristics and immortal time incurred

Introduction: When developing phenotype algorithms for observational research, there is usually a trade-off between definitions that are sensitive or specific. The objective of this study was to estimate the performance characteristics of phenotype algorithms designed for increasing specificity and to estimate the immortal time associated with each algorithm. Materials and methods: We examined algorithms for 11 chronic health conditions. The analyses were from data from five databases. For each health condition, we created five algorithms to examine performance (sensitivity and positive predictive value (PPV)) differences: one broad algorithm using a single code for the health condition and four narrow algorithms where a second diagnosis code was required 1-30 days, 1-90 days, 1-365 days, or 1- all days in a subject's continuous observation period after the first code. We also examined the proportion of immortal time relative to time-at-risk (TAR) for four outcomes. The TAR's were: 0-30 days after the first condition occurrence (the index date), 0-90 days post-index, 0-365 days post-index, and 0-1,095 days post-index. Performance of algorithms for chronic health conditions was estimated using PheValuator (V2.1.4) from the OHDSI toolstack. Immortal time was calculated as the time from the index date until the first of the following: 1) the outcome; 2) the end of the outcome TAR; 3) the occurrence of the second code for the chronic health condition. Results: In the first analysis, the narrow phenotype algorithms, i.e., those requiring a second condition code, produced higher estimates for PPV and lower estimates for sensitivity compared to the single code algorithm. In all conditions, increasing the time to the required second code increased the sensitivity of the algorithm. In the second analysis, the amount of immortal time increased as the window used to identify the second diagnosis code increased. The proportion of TAR that was immortal was highest in the 30 days TAR analyses compared to the 1,095 days TAR analyses. Conclusion: Attempting to increase the specificity of a health condition algorithm by adding a second code is a potentially valid approach to increase specificity, albeit at the cost of incurring immortal time.