The FDA's sentinel initiative-A comprehensive approach to medical product surveillance

Interplay of Spontaneous Reporting and Longitudinal Healthcare Databases for Signal Management: Position Statement from the Real-World Evidence and Big Data Special Interest Group of the International Society of Pharmacovigilance

Signal management, defined as the set of activities from signal detection to recommendations for action, is conducted using different data sources and leveraging data from spontaneous reporting databases (SRDs), which represent the cornerstone of pharmacovigilance. However, the exponentially increasing generation and availability of real-world data collected in longitudinal healthcare databases (LHDs), along with the rapid evolution of artificial intelligence-based algorithms and other advanced analytical methods, offers a wide range of opportunities to complement SRDs throughout all stages of signal management, especially signal detection. Integrating information derived from SRDs and LHDs may reduce their respective limitations, thus potentially enhancing post-marketing surveillance. The aim of this position statement is to critically evaluate the complementary role of SRDs and LHDs in signal management, exploring the potential benefits and challenges in integrating information coming from these two data sources. Furthermore, we presented successful cases of the interplay between SRDs and LHDs for signal management, along with future opportunities and directions to improve such interplay.

Patient characteristics and use for bevacizumab in ophthalmology and oncology in a distributed research network

BACKGROUND

Although bevacizumab and its biosimilars are commonly used, there are limited real-world data on bevacizumab use in the United States, especially biosimilar bevacizumab used in ophthalmologic conditions.

OBJECTIVE

To evaluate use patterns and patient characteristics for the originator bevacizumab relative to its biosimilars for labeled and off-label oncology and ophthalmology conditions and characterize adverse events in patients using bevacizumab for oncologic indications.

METHODS

We conducted a retrospective cohort study with the Biologics and Biosimilars Collective Intelligence Consortium-distributed database to identify patients aged 21 years and older who received bevacizumab between January 1, 2010, and June 30, 2021. Oncology indications included colon, lung, and gynecologic (cervical, uterine, and ovarian) cancers. Ophthalmologic indications included neovascular age-related macular degeneration (AMD), retinal vein occlusion (RVO), choroidal neovascularization (CNV), and proliferative diabetic retinopathy (PDR). We also captured patients' demographic and clinical characteristics.

RESULTS

Total bevacizumab product (originator and biosimilars) use increased over time for RVO, CNV, and PDR starting in 2015 but decreased for AMD after 2016. For ophthalmology, bevacizumab product users were primarily male (56.8%), had a mean age of 62.9 years (SD = 0.08), and had a mean Charlson/Elixhauser combined comorbidity score ranging from 0.7 (CNV) to 2.7 (PDR). Bevacizumab users for oncology indications were mostly female (61.8%), had a mean age of 62.9 years (SD = 12.2), and had a mean Charlson/Elixhauser combined comorbidity score of 7.4 (SD = 3.0). Oncologic biosimilar product use increased over time between 2019 and 2020 as follows: colon cancer, 6.2% to 49.4%; lung cancer, 1.9% to 36.2%; and gynecologic cancer, 2.4% to 38.1%.

CONCLUSIONS

Bevacizumab product use increased across most indications during the study period. Use for biosimilars increased in later years relative to the originator once available on the market. Limited data are available on real-world biosimilar use in the United States; future research should include monitoring for use and adverse events of these products.

A simple Cox approach to estimating risk ratios without sharing individual-level data in multisite studies

Epidemiologic studies frequently use risk ratios to quantify associations between exposures and binary outcomes. When the data are physically stored at the sites of multiple data partners, it can be challenging to perform individual-level analysis if data cannot be pooled centrally due to privacy constraints. Existing methods either require multiple file transfers between each data partner and an analysis center (eg, distributed regression) or only provide approximate estimation of the risk ratio (eg, meta-analysis). Here we develop a practical method that requires a single transfer of 8 summary-level quantities from each data partner. Our approach leverages an existing risk-set method and software originally developed for Cox regression. Sharing only summary-level information, the proposed method provides risk ratio estimates and 95% CIs identical to those that would be provided-if individual-level data were pooled-by the modified Poisson regression. We justify the method theoretically, confirm its performance using simulated data, and implement it in a distributed analysis of COVID-19 data from the US Food and Drug Administration's Sentinel System. This article is part of a Special Collection on Pharmacoepidemiology.

Project Victoria: A pragmatic data model to automate RWE generation from the national French claims database

Objective: This paper describes Victoria, an empirically built data pipeline for SNDS to: - Build an automated, scalable pipeline supporting changes to the data model inherent to the use of large databases, - Deliver a documented pipeline with clear processes, enabling scientific, epidemiological researches, - Ease access to SNDS data in compliance with regulatory requirements. Methods: This paper describes the 2-steps process of the Victoria pipeline and its final output. The initial cleaning step consists in formatting, deleting empty, error or duplicate records and renaming variables without changing their values, accordingly with the official SNDS documentation. The second step consists in creating 2 linearised data models: every line of each table is an event, and each table is indexed with a unique patient identifier, without the need for a central patient or identifier table. These 2 models are: - the epidemiological model, used for answering most of the research questions requiring population phenotyping (demography, diagnosis, procedures characteristics). - the medico-economic model is used for costs and healthcare consumption analyses. It contains more complex information about reimbursements rates and the data quality assessment is focused on costs rather than medico-administrative information. Results: The pipeline was executed on 2 different datasets representing ∼85 000 and ∼870 000 beneficiaries with the following configuration: one master with 4 cores and 16Go of RAM and respectively 4 and 6 workers. The total execution time for the smaller dataset was 25 h and 96 h for the larger one. The longest part of those times is represented by the format conversion to parquet. The cleaning step took only 4 h in both cases. The epidemiological model took 344 min for the smaller dataset and 1934 min for the larger one. The medico-economic model took the longest time with 704 min and 2145 min, respectively. Conclusion: Victoria pipeline is a successfully implemented SNDS pipeline. Compared to previous pipelines, reviewability is part of its design as unit tests and quality assessments can natively be developed to ensure data and analysis quality. The pipeline has been used for 2 published studies. The recent work toward OMOP conversion will be integrated in upcoming versions and, as Victoria is set to run on a CD platform, the potential evolution if SNDS format can be considered.

INSIGHT: A Tool for Fit-for-Purpose Evaluation and Quality Assessment of Standardized Observational Data Sources for Real World Evidence on Medicine and Vaccine Safety

PURPOSE

To describe the development of INSIGHT, a real-world data quality tool to assess completeness, consistency, and fitness-for-purpose of observational health data sources.

METHODS

We designed a three-level pipeline with data quality assessments (DQAs) to be performed in ConcePTION Common Data Model (CDM) instances. The pipeline has been coded using R.

RESULTS

INSIGHT is an open-source tool that identifies potential data quality issues in CDM-standardized instances through the systematic execution and summary of over 588 configurable DQAs. Level 1 focuses on conformance to the ConcePTION CDM specifications. Level 2 evaluates the temporal plausibility of events and uniqueness of records. Level 3 provides an overview of distributions, outliers, and trends over time to facilitate fit-for-purpose evaluation. Therefore, level 1 and 2 assure a proper data standardization, while level 3 provides information regarding the study population, and potential sub-populations. The DQAs are run locally and assessed centrally by a data quality revisor together with the data access provider's representatives.

DISCUSSION

Data quality is the sum of several internal and external features of the data. While DQAs can provide reassurance about fitness-for-purpose for secondary-use data sources, improvements in data collection are essential to reduce errors and enhance overall data quality for Real World Evidence.

CONCLUSION

INSIGHT aims to support clinical and regulatory decision-making for medicines and vaccines by evaluating the quality of observational health data sources to support fit for purpose assessment. Assessing and improving data quality will enhance the reliability and quality of the generated evidence.

STUDY REGISTRATION

This research was registered in EU PAS registration with number EU50142.

Epidemiological investigation of adult emergency infusion adverse drug reactions (EIADR) in Nanjing, China: a prospective cross-sectional study (EIADR II)

OBJECTIVE

To analyze the clinical characteristics of adverse drug reactions (ADRs) in adults receiving emergency infusions at a tertiary hospital.

METHODS

We conducted a prospective observational cohort study involving 585 adult patients who experienced adverse drug reactions (ADRs) between 20 November 2019, and 20 November 2023, during intravenous infusions in the emergency infusion room of a tertiary hospital. The analysis included patients' gender, age, type of drugs involved, organ-system involvement, clinical manifestations of ADRs, severity grading of ADRs, and preventability of ADRs.

RESULTS

The highest percentage of ADRs occurred in the 30-39 age group. Antimicrobials were the most common cause of ADRs, with skin manifestations being the predominant clinical feature. Approximately 23.93% of ADRs were deemed preventable.

CONCLUSION

Monitoring ADRs related to antimicrobials is crucial in adult emergency infusions. The 30-39 age group is particularly susceptible to ADRs. Preventive measures and a well-established Electronic Health Record (EHR) system can effectively reduce ADRs incidence.

Scalable incident detection via natural language processing and probabilistic language models

Post marketing safety surveillance depends in part on the ability to detect concerning clinical events at scale. Spontaneous reporting might be an effective component of safety surveillance, but it requires awareness and understanding among healthcare professionals to achieve its potential. Reliance on readily available structured data such as diagnostic codes risks under-coding and imprecision. Clinical textual data might bridge these gaps, and natural language processing (NLP) has been shown to aid in scalable phenotyping across healthcare records in multiple clinical domains. In this study, we developed and validated a novel incident phenotyping approach using unstructured clinical textual data agnostic to Electronic Health Record (EHR) and note type. It's based on a published, validated approach (PheRe) used to ascertain social determinants of health and suicidality across entire healthcare records. To demonstrate generalizability, we validated this approach on two separate phenotypes that share common challenges with respect to accurate ascertainment: (1) suicide attempt; (2) sleep-related behaviors. With samples of 89,428 records and 35,863 records for suicide attempt and sleep-related behaviors, respectively, we conducted silver standard (diagnostic coding) and gold standard (manual chart review) validation. We showed Area Under the Precision-Recall Curve of ~ 0.77 (95% CI 0.75-0.78) for suicide attempt and AUPR ~ 0.31 (95% CI 0.28-0.34) for sleep-related behaviors. We also evaluated performance by coded race and demonstrated differences in performance by race differed across phenotypes. Scalable phenotyping models, like most healthcare AI, require algorithmovigilance and debiasing prior to implementation.

Utilization and patient characteristics for the trastuzumab reference and biosimilars, and other human epidermal growth factor receptor 2 inhibitors in the United States

BACKGROUND

Trastuzumab is an antihuman epidermal growth factor receptor 2 monoclonal antibody used to treat breast and other cancers. Trastuzumab biosimilars were approved in the United States beginning in 2017. Utilization information on these biosimilars is limited.

OBJECTIVE

To examine utilization patterns and characteristics of patients treated with trastuzumab (biosimilars and reference) and other human epidermal growth factor receptor 2 products.

METHODS

We evaluated health care claims data from the Biologics and Biosimilars Collective Intelligence Consortium distributed research network, representing a large, geographically diverse US population of commercially insured individuals. We queried 4 distributed research network health plan partners to capture product usage data and patient information from October 1, 2016, to October 31, 2022. Patients were required to be continuously enrolled in their health plan for at least 365 days before their first observed trastuzumab utilization date in this study period. Data were aggregated across data partners.

RESULTS

More than 16 million eligible health plan members representing more than 31 million person-years of data were evaluated. We identified 5,984 incident treatment episodes; 3,878 (64.8%) episodes were with the reference trastuzumab. The mean ages were consistent across trastuzumab products (60.2 to 65.1 years) and at least 80% of the episodes were among female patients. The mean comorbidity index score was 1.2 (SD = 1.9) among users of the reference vs the biosimilars (range 1.2-2.5). Other clinical characteristics (eg, diabetes, hypertension) were comparable across products. The proportion of total incident episodes of the reference trastuzumab decreased substantially over time (96% in 2016 vs 28% in 2021) as utilization of the biosimilars increased (eg, use of trastuzumab-anns increased from 2% [2019] to 36% [2021]). Similar utilization trends were seen among patients with and without metastatic breast cancer.

CONCLUSIONS

Trastuzumab biosimilars utilization has grown since their introduction to the US market. Exploration of these biosimilars' comparative effectiveness and safety to their reference product is warranted.

Use of the Sentinel System to Examine Medical Product Use and Outcomes During Pregnancy

While many pregnant individuals use prescription medications, evidence supporting product safety during pregnancy is often inadequate. Existing electronic healthcare data sources provide large, diverse samples of health plan members to allow for the study of medical product utilization during pregnancy, as well as pregnancy, maternal, and infant outcomes. The Sentinel System is a national medical product surveillance system that includes administrative claims and electronic health record databases from large national and regional health insurers. In addition to these data sources, Sentinel develops and maintains a sizeable selection of analytic tools to facilitate epidemiologic analyses in a way that protects patient privacy and health system autonomy. In this article, we provide an overview of Sentinel System infrastructure, including the Mother-Infant Linkage Table, parameterizable analytic tools, and algorithms to estimate gestational age and identify pregnancy outcomes. We also describe past and future Sentinel work that contributes to our understanding of the way medical products are used and the safety of these products during pregnancy.

The FDA Sentinel Real World Evidence Data Enterprise (RWE-DE)

PURPOSE

The US Food and Drug Administration's Sentinel Innovation Center aimed to establish a query-ready, quality-checked distributed data network containing electronic health records (EHRs) linked with insurance claims data for at least 10 million individuals to expand the utility of real-world data for regulatory decision-making.

METHODS

In this report, we describe the resulting network, the Real-World Evidence Data Enterprise (RWE-DE), including data from two commercial EHR-claims linked assets collectively termed the Commercial Network covering 21 million lives, and four academic partner institutions collectively termed the Development Network covering 4.5 million lives.

RESULTS

We discuss provenance and completeness of the data converted in the Sentinel Common Data Model (SCDM), describe patient populations, and report on EHR-claims linkage characterization for all contributing data sources. Further, we introduce a standardized process to store free-text notes in the Development Network for efficient retrieval as needed.

CONCLUSIONS

Finally, we outline typical use cases for the RWE-DE where it can broaden the reach of the types of questions that can be addressed by the Sentinel system.

Use of Hydrochlorothiazide in the United States Following Label Update About Skin Cancer Risk

PURPOSE

On August 20, 2020, the United States (U.S.) Food and Drug Administration (FDA) issued a Drug Safety Communication (DSC) along with labeling updates to inform the public about a small increased risk of non-melanoma skin cancer (NMSC) associated with hydrochlorothiazide (HCTZ) use. This study aims to assess whether the DSC impacted HCTZ use in the U.S.

METHODS

We conducted a trend analysis in the Sentinel Distributed Database using national healthcare administrative data from January 2017 to November 2022. We identified two cohorts each month: An overall cohort of all enrollees and a skin cancer cohort of those with a history of NMSC. For each cohort, we plotted the monthly proportion of patients receiving HCTZ-containing products among those receiving any thiazide diuretics. We performed interrupted time series analyses to quantify the impact of the DSC on these monthly proportions. Secondary analyses were conducted on the proportion of HCTZ users among patients receiving any antihypertensives.

RESULTS

In the overall cohort, the DSC was only associated with a statistically significant but clinically negligible trend change of monthly HCTZ proportion within this cohort (0.018%; 95% CI, 0.012%-0.025%). Similar results were observed in the skin cancer cohort. The secondary analysis found no significant level change or trend change in the monthly proportion of HCTZ use among antihypertensive users.

CONCLUSIONS

We did not observe significant changes in HCTZ use following the DSC about its NMSC risk, among the overall population and those with a history of NMSC. Our findings were in accordance with the DSC recommendation.

A general framework for developing computable clinical phenotype algorithms

OBJECTIVE

To present a general framework providing high-level guidance to developers of computable algorithms for identifying patients with specific clinical conditions (phenotypes) through a variety of approaches, including but not limited to machine learning and natural language processing methods to incorporate rich electronic health record data.

MATERIALS AND METHODS

Drawing on extensive prior phenotyping experiences and insights derived from 3 algorithm development projects conducted specifically for this purpose, our team with expertise in clinical medicine, statistics, informatics, pharmacoepidemiology, and healthcare data science methods conceptualized stages of development and corresponding sets of principles, strategies, and practical guidelines for improving the algorithm development process.

RESULTS

We propose 5 stages of algorithm development and corresponding principles, strategies, and guidelines: (1) assessing fitness-for-purpose, (2) creating gold standard data, (3) feature engineering, (4) model development, and (5) model evaluation.

DISCUSSION AND CONCLUSION

This framework is intended to provide practical guidance and serve as a basis for future elaboration and extension.

Assessing Real-World Data From Electronic Health Records for Health Technology Assessment: The SUITABILITY Checklist: A Good Practices Report of an ISPOR Task Force

This ISPOR Good Practices report provides a framework for assessing the suitability of electronic health records data for use in health technology assessments (HTAs). Although electronic health record (EHR) data can fill evidence gaps and improve decisions, several important limitations can affect its validity and relevance. The ISPOR framework includes 2 components: data delineation and data fitness for purpose. Data delineation provides a complete understanding of the data and an assessment of its trustworthiness by describing (1) data characteristics; (2) data provenance; and (3) data governance. Fitness for purpose comprises (1) data reliability items, ie, how accurate and complete the estimates are for answering the question at hand and (2) data relevance items, which assess how well the data are suited to answer the particular question from a decision-making perspective. The report includes a checklist specific to EHR data reporting: the ISPOR SUITABILITY Checklist. It also provides recommendations for HTA agencies and policy makers to improve the use of EHR-derived data over time. The report concludes with a discussion of limitations and future directions in the field, including the potential impact from the substantial and rapid advances in the diffusion and capabilities of large language models and generative artificial intelligence. The report's immediate audiences are HTA evidence developers and users. We anticipate that it will also be useful to other stakeholders, particularly regulators and manufacturers, in the future.

Transparency, reproducibility, and replicability of pharmacoepidemiology studies in a distributed network environment

PURPOSE

Our objective is to describe how the U.S. Food and Drug Administration (FDA)'s Sentinel System implements best practices to ensure trust in drug safety studies using real-world data from disparate sources.

METHODS

We present a stepwise schematic for Sentinel's data harmonization, data quality check, query design and implementation, and reporting practices, and describe approaches to enhancing the transparency, reproducibility, and replicability of studies at each step.

CONCLUSIONS

Each Sentinel data partner converts its source data into the Sentinel Common Data Model. The transformed data undergoes rigorous quality checks before it can be used for Sentinel queries. The Sentinel Common Data Model framework, data transformation codes for several data sources, and data quality assurance packages are publicly available. Designed to run against the Sentinel Common Data Model, Sentinel's querying system comprises a suite of pre-tested, parametrizable computer programs that allow users to perform sophisticated descriptive and inferential analysis without having to exchange individual-level data across sites. Detailed documentation of capabilities of the programs as well as the codes and information required to execute them are publicly available on the Sentinel website. Sentinel also provides public trainings and online resources to facilitate use of its data model and querying system. Its study specifications conform to established reporting frameworks aimed at facilitating reproducibility and replicability of real-world data studies. Reports from Sentinel queries and associated design and analytic specifications are available for download on the Sentinel website. Sentinel is an example of how real-world data can be used to generate regulatory-grade evidence at scale using a transparent, reproducible, and replicable process.

Neurocircuitry underlying the actions of glucagon-like peptide 1 and peptide YY3-36 in the suppression of food, drug-seeking, and anxiogenesis

Obesity is a critical health condition worldwide that increases the risks of comorbid chronic diseases, but it can be managed with weight loss. However, conventional interventions relying on diet and exercise are inadequate for achieving and maintaining weight loss, thus there is significant market interest for pharmaceutical anti-obesity agents. For decades, receptor agonists for the gut peptide glucagon-like peptide 1 (GLP-1) featured prominently in anti-obesity medications by suppressing appetite and food reward to elicit rapid weight loss. As the neurocircuitry underlying food motivation overlaps with that for drugs of abuse, GLP-1 receptor agonism has also been shown to decrease substance use and relapse, thus its therapeutic potential may extend beyond weight management to treat addictions. However, as prolonged use of anti-obesity drugs may increase the risk of mood-related disorders like anxiety and depression, and individuals taking GLP-1-based medication commonly report feeling demotivated, the long-term safety of such drugs is an ongoing concern. Interestingly, current research now focuses on dual agonist approaches that include GLP-1 receptor agonism to enable synergistic effects on weight loss or associated functions. GLP-1 is secreted from the same intestinal cells as the anorectic gut peptide, Peptide YY3-36 (PYY3-36), thus this review assessed the therapeutic potential and underlying neural circuits targeted by PYY3-36 when administered independently or in combination with GLP-1 to curb the appetite for food or drugs of abuse like opiates, alcohol, and nicotine. Additionally, we also reviewed animal and human studies to assess the impact, if any, for GLP-1 and/or PYY3-36 on mood-related behaviors in relation to anxiety and depression. As dual agonists targeting GLP-1 and PYY3-36 may produce synergistic effects, they can be effective at lower doses and offer an alternative approach for therapeutic benefits while mitigating undesirable side effects.

Generating and using real-world data: A worthwhile uphill battle

The precision oncology paradigm challenges the feasibility and data generalizability of traditional clinical trials. Consequently, an unmet need exists for practical approaches to test many subgroups, evaluate real-world drug value, and gather comprehensive, accessible datasets to validate novel biomarkers. Real-world data (RWD) are increasingly recognized to have the potential to fill this gap in research methodology. Established applications of RWD include informing disease epidemiology, pharmacovigilance, and healthcare quality assessment. Currently, concerns regarding RWD quality and comprehensiveness, privacy, and biases hamper their broader application. Nonetheless, RWD may play a pivotal role in supplementing clinical trials, enabling conditional reimbursement and accelerated drug access, and innovating trial conduct. Moreover, purpose-built RWD repositories may support the extension or refinement of drug indications and facilitate the discovery and validation of new biomarkers. This perspective explores the potential of leveraging RWD to advance oncology, highlights its benefits and challenges, and suggests a path forward in this evolving field.

Staging and clean room: Constructs designed to facilitate transparency and reduce bias in comparative analyses of real-world data

PURPOSE

We describe constructs designed to protect the integrity of the results from comparative analyses using real-world data (RWD): staging and clean room.

METHODS

Staging involves performing sequential preliminary analyses and evaluating the population size available and potential bias before conducting comparative analyses. A clean room involves restricted access to data and preliminary results, policies governing exploratory analyses and protocol deviations, and audit trail. These constructs are intended to allow decisions about protocol deviations, such as changes to design or model specification, to be made without knowledge of how they might affect subsequent analyses. We describe an example for implementing staging with a clean room.

RESULTS

Stage 1 may involve selecting a data source, developing and registering a protocol, establishing a clean room, and applying inclusion/exclusion criteria. Stage 2 may involve attempting to achieve covariate balance, often through propensity score models. Stage 3 may involve evaluating the presence of residual confounding using negative control outcomes. After each stage, check points may be implemented when a team of statisticians, epidemiologists and clinicians masked to how their decisions may affect study outcomes, reviews the results. This review team may be tasked with making recommendations for protocol deviations to address study precision or bias. They may recommend proceeding to the next stage, conducting additional analyses to address bias, or terminating the study. Stage 4 may involve conducting the comparative analyses.

CONCLUSIONS

The staging and clean room constructs are intended to protect the integrity and enhance confidence in the results of analyses of RWD.

Data-driven automated classification algorithms for acute health conditions: applying PheNorm to COVID-19 disease

OBJECTIVES

Automated phenotyping algorithms can reduce development time and operator dependence compared to manually developed algorithms. One such approach, PheNorm, has performed well for identifying chronic health conditions, but its performance for acute conditions is largely unknown. Herein, we implement and evaluate PheNorm applied to symptomatic COVID-19 disease to investigate its potential feasibility for rapid phenotyping of acute health conditions.

MATERIALS AND METHODS

PheNorm is a general-purpose automated approach to creating computable phenotype algorithms based on natural language processing, machine learning, and (low cost) silver-standard training labels. We applied PheNorm to cohorts of potential COVID-19 patients from 2 institutions and used gold-standard manual chart review data to investigate the impact on performance of alternative feature engineering options and implementing externally trained models without local retraining.

RESULTS

Models at each institution achieved AUC, sensitivity, and positive predictive value of 0.853, 0.879, 0.851 and 0.804, 0.976, and 0.885, respectively, at quantiles of model-predicted risk that maximize F1. We report performance metrics for all combinations of silver labels, feature engineering options, and models trained internally versus externally.

DISCUSSION

Phenotyping algorithms developed using PheNorm performed well at both institutions. Performance varied with different silver-standard labels and feature engineering options. Models developed locally at one site also worked well when implemented externally at the other site.

CONCLUSION

PheNorm models successfully identified an acute health condition, symptomatic COVID-19. The simplicity of the PheNorm approach allows it to be applied at multiple study sites with substantially reduced overhead compared to traditional approaches.

Design differences and variation in results between randomised trials and non-randomised emulations: meta-analysis of RCT-DUPLICATE data

Objective

To explore how design emulation and population differences relate to variation in results between randomised controlled trials (RCT) and non-randomised real world evidence (RWE) studies, based on the RCT-DUPLICATE initiative (Randomised, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology).

Design

Meta-analysis of RCT-DUPLICATE data.

Data sources

Trials included in RCT-DUPLICATE, a demonstration project that emulated 32 randomised controlled trials using three real world data sources: Optum Clinformatics Data Mart, 2004-19; IBM MarketScan, 2003-17; and subsets of Medicare parts A, B, and D, 2009-17.

Eligibility criteria for selecting studies

Trials where the primary analysis resulted in a hazard ratio; 29 RCT-RWE study pairs from RCT-DUPLICATE.

Results

Differences and variation in effect sizes between the results from randomised controlled trials and real world evidence studies were investigated. Most of the heterogeneity in effect estimates between the RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: treatment started in hospital (which does not appear in health insurance claims data), discontinuation of some baseline treatments at randomisation (which would have been an unusual care decision in clinical practice), and delayed onset of drug effects (which would be under-reported in real world clinical practice because of the relatively short persistence of the treatment). Adding the three emulation differences to the meta-regression reduced heterogeneity from 1.9 to almost 1 (absence of heterogeneity).

Conclusions

This analysis suggests that a substantial proportion of the observed variation between results from randomised controlled trials and real world evidence studies can be attributed to differences in design emulation.

Scalable Incident Detection via Natural Language Processing and Probabilistic Language Models

Post marketing safety surveillance depends in part on the ability to detect concerning clinical events at scale. Spontaneous reporting might be an effective component of safety surveillance, but it requires awareness and understanding among healthcare professionals to achieve its potential. Reliance on readily available structured data such as diagnostic codes risk under-coding and imprecision. Clinical textual data might bridge these gaps, and natural language processing (NLP) has been shown to aid in scalable phenotyping across healthcare records in multiple clinical domains. In this study, we developed and validated a novel incident phenotyping approach using unstructured clinical textual data agnostic to Electronic Health Record (EHR) and note type. It's based on a published, validated approach (PheRe) used to ascertain social determinants of health and suicidality across entire healthcare records. To demonstrate generalizability, we validated this approach on two separate phenotypes that share common challenges with respect to accurate ascertainment: 1) suicide attempt; 2) sleep-related behaviors. With samples of 89,428 records and 35,863 records for suicide attempt and sleep-related behaviors, respectively, we conducted silver standard (diagnostic coding) and gold standard (manual chart review) validation. We showed Area Under the Precision-Recall Curve of ∼ 0.77 (95% CI 0.75-0.78) for suicide attempt and AUPR ∼ 0.31 (95% CI 0.28-0.34) for sleep-related behaviors. We also evaluated performance by coded race and demonstrated differences in performance by race were dissimilar across phenotypes and require algorithmovigilance and debiasing prior to implementation.

Real-world data analysis on effectiveness of integrative therapies: A practical guide to study design and data analysis using healthcare databases

Real world data (RWD) is increasingly used to investigate health outcomes and treatment efficacy in the field of integrative medicine. Due to the fact that the majority of RWDs are not intended for research, their secondary use in research necessitates complex study designs to account for bias and confounding. To conduct a robust analysis of RWD in integrative medicine, a comprehensive study design process that reflects the characteristics of integrative therapies is necessary. In this paper, we present a guide for designing comparative effectiveness RWE research in integrative medicine. We discuss key factors to consider when selecting RWDs for research on integrative medicine. We provide practical steps for developing a research question, formulating the PICOT objectives (population, intervention, comparator, outcome, and time horizon), and selecting and defining covariates with a summary table. Specific study designs are depicted with corresponding diagrams. Finally, data analysis procedures are introduced. We hope this article clarifies the importance of RWE research design and related processes in order to improve the rigor of RWD studies in the field of integrative medicine research.

Adult Moyamoya Disease and Syndrome: Current Perspectives and Future Directions: A Scientific Statement From the American Heart Association/American Stroke Association

Adult moyamoya disease and syndrome are rare disorders with significant morbidity and mortality. A writing group of experts was selected to conduct a literature search, summarize the current knowledge on the topic, and provide a road map for future investigation. The document presents an update in the definitions of moyamoya disease and syndrome, modern methods for diagnosis, and updated information on pathophysiology, epidemiology, and both medical and surgical treatment. Despite recent advancements, there are still many unresolved questions about moyamoya disease and syndrome, including lack of unified diagnostic criteria, reliable biomarkers, better understanding of the underlying pathophysiology, and stronger evidence for treatment guidelines. To advance progress in this area, it is crucial to acknowledge the limitations and weaknesses of current studies and explore new approaches, which are outlined in this scientific statement for future research strategies.

Six Years of the US Food and Drug Administration's Postmarket Active Risk Identification and Analysis System in the Sentinel Initiative: Implications for Real World Evidence Generation

Congress mandated the creation of a postmarket Active Risk Identification and Analysis (ARIA) system containing data on 100 million individuals for monitoring risks associated with drug and biologic products using data from disparate sources to complement the US Food and Drug Administration's (FDA's) existing postmarket capabilities. We report on the first 6 years of ARIA utilization in the Sentinel System (2016-2021). The FDA has used the ARIA system to evaluate 133 safety concerns; 54 of these evaluations have closed with regulatory determinations, whereas the rest remain in progress. If the ARIA system and the FDA's Adverse Event Reporting System are deemed insufficient to address a safety concern, then the FDA may issue a postmarket requirement to a product's manufacturer. One hundred ninety-seven ARIA insufficiency determinations have been made. The most common situation for which ARIA was found to be insufficient is the evaluation of adverse pregnancy and fetal outcomes following in utero drug exposure, followed by neoplasms and death. ARIA was most likely to be sufficient for thromboembolic events, which have high positive predictive value in claims data alone and do not require supplemental clinical data. The lessons learned from this experience illustrate the continued challenges using administrative claims data, especially to define novel clinical outcomes. This analysis can help to identify where more granular clinical data are needed to fill gaps to improve the use of real-world data for drug safety analyses and provide insights into what is needed to efficiently generate high-quality real-world evidence for efficacy.

Cardiovascular Care Innovation through Data-Driven Discoveries in the Electronic Health Record

The electronic health record (EHR) represents a rich source of patient information, increasingly being leveraged for cardiovascular research. Although its primary use remains the seamless delivery of health care, the various longitudinally aggregated structured and unstructured data elements for each patient within the EHR can define the computational phenotypes of disease and care signatures and their association with outcomes. Although structured data elements, such as demographic characteristics, laboratory measurements, problem lists, and medications, are easily extracted, unstructured data are underused. The latter include free text in clinical narratives, documentation of procedures, and reports of imaging and pathology. Rapid scaling up of data storage and rapid innovation in natural language processing and computer vision can power insights from unstructured data streams. However, despite an array of opportunities for research using the EHR, specific expertise is necessary to adequately address confidentiality, accuracy, completeness, and heterogeneity challenges in EHR-based research. These often require methodological innovation and best practices to design and conduct successful research studies. Our review discusses these challenges and their proposed solutions. In addition, we highlight the ongoing innovations in federated learning in the EHR through a greater focus on common data models and discuss ongoing work that defines such an approach to large-scale, multicenter, federated studies. Such parallel improvements in technology and research methods enable innovative care and optimization of patient outcomes.

Combining meta-analysis with multiple imputation for one-step, privacy-protecting estimation of causal treatment effects in multi-site studies

Missing data complicates statistical analyses in multi-site studies, especially when it is not feasible to centrally pool individual-level data across sites. We combined meta-analysis with within-site multiple imputation for one-step estimation of the average causal effect (ACE) of a target population comprised of all individuals from all data-contributing sites within a multi-site distributed data network, without the need for sharing individual-level data to handle missing data. We considered two orders of combination and three choices of weights for meta-analysis, resulting in six approaches. The first three approaches, denoted as RR + metaF, RR + metaR and RR + std, first combined results from imputed data sets within each site using Rubin's rules and then meta-analyzed the combined results across sites using fixed-effect, random-effects and sample-standardization weights, respectively. The last three approaches, denoted as metaF + RR, metaR + RR and std + RR, first meta-analyzed results across sites separately for each imputation and then combined the meta-analysis results using Rubin's rules. Simulation results confirmed very good performance of RR + std and std + RR under various missing completely at random and missing at random settings. A direct application of the inverse-variance weighted meta-analysis based on site-specific ACEs can lead to biased results for the targeted network-wide ACE in the presence of treatment effect heterogeneity by site, demonstrating the need to clearly specify the target population and estimand and properly account for potential site heterogeneity in meta-analyses seeking to draw causal interpretations. An illustration using a large administrative claims database is presented.

DESIGN DIFFERENCES EXPLAIN VARIATION IN RESULTS BETWEEN RANDOMIZED TRIALS AND THEIR NON-RANDOMIZED EMULATIONS

Objectives

While randomized controlled trials (RCTs) are considered a standard for evidence on the efficacy of medical treatments, non-randomized real-world evidence (RWE) studies using data from health insurance claims or electronic health records can provide important complementary evidence. The use of RWE to inform decision-making has been questioned because of concerns regarding confounding in non-randomized studies and the use of secondary data. RCT-DUPLICATE was a demonstration project that emulated the design of 32 RCTs with non-randomized RWE studies. We sought to explore how emulation differences relate to variation in results between the RCT-RWE study pairs.

Methods

We include all RCT-RWE study pairs from RCT-DUPLICATE where the measure of effect was a hazard ratio and use exploratory meta-regression methods to explain differences and variation in the effect sizes between the results from the RCT and the RWE study. The considered explanatory variables are related to design and population differences.

Results

Most of the observed variation in effect estimates between RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: (i) in-hospital start of treatment (not observed in claims data), (ii) discontinuation of certain baseline therapies at randomization (not part of clinical practice), (iii) delayed onset of drug effects (missed by short medication persistence in clinical practice).

Conclusions

This analysis suggests that a substantial proportion of the observed variation between results from RCTs and RWE studies can be attributed to design emulation differences. (238 words).

Treatment and care received by children hospitalized with COVID-19 in a large hospital network in the United States, February 2020 to September 2021

We described care received by hospitalized children with COVID-19 or multi-system inflammatory syndrome (MIS-C) prior to the 2021 COVID-19 Omicron variant surge in the US. We identified hospitalized children <18 years of age with a COVID-19 or MIS-C diagnosis (COVID-19 not required), separately, from February 2020-September 2021 (n = 126 hospitals). We described high-risk conditions, inpatient treatments, and complications among these groups. Among 383,083 pediatric hospitalizations, 2,186 had COVID-19 and 395 had MIS-C diagnosis. Less than 1% had both COVID-19 and MIS-C diagnosis (n = 154). Over half were >6 years old (54% COVID-19, 70% MIS-C). High-risk conditions included asthma (14% COVID-19, 11% MIS-C), and obesity (9% COVID-19, 10% MIS-C). Pulmonary complications in children with COVID-19 included viral pneumonia (24%) and acute respiratory failure (11%). In reference to children with COVID-19, those with MIS-C had more hematological disorders (62% vs 34%), sepsis (16% vs 6%), pericarditis (13% vs 2%), myocarditis (8% vs 1%). Few were ventilated or died, but some required oxygen support (38% COVID-19, 45% MIS-C) or intensive care (42% COVID-19, 69% MIS-C). Treatments included: methylprednisolone (34% COVID-19, 75% MIS-C), dexamethasone (25% COVID-19, 15% MIS-C), remdesivir (13% COVID-19, 5% MIS-C). Antibiotics (50% COVID-19, 68% MIS-C) and low-molecular weight heparin (17% COVID-19, 34% MIS-C) were frequently administered. Markers of illness severity among hospitalized children with COVID-19 prior to the 2021 Omicron surge are consistent with previous studies. We report important trends on treatments in hospitalized children with COVID-19 to improve the understanding of real-world treatment patterns in this population.

Use of the Electronic Health Record for Monitoring Adverse Drug Reactions

PURPOSE OF REVIEW

Adverse drug reactions (ADRs) are a significant cause of morbidity and mortality. The electronic health record (EHR) provides an opportunity to monitor ADRs, mainly through the utilization of drug allergy data and pharmacogenomics. This review article explores the current use of the EHR for ADR monitoring and highlights areas that require improvement.

RECENT FINDINGS

Recent research has identified several issues with using EHR for ADR monitoring. These include the lack of standardization between EHR systems, specificity in data entry options, incomplete and inaccurate documentation, and alert fatigue. These issues can limit the effectiveness of ADR monitoring and compromise patient safety. The EHR has great potential for monitoring ADR but needs significant updates to improve patient safety and optimize care. Future research should concentrate on developing standardized documentation and clinical decision support systems within EHRs. Healthcare professionals should also be educated on the significance of accurate and complete ADR monitoring.

Development and Verification of Time-Series Deep Learning for Drug-Induced Liver Injury Detection in Patients Taking Angiotensin II Receptor Blockers: A Multicenter Distributed Research Network Approach

OBJECTIVES

The objective of this study was to develop and validate a multicenter-based, multi-model, time-series deep learning model for predicting drug-induced liver injury (DILI) in patients taking angiotensin receptor blockers (ARBs). The study leveraged a national-level multicenter approach, utilizing electronic health records (EHRs) from six hospitals in Korea.

METHODS

A retrospective cohort analysis was conducted using EHRs from six hospitals in Korea, comprising a total of 10,852 patients whose data were converted to the Common Data Model. The study assessed the incidence rate of DILI among patients taking ARBs and compared it to a control group. Temporal patterns of important variables were analyzed using an interpretable timeseries model.

RESULTS

The overall incidence rate of DILI among patients taking ARBs was found to be 1.09%. The incidence rates varied for each specific ARB drug and institution, with valsartan having the highest rate (1.24%) and olmesartan having the lowest rate (0.83%). The DILI prediction models showed varying performance, measured by the average area under the receiver operating characteristic curve, with telmisartan (0.93), losartan (0.92), and irbesartan (0.90) exhibiting higher classification performance. The aggregated attention scores from the models highlighted the importance of variables such as hematocrit, albumin, prothrombin time, and lymphocytes in predicting DILI.

CONCLUSIONS

Implementing a multicenter-based timeseries classification model provided evidence that could be valuable to clinicians regarding temporal patterns associated with DILI in ARB users. This information supports informed decisions regarding appropriate drug use and treatment strategies.

Rewiring Drug Research and Development through Human Data-Driven Discovery (HD3)

In an era of unparalleled technical advancement, the pharmaceutical industry is struggling to transform data into increased research and development efficiency, and, as a corollary, new drugs for patients. Here, we briefly review some of the commonly discussed issues around this counterintuitive innovation crisis. Looking at both industry- and science-related factors, we posit that traditional preclinical research is front-loading the development pipeline with data and drug candidates that are unlikely to succeed in patients. Applying a first principles analysis, we highlight the critical culprits and provide suggestions as to how these issues can be rectified through the pursuit of a Human Data-driven Discovery (HD3) paradigm. Consistent with other examples of disruptive innovation, we propose that new levels of success are not dependent on new inventions, but rather on the strategic integration of existing data and technology assets. In support of these suggestions, we highlight the power of HD3, through recently published proof-of-concept applications in the areas of drug safety analysis and prediction, drug repositioning, the rational design of combination therapies and the global response to the COVID-19 pandemic. We conclude that innovators must play a key role in expediting the path to a largely human-focused, systems-based approach to drug discovery and research.

Assessment and Improvement of Drug Data Structuredness From Electronic Health Records: Algorithm Development and Validation

BACKGROUND

Digitization offers a multitude of opportunities to gain insights into current diagnostics and therapies from retrospective data. In this context, real-world data and their accessibility are of increasing importance to support unbiased and reliable research on big data. However, routinely collected data are not readily usable for research owing to the unstructured nature of health care systems and a lack of interoperability between these systems. This challenge is evident in drug data.

OBJECTIVE

This study aimed to present an approach that identifies and increases the structuredness of drug data while ensuring standardization according to Anatomical Therapeutic Chemical (ATC) classification.

METHODS

Our approach was based on available drug prescriptions and a drug catalog and consisted of 4 steps. First, we performed an initial analysis of the structuredness of local drug data to define a point of comparison for the effectiveness of the overall approach. Second, we applied 3 algorithms to unstructured data that translated text into ATC codes based on string comparisons in terms of ingredients and product names and performed similarity comparisons based on Levenshtein distance. Third, we validated the results of the 3 algorithms with expert knowledge based on the 1000 most frequently used prescription texts. Fourth, we performed a final validation to determine the increased degree of structuredness.

RESULTS

Initially, 47.73% (n=843,980) of 1,768,153 drug prescriptions were classified as structured. With the application of the 3 algorithms, we were able to increase the degree of structuredness to 85.18% (n=1,506,059) based on the 1000 most frequent medication prescriptions. In this regard, the combination of algorithms 1, 2, and 3 resulted in a correctness level of 100% (with 57,264 ATC codes identified), algorithms 1 and 3 resulted in 99.6% (with 152,404 codes identified), and algorithms 1 and 2 resulted in 95.9% (with 39,472 codes identified).

CONCLUSIONS

As shown in the first analysis steps of our approach, the availability of a product catalog to select during the documentation process is not sufficient to generate structured data. Our 4-step approach reduces the problems and reliably increases the structuredness automatically. Similarity matching shows promising results, particularly for entries with no connection to a product catalog. However, further enhancement of the correctness of such a similarity matching algorithm needs to be investigated in future work.

HARmonized Protocol Template to Enhance Reproducibility of hypothesis evaluating real-world evidence studies on treatment effects: A good practices report of a joint ISPE/ISPOR task force

PROBLEM

Ambiguity in communication of key study parameters limits the utility of real-world evidence (RWE) studies in healthcare decision-making. Clear communication about data provenance, design, analysis, and implementation is needed. This would facilitate reproducibility, replication in independent data, and assessment of potential sources of bias.

WHAT WE DID

The International Society for Pharmacoepidemiology (ISPE) and ISPOR-The Professional Society for Health Economics and Outcomes Research (ISPOR) convened a joint task force, including representation from key international stakeholders, to create a harmonized protocol template for RWE studies that evaluate a treatment effect and are intended to inform decision-making. The template builds on existing efforts to improve transparency and incorporates recent insights regarding the level of detail needed to enable RWE study reproducibility. The overarching principle was to reach for sufficient clarity regarding data, design, analysis, and implementation to achieve 3 main goals. One, to help investigators thoroughly consider, then document their choices and rationale for key study parameters that define the causal question (e.g., target estimand), two, to facilitate decision-making by enabling reviewers to readily assess potential for biases related to these choices, and three, to facilitate reproducibility.

STRATEGIES TO DISSEMINATE AND FACILITATE USE

Recognizing that the impact of this harmonized template relies on uptake, we have outlined a plan to introduce and pilot the template with key international stakeholders over the next 2 years.

CONCLUSION

The HARmonized Protocol Template to Enhance Reproducibility (HARPER) helps to create a shared understanding of intended scientific decisions through a common text, tabular and visual structure. The template provides a set of core recommendations for clear and reproducible RWE study protocols and is intended to be used as a backbone throughout the research process from developing a valid study protocol, to registration, through implementation and reporting on those implementation decisions.

A novel drug selection decision support model based on real-world medical data by the hybrid entropic weight TOPSIS method

BACKGROUND

The medicine selection method is a critical and challenging issue in medical insurance decision-making.

OBJECTIVES

This study proposed a real-world data-based multi-criteria decision analysis (MCDA) model with a hybrid entropic weight Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) algorithms to select satisfactory drugs.

METHODS

The evaluation index includes two levels: primary criteria and sub-criteria. Firstly, we proposed six primary criteria to form the value health framework. The primary criteria's weights were derived from the policymakers' questionnaire. Meanwhile, clinically relevant sub-criteria were derived from high-quality (screened by GRADE scores) clinical-research literature. Their weights are determined by the entropy weight (EW) algorithm. Secondly, we split the primary criteria into six mini-EW-TOPSIS models. Then, we obtained six ideal closeness degree scores (ICDS) for each candidate drug. Thirdly, we get the total utility score by linear weighting the ICDS. The higher the utility score, the higher the ranking.

RESULTS

A national multicenter real-world case study of the ranking of four generic antibiotics validated the proposed model. This model is verified by comparative experiments and sensitivity analysis. The whole ranking model was consistent and reliable. Based on these results, medical policymakers can intuitively and easily understand the characteristics of each drug to facilitate follow-up drug policy-making.

CONCLUSION

The ranking algorithm combines the objective characteristics of medicine and policy makers' opinions, which can improve the applicability of the results. This model can help decision-makers, clinicians, and related researchers better understand the drug assessment process.

A comparative study of antihypertensive drugs prediction models for the elderly based on machine learning algorithms

Background

Globally, blood pressure management strategies were ineffective, and a low percentage of patients receiving hypertension treatment had their blood pressure controlled. In this study, we aimed to build a medication prediction model by correlating patient attributes with medications to help physicians quickly and rationally match appropriate medications.

Methods

We collected clinical data from elderly hypertensive patients during hospitalization and combined statistical methods and machine learning (ML) algorithms to filter out typical indicators. We constructed five ML models to evaluate all datasets using 5-fold cross-validation. Include random forest (RF), support vector machine (SVM), light gradient boosting machine (LightGBM), artificial neural network (ANN), and naive Bayes (NB) models. And the performance of the models was evaluated using the micro-F1 score.

Results

Our experiments showed that by statistical methods and ML algorithms for feature selection, we finally selected Age, SBP, DBP, Lymph, RBC, HCT, MCHC, PLT, AST, TBIL, Cr, UA, Urea, K, Na, Ga, TP, GLU, TC, TG, γ-GT, Gender, HTN CAD, and RI as feature metrics of the models. LightGBM had the best prediction performance with the micro-F1 of 78.45%, which was higher than the other four models.

Conclusion

LightGBM model has good results in predicting antihypertensive medication regimens, and the model can be beneficial in improving the personalization of hypertension treatment.

The US Food and Drug Administration Sentinel System: a national resource for a learning health system

The US Food and Drug Administration (FDA) created the Sentinel System in response to a requirement in the FDA Amendments Act of 2007 that the agency establish a system for monitoring risks associated with drug and biologic products using data from disparate sources. The Sentinel System has completed hundreds of analyses, including many that have directly informed regulatory decisions. The Sentinel System also was designed to support a national infrastructure for a learning health system. Sentinel governance and guiding principles were designed to facilitate Sentinel's role as a national resource. The Sentinel System infrastructure now supports multiple non-FDA projects for stakeholders ranging from regulated industry to other federal agencies, international regulators, and academics. The Sentinel System is a working example of a learning health system that is expanding with the potential to create a global learning health system that can support medical product safety assessments and other research.

Understanding variation in the results of real-world evidence studies that seem to address the same question

OBJECTIVES

Multiple database studies on the same question, conducted by different investigators using different approaches or different data sources, can be considered sensitivity analyses for the same causal treatment effect question. We evaluated the contribution of alternative study design parameters and analysis choices to variation in estimates of the risk of major bleeding with dabigatran compared with warfarin.

STUDY DESIGN AND SETTING

We followed a 7-step process: (1) identify published studies asking the same question, (2) independently reproduce selected studies in the same data sources as the original authors, (3) contact original authors, (4) evaluate validity, (5) document critical study parameter specifications, (6) implement a designed matrix of variations in study parameters based on the original studies, and (7) evaluate contributors to variation in results.

RESULTS

Most variation remained unexplained (60-88%). Of the explained variation, two-thirds were related to data and population differences, and one-third were related to the use of alternative study design and analysis parameters. Among these, the most prominent were differences in outcome algorithms and criteria used to define follow-up.

CONCLUSION

When making policy decisions based on database study findings, it is important to evaluate the validity, consistency, and robustness of results to alternative design and analysis decisions.

HARmonized Protocol Template to Enhance Reproducibility of Hypothesis Evaluating Real-World Evidence Studies on Treatment Effects: A Good Practices Report of a Joint ISPE/ISPOR Task Force

OBJECTIVES

Ambiguity in communication of key study parameters limits the utility of real-world evidence (RWE) studies in healthcare decision-making. Clear communication about data provenance, design, analysis, and implementation is needed. This would facilitate reproducibility, replication in independent data, and assessment of potential sources of bias.

METHODS

The International Society for Pharmacoepidemiology (ISPE) and ISPOR-The Professional Society for Health Economics and Outcomes Research (ISPOR) convened a joint task force, including representation from key international stakeholders, to create a harmonized protocol template for RWE studies that evaluate a treatment effect and are intended to inform decision-making. The template builds on existing efforts to improve transparency and incorporates recent insights regarding the level of detail needed to enable RWE study reproducibility. The over-arching principle was to reach for sufficient clarity regarding data, design, analysis, and implementation to achieve 3 main goals. One, to help investigators thoroughly consider, then document their choices and rationale for key study parameters that define the causal question (e.g., target estimand), two, to facilitate decision-making by enabling reviewers to readily assess potential for biases related to these choices, and three, to facilitate reproducibility.

STRATEGIES TO DISSEMINATE AND FACILITATE USE

Recognizing that the impact of this harmonized template relies on uptake, we have outlined a plan to introduce and pilot the template with key international stakeholders over the next 2 years.

CONCLUSION

The HARmonized Protocol Template to Enhance Reproducibility (HARPER) helps to create a shared understanding of intended scientific decisions through a common text, tabular and visual structure. The template provides a set of core recommendations for clear and reproducible RWE study protocols and is intended to be used as a backbone throughout the research process from developing a valid study protocol, to registration, through implementation and reporting on those implementation decisions.

Developing a common data model approach for DISCOVER CKD: A retrospective, global cohort of real-world patients with chronic kidney disease

OBJECTIVES

To describe a flexible common data model (CDM) approach that can be efficiently tailored to study-specific needs to facilitate pooled patient-level analysis and aggregated/meta-analysis of routinely collected retrospective patient data from disparate data sources; and to detail the application of this CDM approach to the DISCOVER CKD retrospective cohort, a longitudinal database of routinely collected (secondary) patient data of individuals with chronic kidney disease (CKD).

METHODS

The flexible CDM approach incorporated three independent, exchangeable components that preceded data mapping and data model implementation: (1) standardized code lists (unifying medical events from different coding systems); (2) laboratory unit harmonization tables; and (3) base cohort definitions. Events between different coding vocabularies were not mapped code-to-code; for each data source, code lists of labels were curated at the entity/event level. A study team of epidemiologists, clinicians, informaticists, and data scientists were included within the validation of each component.

RESULTS

Applying the CDM to the DISCOVER CKD retrospective cohort, secondary data from 1,857,593 patients with CKD were harmonized from five data sources, across three countries, into a discrete database for rapid real-world evidence generation.

CONCLUSIONS

This flexible CDM approach facilitates evidence generation from real-world data within the DISCOVER CKD retrospective cohort, providing novel insights into the epidemiology of CKD that may expedite improvements in diagnosis, prognosis, early intervention, and disease management. The adaptable architecture of this CDM approach ensures scalable, fast, and efficient application within other therapy areas to facilitate the combined analysis of different types of secondary data from multiple, heterogeneous sources.

Patient characteristics, pain treatment patterns, and incidence of total joint replacement in a US population with osteoarthritis

BACKGROUND

Currently available medications for chronic osteoarthritis pain are only moderately effective, and their use is limited in many patients because of serious adverse effects and contraindications. The primary surgical option for osteoarthritis is total joint replacement (TJR). The objectives of this study were to describe the treatment history of patients with osteoarthritis receiving prescription pain medications and/or intra-articular corticosteroid injections, and to estimate the incidence of TJR in these patients.

METHODS

This retrospective, multicenter, cohort study utilized health plan administrative claims data (January 1, 2013, through December 31, 2019) of adult patients with osteoarthritis in the Innovation in Medical Evidence Development and Surveillance Distributed Database, a subset of the US FDA Sentinel Distributed Database. Patients were analyzed in two cohorts: those with prevalent use of "any pain medication" (prescription non-steroidal anti-inflammatory drugs [NSAIDs], opioids, and/or intra-articular corticosteroid injections) using only the first qualifying dispensing (index date); and those with prevalent use of "each specific pain medication class" with all qualifying treatment episodes identified.

RESULTS

Among 1 992 670 prevalent users of "any pain medication", pain medications prescribed on the index date were NSAIDs (596 624 [29.9%] patients), opioids (1 161 806 [58.3%]), and intra-articular corticosteroids (323 459 [16.2%]). Further, 92 026 patients received multiple pain medications on the index date, including 71 632 (3.6%) receiving both NSAIDs and opioids. Altogether, 20.6% of patients used an NSAID at any time following an opioid index dispensing and 17.2% used an opioid following an NSAID index dispensing. The TJR incidence rates per 100 person-years (95% confidence interval [CI]) were 3.21 (95% CI: 3.20-3.23) in the "any pain medication" user cohort, and among those receiving "each specific pain medication class" were NSAIDs, 4.63 (95% CI: 4.58-4.67); opioids, 7.45 (95% CI: 7.40-7.49); and intra-articular corticosteroids, 8.05 (95% CI: 7.97-8.13).

CONCLUSIONS

In patients treated with prescription medications for osteoarthritis pain, opioids were more commonly prescribed at index than NSAIDs and intra-articular corticosteroid injections. Of the pain medication classes examined, the incidence of TJR was highest in patients receiving intra-articular corticosteroids and lowest in patients receiving NSAIDs.

Reproducibility of real-world evidence studies using clinical practice data to inform regulatory and coverage decisions

Studies that generate real-world evidence on the effects of medical products through analysis of digital data collected in clinical practice provide key insights for regulators, payers, and other healthcare decision-makers. Ensuring reproducibility of such findings is fundamental to effective evidence-based decision-making. We reproduce results for 150 studies published in peer-reviewed journals using the same healthcare databases as original investigators and evaluate the completeness of reporting for 250. Original and reproduction effect sizes were positively correlated (Pearson's correlation = 0.85), a strong relationship with some room for improvement. The median and interquartile range for the relative magnitude of effect (e.g., hazard ratiooriginal/hazard ratioreproduction) is 1.0 [0.9, 1.1], range [0.3, 2.1]. While the majority of results are closely reproduced, a subset are not. The latter can be explained by incomplete reporting and updated data. Greater methodological transparency aligned with new guidance may further improve reproducibility and validity assessment, thus facilitating evidence-based decision-making. Study registration number: EUPAS19636.

The IeDEA harmonist data toolkit: A data quality and data sharing solution for a global HIV research consortium

We describe the design, implementation, and impact of a data harmonization, data quality checking, and dynamic report generation application in an international observational HIV research network. The IeDEA Harmonist Data Toolkit is a web-based application written in the open source programming language R, employs the R/Shiny and RMarkdown packages, and leverages the REDCap data collection platform for data model definition and user authentication. The Toolkit performs data quality checks on uploaded datasets, checks for conformance with the network's common data model, displays the results both interactively and in downloadable reports, and stores approved datasets in secure cloud storage for retrieval by the requesting investigator. Including stakeholders and users in the design process was key to the successful adoption of the application. A survey of regional data managers as well as initial usage metrics indicate that the Toolkit saves time and results in improved data quality, with a 61% mean reduction in the number of error records in a dataset. The generalized application design allows the Toolkit to be easily adapted to other research networks.

Distributed Quasi-Poisson regression algorithm for modeling multi-site count outcomes in distributed data networks

BACKGROUND

Observational studies incorporating real-world data from multiple institutions facilitate study of rare outcomes or exposures and improve generalizability of results. Due to privacy concerns surrounding patient-level data sharing across institutions, methods for performing regression analyses distributively are desirable. Meta-analysis of institution-specific estimates is commonly used, but has been shown to produce biased estimates in certain settings. While distributed regression methods are increasingly available, methods for analyzing count outcomes are currently limited. Count data in practice are commonly subject to overdispersion, exhibiting greater variability than expected under a given statistical model.

OBJECTIVE

We propose a novel computational method, a one-shot distributed algorithm for quasi-Poisson regression (ODAP), to distributively model count outcomes while accounting for overdispersion.

METHODS

ODAP incorporates a surrogate likelihood approach to perform distributed quasi-Poisson regression without requiring patient-level data sharing, only requiring sharing of aggregate data from each participating institution. ODAP requires at most three rounds of non-iterative communication among institutions to generate coefficient estimates and corresponding standard errors. In simulations, we evaluate ODAP under several data scenarios possible in multi-site analyses, comparing ODAP and meta-analysis estimates in terms of error relative to pooled regression estimates, considered the gold standard. In a proof-of-concept real-world data analysis, we similarly compare ODAP and meta-analysis in terms of relative error to pooled estimatation using data from the OneFlorida Clinical Research Consortium, modeling length of stay in COVID-19 patients as a function of various patient characteristics. In a second proof-of-concept analysis, using the same outcome and covariates, we incorporate data from the UnitedHealth Group Clinical Discovery Database together with the OneFlorida data in a distributed analysis to compare estimates produced by ODAP and meta-analysis.

RESULTS

In simulations, ODAP exhibited negligible error relative to pooled regression estimates across all settings explored. Meta-analysis estimates, while largely unbiased, were increasingly variable as heterogeneity in the outcome increased across institutions. When baseline expected count was 0.2, relative error for meta-analysis was above 5% in 25% of iterations (250/1000), while the largest relative error for ODAP in any iteration was 3.59%. In our proof-of-concept analysis using only OneFlorida data, ODAP estimates were closer to pooled regression estimates than those produced by meta-analysis for all 15 covariates. In our distributed analysis incorporating data from both OneFlorida and the UnitedHealth Group Clinical Discovery Database, ODAP and meta-analysis estimates were largely similar, while some differences in estimates (as large as 13.8%) could be indicative of bias in meta-analytic estimates.

CONCLUSIONS

ODAP performs privacy-preserving, communication-efficient distributed quasi-Poisson regression to analyze count outcomes using data stored within multiple institutions. Our method produces estimates nearly matching pooled regression estimates and sometimes more accurate than meta-analysis estimates, most notably in settings with relatively low counts and high outcome heterogeneity across institutions.

Advancing drug safety science by integrating molecular knowledge with post-marketing adverse event reports

Promising drug development efforts may frequently fail due to unintended adverse reactions. Several methods have been developed to analyze such data, aiming to improve pharmacovigilance and drug safety. In this work, we provide a brief review of key directions to quantitatively analyzing adverse events and explore the potential of augmenting these methods using additional molecular data descriptors. We argue that molecular expansion of adverse event data may provide a path to improving the insights gained through more traditional pharmacovigilance approaches. Examples include the ability to assess statistical relevance with respect to underlying biomolecular mechanisms, the ability to generate plausible causative hypotheses and/or confirmation where possible, the ability to computationally study potential clinical trial designs and/or results, as well as the further provision of advanced features incorporated in innovative methods, such as machine learning. In summary, molecular data expansion provides an elegant way to extend mechanistic modeling, systems pharmacology, and patient‐centered approaches for the assessment of drug safety. We anticipate that such advances in real‐world data informatics and outcome analytics will help to better inform public health, via the improved ability to prospectively understand and predict various types of drug‐induced molecular perturbations and adverse events.

A Framework for Visualizing Study Designs and Data Observability in Electronic Health Record Data

Background

There is growing interest in using evidence generated from clinical practice data to support regulatory, coverage and other healthcare decision-making. A graphical framework for depicting longitudinal study designs to mitigate this barrier was introduced and has found wide acceptance. We sought to enhance the framework to contain information that helps readers assess the appropriateness of the source data in which the study design was applied.

Methods

For the enhanced graphical framework, we added a simple visualization of data type and observability to capture differences between electronic health record (EHR) and other registry data that may have limited data continuity and insurance claims data that have enrollment files.

Results

We illustrate the revised graphical framework with 2 example studies conducted using different data sources, including administrative claims only, EHR only, linked claims and EHR, as well as specialty community based EHRs with and without external linkages.

Conclusion

The enhanced visualization framework is important because evaluation of study validity needs to consider the triad of study question, design, and data together. Any given data source or study design may be appropriate for some questions but not others.

eTRANSAFE: Building a sustainable framework to share reproducible drug safety knowledge with the public domain

Integrative drug safety research in translational health informatics has rapidly evolved and included data that are drawn in from many resources, combining diverse data that are either reused from (curated) repositories, or newly generated at source. Each resource is mandated by different sets of metadata rules that are imposed on the incoming data. Combination of the data cannot be readily achieved without interference of data stewardship and the top-down policy guidelines that supervise and inform the process for data combination to aid meaningful interpretation and analysis of such data.

The eTRANSAFE Consortium's effort to drive integrative drug safety research at a large scale hereby present the lessons learnt and the proposal of solution at the guidelines in practice at this Innovative Medicines Initiative (IMI) project. Recommendations in these guidelines were compiled from feedback received from key stakeholders in regulatory agencies, EFPIA companies, and academic partners. The research reproducibility guidelines presented in this study lay the foundation for a comprehensive data sharing and knowledge management plans accounting for research data management in the drug safety space - FAIR data sharing guidelines, and the model verification guidelines as generic deliverables that best practices that can be reused by other scientific community members at large.

FAIR data sharing is a dynamic landscape that rapidly evolves with fast-paced technology advancements. The research reproducibility in drug safety guidelines introduced in this study provides a reusable framework that can be adopted by other research communities that aim to integrate public and private data in biomedical research space.

Using inpatient electronic medical records to study influenza for pandemic preparedness

BACKGROUND

We assessed the ability to identify key data relevant to influenza and other respiratory virus surveillance in a large-scale US-based hospital electronic medical record (EMR) dataset using seasonal influenza as a use case. We describe characteristics and outcomes of hospitalized influenza cases across three seasons.

METHODS

We identified patients with an influenza diagnosis between March 2017 and March 2020 in 140 US hospitals as part of the US FDA's Sentinel System. We calculated descriptive statistics on the presence of high-risk conditions, influenza antiviral administrations, and severity endpoints.

RESULTS

Among 5.1 million hospitalizations, we identified 29,520 hospitalizations with an influenza diagnosis; 64% were treated with an influenza antiviral within 2 days of admission, and 25% were treated >2 days after admission. Patients treated >2 days after admission had more comorbidities than patients treated within 2 days of admission. Patients never treated during hospitalization had more documentation of cardiovascular and other diseases than treated patients. We observed more severe endpoints in patients never treated (death = 3%, mechanical ventilation [MV] = 9%, intensive care unit [ICU] = 26%) or patients treated >2 days after admission (death = 2%, MV = 14%, ICU = 32%) than in patients treated earlier (treated on admission: death = 1%, MV = 5%, ICU = 23%, treated within 2 days of admission: death = 1%, MV = 7%, ICU = 27%).

CONCLUSIONS

We identified important trends in influenza severity related to treatment timing in a large inpatient dataset, laying the groundwork for the use of this and other inpatient EMR data for influenza and other respiratory virus surveillance.