Quantifying outcome misclassification in multi-database studies: The case study of pertussis in the ADVANCE project

Validation of algorithms in studies based on routinely collected health data: general principles

Clinicians, researchers, regulators, and other decision-makers increasingly rely on evidence from real-world data (RWD), including data routinely accumulating in health and administrative databases. RWD studies often rely on algorithms to operationalize variable definitions. An algorithm is a combination of codes or concepts used to identify persons with a specific health condition or characteristic. Establishing the validity of algorithms is a prerequisite for generating valid study findings that can ultimately inform evidence-based health care. In this paper, we aim to systematize terminology, methods, and practical considerations relevant to the conduct of validation studies of RWD-based algorithms. We discuss measures of algorithm accuracy, gold/reference standards, study size, prioritization of accuracy measures, algorithm portability, and implications for interpretation. Information bias is common in epidemiologic studies, underscoring the importance of transparency in decisions regarding choice and prioritizing measures of algorithm validity. The validity of an algorithm should be judged in the context of a data source, and one size does not fit all. Prioritizing validity measures within a given data source depends on the role of a given variable in the analysis (eligibility criterion, exposure, outcome, or covariate). Validation work should be part of routine maintenance of RWD sources. This article is part of a Special Collection on Pharmacoepidemiology.

Describing diversity of real world data sources in pharmacoepidemiologic studies: The DIVERSE scoping review

PURPOSE

Real-world evidence (RWE) is increasingly used for medical regulatory decisions, yet concerns persist regarding its reproducibility and hence validity. This study addresses reproducibility challenges associated with diversity across real-world data sources (RWDS) repurposed for secondary use in pharmacoepidemiologic studies. Our aims were to identify, describe and characterize practices, recommendations and tools for collecting and reporting diversity across RWDSs, and explore how leveraging diversity could improve the quality of evidence.

METHODS

In a preliminary phase, keywords for a literature search and selection tool were designed using a set of documents considered to be key by the coauthors. Next, a systematic search was conducted up to December 2021. The resulting documents were screened based on titles and abstracts, then based on full texts using the selection tool. Selected documents were reviewed to extract information on topics related to collecting and reporting RWDS diversity. A content analysis of the topics identified explicit and latent themes.

RESULTS

Across the 91 selected documents, 12 topics were identified: 9 dimensions used to describe RWDS (organization accessing the data source, data originator, prompt, inclusion of population, content, data dictionary, time span, healthcare system and culture, and data quality), tools to summarize such dimensions, challenges, and opportunities arising from diversity. Thirty-six themes were identified within the dimensions. Opportunities arising from data diversity included multiple imputation and standardization.

CONCLUSIONS

The dimensions identified across a large number of publications lay the foundation for formal guidance on reporting diversity of data sources to facilitate interpretation and enhance replicability and validity of RWE.

Methodology of the brodalumab assessment of hazards: a multicentre observational safety (BRAHMS) study

INTRODUCTION

Safe and effective pharmacological treatment is of paramount importance for treating severe psoriasis. Brodalumab, a monoclonal antibody against interleukin (IL) 17 receptor A, was granted marketing authorisation in the EU in 2017. The European Medicines Agency requested a postauthorisation safety study of brodalumab to address potential safety issues raised during drug development regarding major adverse cardiovascular events, suicidal conduct, cancer and serious infections.

METHODS AND ANALYSIS

BRodalumab Assessment of Hazards: A Multinational Safety is a multicentre observational safety study of brodalumab running from 2017 to 2029 using population-based healthcare databases from Denmark, Sweden, Norway, Netherlands, Germany and three different centres in Italy. A distributed database network approach is used, such that only aggregate data are exchanged between sites.Two types of designs are used: a case-time-control design to study acute effects of transient treatment and a variation of the new user active comparator design to study the effects of transient or chronic treatment. As comparators, inhibitors of TNF-α, inhibitors of IL-12 and IL-23, and other inhibitors of cytokine IL-17A are included.In the self-controlled case-time-control design, the risk of developing the outcome of interest during periods of brodalumab use is compared within individuals to the risk in periods without use.In the active comparator cohort design, new users of brodalumab are identified and matched to new users of active comparators. Potential baseline confounders are adjusted for by using propensity score modelling. For outcomes that potentially require large cumulative exposure, an adapted active comparator design has been developed.

ETHICS AND DISSEMINATION

The study is approved by relevant authorities in Denmark, Norway, Sweden, the Netherlands, Germany and Italy in line with the relevant legislation at each site. Data confidentiality is secured by the distributed network approach. Results will be published in peer-reviewed journals.

TRIAL REGISTRATION NUMBER

EUPAS30280.

From Inception to ConcePTION: Genesis of a Network to Support Better Monitoring and Communication of Medication Safety During Pregnancy and Breastfeeding

In 2019, the Innovative Medicines Initiative (IMI) funded the ConcePTION project-Building an ecosystem for better monitoring and communicating safety of medicines use in pregnancy and breastfeeding: validated and regulatory endorsed workflows for fast, optimised evidence generation-with the vision that there is a societal obligation to rapidly reduce uncertainty about the safety of medication use in pregnancy and breastfeeding. The present paper introduces the set of concepts used to describe the European data sources involved in the ConcePTION project and illustrates the ConcePTION Common Data Model (CDM), which serves as the keystone of the federated ConcePTION network. Based on data availability and content analysis of 21 European data sources, the ConcePTION CDM has been structured with six tables designed to capture data from routine healthcare, three tables for data from public health surveillance activities, three curated tables for derived data on population (e.g., observation time and mother-child linkage), plus four metadata tables. By its first anniversary, the ConcePTION CDM has enabled 13 data sources to run common scripts to contribute to major European projects, demonstrating its capacity to facilitate effective and transparent deployment of distributed analytics, and its potential to address questions about utilization, effectiveness, and safety of medicines in special populations, including during pregnancy and breastfeeding, and, more broadly, in the general population.

Methodological frontiers in vaccine safety: qualifying available evidence for rare events, use of distributed data networks to monitor vaccine safety issues, and monitoring the safety of pregnancy interventions

While vaccines are rigorously tested for safety and efficacy in clinical trials, these trials do not include enough subjects to detect rare adverse events, and they generally exclude special populations such as pregnant women. It is therefore necessary to conduct postmarketing vaccine safety assessments using observational data sources. The study of rare events has been enabled in through large linked databases and distributed data networks, in combination with development of case-centred methods. Distributed data networks necessitate common protocols, definitions, data models and analytics and the processes of developing and employing these tools are rapidly evolving. Assessment of vaccine safety in pregnancy is complicated by physiological changes, the challenges of mother-child linkage and the need for long-term infant follow-up. Potential sources of bias including differential access to and utilisation of antenatal care, immortal time bias, seasonal timing of pregnancy and unmeasured determinants of pregnancy outcomes have yet to be fully explored. Available tools for assessment of evidence generated in postmarketing studies may downgrade evidence from observational data and prioritise evidence from randomised controlled trials. However, real-world evidence based on real-world data is increasingly being used for safety assessments, and new tools for evaluating real-world evidence have been developed. The future of vaccine safety surveillance, particularly for rare events and in special populations, comprises the use of big data in single countries as well as in collaborative networks. This move towards the use of real-world data requires continued development of methodologies to generate and assess real world evidence.

Incidence Rates of Autoimmune Diseases in European Healthcare Databases: A Contribution of the ADVANCE Project

INTRODUCTION

The public-private ADVANCE collaboration developed and tested a system to generate evidence on vaccine benefits and risks using European electronic healthcare databases. In the safety of vaccines, background incidence rates are key to allow proper monitoring and assessment. The goals of this study were to compute age-, sex-, and calendar-year stratified incidence rates of nine autoimmune diseases in seven European healthcare databases from four countries and to assess validity by comparing with published data.

METHODS

Event rates were calculated for the following outcomes: acute disseminated encephalomyelitis, Bell's palsy, Guillain-Barré syndrome, immune thrombocytopenia purpura, Kawasaki disease, optic neuritis, narcolepsy, systemic lupus erythematosus, and transverse myelitis. Cases were identified by diagnosis codes. Participating organizations/databases originated from Denmark, Italy, Spain, and the UK. The source population comprised all persons registered, with at least 1 year of data prior to the study start, or follow-up from birth. Stratified incidence rates were computed per database over the period 2003 to 2014.

RESULTS

Between 2003 and 2014, 148,947 incident cases of nine autoimmune diseases were identified. Crude incidence rates were highest for Bell's palsy [23.8/100,000 person-years (PYs), 95% confidence interval (CI) 23.6-24.1] and lowest for Kawasaki disease (0.7/100,000 PYs, 95% CI 0.6-0.7). Specific patterns were observed by sex, age, calendar time, and data sources. Rates were comparable with published estimates.

CONCLUSION

A range of autoimmune events could be identified in the ADVANCE system. Estimation of rates indicated consistency across selected European healthcare databases, as well as consistency with US published data.

Different Strategies to Execute Multi-Database Studies for Medicines Surveillance in Real-World Setting: A Reflection on the European Model

Although postmarketing studies conducted in population-based databases often contain information on patients in the order of millions, they can still be underpowered if outcomes or exposure of interest is rare, or the interest is in subgroup effects. Combining several databases might provide the statistical power needed. A multi-database study (MDS) uses at least two healthcare databases, which are not linked with each other at an individual person level, with analyses carried out in parallel across each database applying a common study protocol. Although many MDSs have been performed in Europe in the past 10 years, there is a lack of clarity on the peculiarities and implications of the existing strategies to conduct them. In this review, we identify four strategies to execute MDSs, classified according to specific choices in the execution: (A) local analyses, where data are extracted and analyzed locally, with programs developed by each site; (B) sharing of raw data, where raw data are locally extracted and transferred without analysis to a central partner, where all the data are pooled and analyzed; (C) use of a common data model with study-specific data, where study-specific data are locally extracted, loaded into a common data model, and processed locally with centrally developed programs; and (D) use of general common data model, where all local data are extracted and loaded into a common data model, prior to and independent of any study protocol, and protocols are incorporated in centrally developed programs that run locally. We illustrate differences between strategies and analyze potential implications.

Disease misclassification in electronic healthcare database studies: Deriving validity indices-A contribution from the ADVANCE project

There is a strong and continuously growing interest in using large electronic healthcare databases to study health outcomes and the effects of pharmaceutical products. However, concerns regarding disease misclassification (i.e. classification errors of the disease status) and its impact on the study results are legitimate. Validation is therefore increasingly recognized as an essential component of database research. In this work, we elucidate the interrelations between the true prevalence of a disease in a database population (i.e. prevalence assuming no disease misclassification), the observed prevalence subject to disease misclassification, and the most common validity indices: sensitivity, specificity, positive and negative predictive value. Based on this, we obtained analytical expressions to derive all the validity indices and true prevalence from the observed prevalence and any combination of two other parameters. The analytical expressions can be used for various purposes. Most notably, they can be used to obtain an estimate of the observed prevalence adjusted for outcome misclassification from any combination of two validity indices and to derive validity indices from each other which would otherwise be difficult to obtain. To allow researchers to easily use the analytical expressions, we additionally developed a user-friendly and freely available web-application.

ADVANCE database characterisation and fit for purpose assessment for multi-country studies on the coverage, benefits and risks of pertussis vaccinations

INTRODUCTION

The public-private ADVANCE consortium (Accelerated development of vaccine benefit-risk collaboration in Europe) aimed to assess if electronic healthcare databases can provide fit-for purpose data for collaborative, distributed studies and monitoring of vaccine coverage, benefits and risks of vaccines.

OBJECTIVE

To evaluate if European healthcare databases can be used to estimate vaccine coverage, benefit and/or risk using pertussis-containing vaccines as an example.

METHODS

Characterisation was conducted using open-source Java-based (Jerboa) software and R scripts. We obtained: (i) The general characteristics of the database and data source (meta-data) and (ii) a detailed description of the database population (size, representatively of age/sex of national population, rounding of birth dates, delay between birth and database entry), vaccinations (number of vaccine doses, recording of doses, pattern of doses by age and coverage) and events of interest (diagnosis codes, incidence rates). A total of nine databases (primary care, regional/national record linkage) provided data on events (pertussis, pneumonia, death, fever, convulsions, injection site reactions, hypotonic hypo-responsive episode, persistent crying) and vaccines (acellular pertussis and whole cell pertussis) related to the pertussis proof of concept studies.

RESULTS

The databases contained data for a total population of 44 million individuals. Seven databases had recorded doses of vaccines. The pertussis coverage estimates were similar to those reported by the World Health Organisation (WHO). Incidence rates of events were comparable in magnitude and age-distribution between databases with the same characteristics. Several conditions (persistent crying and somnolence) were not captured by the databases for which outcomes were restricted to hospital discharge diagnoses.

CONCLUSION

The database characterisation programs and workflows allowed for an efficient, transparent and standardised description and verification of electronic healthcare databases which may participate in pertussis vaccine coverage, benefit and risk studies. This approach is ready to be used for other vaccines/events to create readiness for participation in other vaccine related studies.

Advance system testing: Vaccine benefit studies using multi-country electronic health data - The example of pertussis vaccination

The Accelerated Development of VAccine benefit-risk Collaboration in Europe (ADVANCE), a public-private consortium, implemented and tested a distributed network system for the generation of evidence on the benefits-risks of marketed vaccines in Europe. We tested the system by estimating the incidence rate (IR) of pertussis and pertussis-related complications in children vaccinated with acellular (aP) and whole-cell (wP) pertussis vaccine. Data from seven electronic databases from four countries (Denmark: AUH and SSI, Spain: SIDIAP and BIFAP, UK: THIN and RCGP RSC and Italy: Pedianet) were included in a retrospective cohort analysis. Exposure was defined as any pertussis vaccination (aP or wP). The follow-up time started 14 days after the first dose. Children who had received any pertussis vaccine from January 1990 to December 2015 were included (those who switched type, or had unknown type were excluded). The outcomes of interest were confirmed or suspected pertussis and pertussis-related pneumonia and generalised convulsions within one month of pertussis diagnosis and death within three months of pertussis diagnosis. The cohort comprised 2,886,367 children ≤5 years of age. Data on wP and aP vaccination were available in three and seven databases, respectively. The IRs (per 100,000 person-years) for pertussis varied largely and ranged between 0.15 (95% CI: 0.12; 0.19) and 1.15 (95% CI: 1.07; 1.23), and the trends over time was consistent with those observed from national surveillance databases for confirmed pertussis. The pertussis IRs decreased as the number of wP and aP vaccine doses increased. Pertussis-related complications were rare (89 pneumonia, 7 generalised convulsions and no deaths) and their relative risk (vs. non-pertussis) could not be reliably estimated. The study demonstrated the feasibility of the ADVANCE system to estimate the change in pertussis IRs following pertussis vaccination. Larger sample sizes would provide additional power to compare the risk for complications between children with and without pertussis. The feasibility of vaccine-type specific effectiveness studies may be considered in the future.

ADVANCE system testing: Can safety studies be conducted using electronic healthcare data? An example using pertussis vaccination

INTRODUCTION

The Accelerated Development of Vaccine benefit-risk Collaboration in Europe (ADVANCE) public-private collaboration, aimed to develop and test a system for rapid benefit-risk monitoring of vaccines using healthcare databases in Europe. The objective of this proof-of-concept (POC) study was to test the feasibility of the ADVANCE system to generate incidence rates (IRs) per 1000 person-years and incidence rate ratios (IRRs) for risks associated with whole cell- (wP) and acellular- (aP) pertussis vaccines, occurring in event-specific risk windows in children prior to their pre-school-entry booster.

METHODS

The study population comprised almost 5.1 million children aged 1 month to <6 years vaccinated with wP or aP vaccines during the study period from 1 January 1990 to 31 December 2015. Data from two Danish hospital (H) databases (AUH and SSI) and five primary care (PC) databases from, UK (THIN and RCGP RSC), Spain (SIDIAP and BIFAP) and Italy (Pedianet) were analysed. Database-specific IRRs between risk vs. non-risk periods were estimated in a self-controlled case series study and pooled using random-effects meta-analyses.

RESULTS

The overall IRs were: fever, 58.2 (95% CI: 58.1; 58.3), 96.9 (96.7; 97.1) for PC DBs and 8.56 (8.5; 8.6) for H DBs; convulsions, 7.6 (95% CI: 7.6; 7.7), 3.55 (3.5; 3.6) for PC and 12.87 (12.8; 13) for H; persistent crying, 3.9 (95% CI: 3.8; 3.9) for PC, injection-site reactions, 2.2 (95% CI 2.1; 2.2) for PC, hypotonic hypo-responsive episode (HHE), 0.4 (95% CI: 0.4; 0.4), 0.6 (0.6; 0.6) for PC and 0.2 (0.2; 0.3) for H; and somnolence: 0.3 (95% CI: 0.3; 0.3) for PC. The pooled IRRs for persistent crying, fever, and ISR, adjusted for age and healthy vaccinee period were higher after wP vs. aP vaccination, and lower for convulsions, for all doses. The IRR for HHE was slightly lower for wP than aP, while wP was associated with somnolence only for dose 1 and dose 3 compared with aP.

CONCLUSIONS

The estimated IRs and IRRs were comparable with published data, therefore demonstrating that the ADVANCE system was able to combine several European healthcare databases to assess vaccine safety data for wP and aP vaccination.