Real-World Data: Assessing Electronic Health Records and Medical Claims Data to Support Regulatory Decision-Making for Drug and Biological Products

Cost-utility evaluation of mammography screening program in Taiwan based on real-world data accounting for false positives

PURPOSE

We estimated stage-specific loss of quality-adjusted life expectancy (loss-of-QALE) and weighted by stage distributions considering false-positive (FP) costs for cost-utility evaluation of mammography screening in Taiwan.

METHODS

FP costs included related reimbursements by the National Health Insurance for inpatients' and outpatients' services within one year after positive screening without BC diagnosis. Using the EuroQol five dimensions questionnaire (EQ-5D-3L), we collected the utilities from 1,181 women who visited a medical center with 2,247 repeated measurements in 2011-2021. We used a rolling-over algorithm to extrapolate survival to lifetime to estimate QALEs by multiplying survival probability with utilities and the loss of QALEs by comparing with age- and calendar year-matched referents. We calculated the incremental cost-effectiveness ratio (ICER) yearly by comparing the stage proportion weighted sums of loss-of-QALE between women detected by screening versus non-screening within six-month observed intervals in 2004-2013.

RESULTS

QALEs of stages I, II, III, and IV were 29.4, 25.0, 18.2, and 4.5 years, respectively, while loss-of-QALEs were 0.3, 4.2, 10.6, and 22.9 quality-adjusted life year (QALYs), respectively. A total of 355,489 (11.1 %) FP were found with an average cost of US$ 2,126 per screen-detected BC. After the nationwide promotion of mammography in 2010-2013, ICER was US$ 855 per QALY.

CONCLUSION

The mammography screening, which exceeded 530,000 women with a 22 % coverage rate, showed promising cost-utility; the ICER was about one-third of the willingness-to-pay (WTP) of one gross domestic product per QALY. Future studies are warranted to explore the saving of productivity loss from a societal perspective.

Real world evidence of insulin and biosimilar insulin therapy-Opportunities to improve adherence, outcomes and cost-effectiveness

Insulin has been discovered for more than a century; however, its benefits to people with diabetes are yet to be fully realized due to barriers related to access, quality of care and costs. Insulin therapy remains the cornerstone of diabetes management. The multicausality of diabetes and its subtypes calls for comprehensive phenotyping and use of biomarkers to ensure timely use of insulin to achieve early glycaemic control for long-term benefits. Biosimilar insulins are biologic products that closely resemble originator insulins without significant differences in safety or efficacy. The lower investment costs needed for research and development make biosimilar insulin more affordable to improve access. While the efficacy of insulin products is proven in controlled settings, real world evidence (RWE) from real world data (RWD) plays a crucial role in assessing the safety, efficacy, cost-effectiveness, adherence to and impacts of different insulin products, including biosimilars, on clinical outcomes. In this narrative review, we summarized the trends of insulin use and patterns of biosimilar insulin utilization in real world practice across different regions. We reviewed RWE on clinical safety, efficacy and cost-effectiveness of biosimilar insulin, as well as therapeutic inertia and non-adherence with insulin therapy. We also highlighted barriers to insulin adherence and enablers for persistence, along with potential solutions to promote the use of insulin and technologies for optimizing glycaemic control in different subtypes of diabetes. During our extensive literature review, we identified data gaps in the usage of biosimilar insulin in real world practice. We advocate for implementing a diabetes register designed fit-for-purpose, managed by a trained doctor-nurse team with system support, to generate RWE. By setting up registers with structured data collection, we can generate high quality data for integrative analysis with electronic health records (EHR) and health claims to evaluate the impacts of insulin products and other diabetes programmes on clinical outcomes, quality of life and healthcare costs to inform practice and policies. PLAIN LANGUAGE SUMMARY: Diabetes affects approximately 10.5% of the global population and insulin is a vital treatment for diabetes management. Insulin was discovered more than a century ago, although its benefits to people with diabetes are yet to be fully realized due to barriers related to access, quality of care, and costs. Real-world evidence from real-world data plays a crucial role in assessing the safety, efficacy, cost-effectiveness, adherence to, and impacts of different insulin products, including biosimilars, on clinical outcomes. In this publication, the authors provided a detailed review of the patterns of use and cost-effectiveness of biosimilar insulin, and identified major data gaps. The authors explained the methodology, utility, and limitations of generating real-world evidence based on real-world data from sources such as registers, electronic health records and health claims for assessing treatment effectiveness and safety. The authors proposed the implementation of a purpose-built diabetes register with structured data collection, managed by a trained doctor-nurse team with system support. These high-quality data can be integrated with electronic health records and health claims for evaluation of interventions, including insulin on outcomes, quality of life, and costs to inform practice and policy. Based on these premises and available data, the authors summarized trends in insulin use including biosimilar insulin, and reviewed real-world evidence on the safety, efficacy, and cost-effectiveness of these products. They also identified barriers like therapeutic inertia and non-adherence, discussed enablers for persistence, and proposed solutions to evaluate the impacts of insulin products and other diabetes programs on clinical outcomes, quality of life, and healthcare costs to inform practice and policies.

Linkage of Clinical Trial Data to Routinely Collected Data Sources: A Scoping Review

Importance

Patients who participate in clinical trials generate valuable routinely collected data (eg, medical records, electronic health records, claims databases, disease registries, or vital statistics) through their routine interactions with the health care system before, during, and after the trial. When this routinely collected data is linked at the participant level, it can supplement active data collection in the trial and provide deeper insights into the benefits, risks, and costs of treatments.

Objective

To review clinical trials linked to routinely collected data in various countries, identifying the use cases and designs of these linkage studies.

Evidence Review

Research articles that reported the linkage of clinical trials to medical records, electronic health records, claims databases, disease registries, or vital statistics in their title or abstract were searched for in PubMed and MEDLINE. The search covered the period from January 1, 2016, through December 30, 2023. Opinion pieces, study protocols, or studies that involved interventions other than medications, dietary supplements, vaccines, devices, procedures, or diagnostics (eg, behavioral interventions) were excluded. Study eligibility and data extraction were performed independently by 2 reviewers to ensure the accuracy of findings.

Findings

Of the 990 abstracts initially screened, a full text review was conducted for 147 articles. In total, 71 studies were included in the results, including 42 medication, vaccine, and dietary supplement trials (59.2%) and 29 device, procedure, or diagnostic trials (40.8%). Of these 71 studies, 24 (32.4%) were conducted in the US. In 32 studies (45.1%), consent for linkage was obtained prospectively as part of the main trial, while 33 studies (46.5%) received a waiver of authorization from the respective ethical review boards. The most frequent use cases of linkage to were posttrial long-term follow-up (22 studies [31.0%]), capturing primary or secondary outcomes of trials (19 studies [26.8%]), validation of routinely collected data outcomes (17 studies [23.9%]), and measuring health care resource utilization and cost in trials (12 studies [16.9%]).

Conclusions and Relevance

This study found that the linkage of patients' clinical trial data to routinely collected data has been implemented in several trials for various use cases and that most studies obtained consent for linkage prospectively as part of the main trial or received a waiver of authorization from ethical review boards. These findings demonstrate the feasibility and provide an overview of the use cases for linking trials to routinely collected data.

Guiding Principles for Data Sharing and Harmonization: Results of a Systematic Review and Modified Delphi From the Society of Critical Care Medicine Data Science Campaign

OBJECTIVES

This study aimed to establish a set of guiding principles for data sharing and harmonization in critical care, focusing on the use of real-world data (RWD) and real-world evidence (RWE) to improve patient outcomes and research efficacy. The principles were developed through a systematic literature review and a modified Delphi process, with the goal of enhancing data accessibility, standardization, and interoperability across critical care settings.

DATA SOURCES

Data sources included a comprehensive search of peer-reviewed literature, specifically studies related to the use of RWD and RWE in healthcare, guidelines, best practices, and recommendations on data sharing and harmonization. A total of 8150 articles were initially identified through databases such as MEDLINE and Web of Science, with 257 studies meeting inclusion criteria.

STUDY SELECTION

Inclusion criteria focused on publications discussing health-related informatics, recommendations for RWD/RWE usage, data sharing, and harmonization principles. Exclusion criteria ruled out non-human studies, case studies, conference abstracts, and articles published before 2013, as well as those not available in English.

DATA EXTRACTION

From the 257 selected studies, 322 statements were extracted. After removing irrelevant definitions and off-topic content, 232 statements underwent content validation and thematic analysis. These statements were then consolidated into 24 candidate guiding principles after rigorous review and consensus-building among the expert panel.

DATA SYNTHESIS

A three-phase modified Delphi process was employed, involving a conceptualization group, a review group, and a Delphi group. In phase 1, experts identified key themes and search terms for the systematic review. Phase 2 involved validating and refining the prospective guiding principles, while phase 3 employed a Delphi panel to rate principles on acceptability, importance, and feasibility. This process resulted in 24 guiding principles, with high consensus achieved in rounds 2 and 3 on their relevance and applicability.

CONCLUSIONS

The systematic review and Delphi process resulted in 24 guiding principles to improve data sharing and harmonization in critical care. These principles address challenges across the data lifecycle, including generation, storage, access, and usage of RWD and RWE. This framework is designed to promote more effective and equitable data practices, with relevance for the development of artificial intelligence-based decision support tools and clinical research. The principles are intended to guide the responsible use of data science in critical care, with emphasis on ethics and equity, while acknowledging the variability of resources across settings.

Standardized patient profile review using large language models for case adjudication in observational research

Using administrative claims and electronic health records for observational studies is common but challenging due to data limitations. Researchers rely on phenotype algorithms, requiring labor-intensive chart reviews for validation. This study investigates whether case adjudication using the previously introduced Knowledge-Enhanced Electronic Profile Review (KEEPER) system with large language models (LLMs) is feasible and could serve as a viable alternative to manual chart review. The task involves adjudicating cases identified by a phenotype algorithm, with KEEPER extracting predefined findings such as symptoms, comorbidities, and treatments from structured data. LLMs then evaluate KEEPER outputs to determine whether a patient truly qualifies as a case. We tested four LLMs including GPT-4, hosted locally to ensure privacy. Using zero-shot prompting and iterative prompt optimization, we found LLM performance, across ten diseases, varied by prompt and model, with sensitivities from 78 to 98% and specificities from 48 to 98%, indicating promise for automating phenotype evaluation.

Barriers to WHO prequalification of similar biotherapeutic insulin

Objective

To identify the barriers preventing manufacturers of similar biotherapeutic human insulin from submitting their products to the World Health Organization (WHO) for prequalification.

Methods

We used a self-administered questionnaire to collect data from companies producing similar biotherapeutic human insulin. We included questions about the insulin products manufactured, knowledge of WHO prequalification requirements, export of the products and compliance with good manufacturing practices. Companies had the possibility to provide additional relevant information. We sent the questionnaire to 20 manufacturers in total. We evaluated responses and organized the data into themes.

Results

We had a response rate of 55% (11/20 companies). Five broad themes emerged: (i) manufacturers and products; (ii) expressions of interest awareness and participation; (iii) need for technical assistance and training; (iv) market and supply chain challenges; and (v) approval for good manufacturing practices. The most important reasons for manufacturers' lack of response to WHO's expression-of-interest invitation were absence of a mechanism to guarantee return on investment, and perceived complexity of prequalification requirements for insulin-similar biotherapeutic products.

Conclusion

To encourage greater participation in the WHO prequalification programme, international procurement agencies associated with the programme should consider establishing a platform to enter into advance purchasing agreements with manufacturers. In addition, WHO's Local Production and Assistance Unit should provide companies with ongoing technical assistance on the development of their human insulin products and improvement of their production facilities to comply with the WHO requirements for good manufacturing practices.

Use of Omalizumab for Pediatric Asthma After US Food and Drug Administration Expanded Indications

PURPOSE

Research and regulatory approval for pediatric uses of prescription drugs often lag years after adult approvals, during which time substantial off-label use of medications in children can occur. We evaluated whether US Food and Drug Administration (FDA) regulatory actions affected the pediatric use of omalizumab, a biologic drug used to treat asthma.

METHODS

In this serial cross-sectional study, we identified quarterly cohorts of children (0-18 years) with moderate-to-severe asthma within two large national claims databases of those with commercial insurance and Medicaid from 2003 to 2019. Using an interrupted time-series analysis, we fit segmented linear regression models to identify changes in the incidence of omalizumab use in 6-11-year-old children compared with 12-18-year-olds after two time points: (1) 2009Q3 when an FDA advisory committee voted against use for 6-11-year-old children and (2) 2016Q2 when FDA expanded omalizumab's labeling to include 6-11-year-old children.

RESULTS

We identified 9298 new pediatric omalizumab users (84% Medicaid). Among 6-11-year-old children, the incidence of omalizumab use did not change following the FDA's initial review of evidence in 2009 and increased after 2016 Q2 FDA approval for this age group in both Medicaid (58 per 100 000 children with asthma, 95% confidence interval [CI] 27-89, p < 0.001) and commercial insurance (57 per 100 000, 95% CI 21-94, p = 0.003) compared with 12-18-year-old children.

CONCLUSIONS

The use of omalizumab among asthmatic children aged 6-11 years remained steady after FDA advisory committee concerns in 2009 and increased after FDA expanded the indication to include this population in 2016. Additional market incentives may help to ensure the timely generation of evidence and regulatory approval of medications for children.

Effectiveness of Treating Obstructive Sleep Apnea by Surgeries and Continuous Positive Airway Pressure: Evaluation Using Objective Sleep Parameters and Patient-Reported Outcomes

Background/Objectives: Uvulopalatopharyngoplasty (UPPP), palatal plus nasal surgery (PNS), and continuous positive airway pressure (CPAP) are widely implemented treatments for obstructive sleep apnea (OSA). This study aims to explore the long-term effects on objective sleep parameters and patient-reported outcomes (PROs) following different therapeutic interventions for OSA. Methods: Data from patients with moderate-to-severe OSA were retrospectively collected from a medical center and a regional hospital, spanning from December 2011 to August 2018. Objective evaluations included the Apnea-Hypopnea Index (AHI), minimum O2 saturation, and sleep efficiency. The PROs consisted of the Snore Outcomes Survey and Epworth Sleepiness Scale. Using mixed-effects models, we evaluated longitudinal changes in sleep parameters and PROs, accounting for repeated measures and variations within individuals over time. Results: Among 448 patients with moderate-to-severe OSA, follow-up data were collected for 42 patients undergoing UPPP surgery, 171 undergoing PNS, 127 using CPAP, and 108 in the non-treated group. The mean follow-up was 16.7 months (SD = 11.9, range: 1.6-77.3). Significant improvements were observed in AHI, minimum O2 saturation, and hypersomnia immediately following interventions with UPPP, PNS, and CPAP therapy (p < 0.05). Moreover, the analysis revealed no significant rate of change in these parameters over time, suggesting that the benefits of these treatments were sustained in the long term. Furthermore, all interventions exhibited a significant short-term effect on self-reported snoring when compared to the control group, with a p-value of less than 0.001. However, the magnitude of this improvement gradually decreased over time. The snore scores seemed to return to pre-treatment levels among the UPPP, PNS, and CPAP groups after averages of 46.4, 63.5, and 74.4 months, respectively (all p < 0.05). Conclusions: Surgical interventions and CPAP therapy showed potential long-term effectiveness in managing OSA. Snoring symptoms reappeared about 3.9-5.3 years after surgical treatments, which seemed earlier than the average of 6.2 years in patients receiving CPAP and should be considered in patient-participatory decision-making processes.

Overcoming the barriers to better evidence generation from clinical trials

Clinical evidence generation from and for representative populations can be improved through increased research access and ease of trial participation. To improve access and participation, a modern trial infrastructure is needed that broadens research into more routine practice. This commentary highlights current barriers, areas of advancement, and actions needed to enable continued transformation toward a modern trial infrastructure for an improved evidence generation system. The focus of this commentary is on the development of medical products (e.g., drugs, devices, biologics) and infrastructure issues within the United States, with the aim to have broader, multi-national applicability.

Evaluating and Enhancing the Fitness-for-Purpose of Electronic Health Record Data: Qualitative Study on Current Practices and Pathway to an Automated Approach Within the Medical Informatics for Research and Care in University Medicine Consortium

BACKGROUND

Leveraging electronic health record (EHR) data for clinical or research purposes heavily depends on data fitness. However, there is a lack of standardized frameworks to evaluate EHR data suitability, leading to inconsistent quality in data use projects (DUPs). This research focuses on the Medical Informatics for Research and Care in University Medicine (MIRACUM) Data Integration Centers (DICs) and examines empirical practices on assessing and automating the fitness-for-purpose of clinical data in German DIC settings.

OBJECTIVE

The study aims (1) to capture and discuss how MIRACUM DICs evaluate and enhance the fitness-for-purpose of observational health care data and examine the alignment with existing recommendations and (2) to identify the requirements for designing and implementing a computer-assisted solution to evaluate EHR data fitness within MIRACUM DICs.

METHODS

A qualitative approach was followed using an open-ended survey across DICs of 10 German university hospitals affiliated with MIRACUM. Data were analyzed using thematic analysis following an inductive qualitative method.

RESULTS

All 10 MIRACUM DICs participated, with 17 participants revealing various approaches to assessing data fitness, including the 4-eyes principle and data consistency checks such as cross-system data value comparison. Common practices included a DUP-related feedback loop on data fitness and using self-designed dashboards for monitoring. Most experts had a computer science background and a master's degree, suggesting strong technological proficiency but potentially lacking clinical or statistical expertise. Nine key requirements for a computer-assisted solution were identified, including flexibility, understandability, extendibility, and practicability. Participants used heterogeneous data repositories for evaluating data quality criteria and practical strategies to communicate with research and clinical teams.

CONCLUSIONS

The study identifies gaps between current practices in MIRACUM DICs and existing recommendations, offering insights into the complexities of assessing and reporting clinical data fitness. Additionally, a tripartite modular framework for fitness-for-purpose assessment was introduced to streamline the forthcoming implementation. It provides valuable input for developing and integrating an automated solution across multiple locations. This may include statistical comparisons to advanced machine learning algorithms for operationalizing frameworks such as the 3×3 data quality assessment framework. These findings provide foundational evidence for future design and implementation studies to enhance data quality assessments for specific DUPs in observational health care settings.

[Data integration centers-from a concept in the Medical Informatics Initiative to its local implementation in the Network of University Medicine]

As part of the Medical Informatics Initiative (MII), data integration centers (DICs) have been established at 38 university and 3 non-university locations in Germany since 2018. At DICs, research and healthcare data are collected. The DICs represent an important pillar in research and healthcare. They establish the technical, organizational, and (ethical) data protection requirements to enable cross-site research with the available routine clinical data.This article presents the three main pillars of DICs: ethical-legal framework, organization, and technology. The organization of DICs and their organizational embedding and interaction are presented, as well as the technical infrastructure. The services that a DIC provides for its own location and for external researchers are explained, and the role of the DIC as an internal and external interface for strengthening cooperation and collaboration is outlined.Legal conformity, organization, and technology form the basis for the processes and structures of a DIC and are decisive for how it is integrated into the healthcare and research landscape of a location, but also for how it can react to national and European requirements and act and function as an interface to the outside world. In this context and with regard to national developments (e.g., introduction of the electronic patient file-ePA), but also international and European initiatives (e.g., European Health Data Space-EHDS), the DIC will play a central role in the future.

The Importance of Training and Assessing Quality Control Reviewers in Technology-Enabled Abstraction of Real-World Data: A Case Study

Accurate cancer registry data is crucial for understanding cancer prevention and treatment strategies. Proper education and training are key for successful quality control (QC) programs and an evaluation process is needed to assess effectiveness. Syapse developed a rigorous QC training program that includes a peer review process to assess data quality and an interrater review (IRR) program to evaluate the consistency of QC reviewers. In reviewing IRR cases, we found high rates of agreement in various cancer types: colon (97.74%), prostate (97.75%), ovarian (96.31%), lung (98.03%), breast (97.86%), and bladder (97.88%). A peer review experience questionnaire was also administered. Results indicated that the program facilitated the acquisition of new skills. Through the implementation of robust QC training and assessment procedures for technology-enabled data curation, our Oncology Data Specialist (ODS)-certified professionals at Syapse ensure data quality in a real-world evidence (RWE) platform. QC reviewers deserve an extensive investment in training and professional development to uphold data quality and support cancer research efforts.

High-throughput target trial emulation for Alzheimer's disease drug repurposing with real-world data

Target trial emulation is the process of mimicking target randomized trials using real-world data, where effective confounding control for unbiased treatment effect estimation remains a main challenge. Although various approaches have been proposed for this challenge, a systematic evaluation is still lacking. Here we emulated trials for thousands of medications from two large-scale real-world data warehouses, covering over 10 years of clinical records for over 170 million patients, aiming to identify new indications of approved drugs for Alzheimer's disease. We assessed different propensity score models under the inverse probability of treatment weighting framework and suggested a model selection strategy for improved baseline covariate balancing. We also found that the deep learning-based propensity score model did not necessarily outperform logistic regression-based methods in covariate balancing. Finally, we highlighted five top-ranked drugs (pantoprazole, gabapentin, atorvastatin, fluticasone, and omeprazole) originally intended for other indications with potential benefits for Alzheimer's patients.

Traceable Research Data Sharing in a German Medical Data Integration Center With FAIR (Findability, Accessibility, Interoperability, and Reusability)-Geared Provenance Implementation: Proof-of-Concept Study

BACKGROUND

Secondary investigations into digital health records, including electronic patient data from German medical data integration centers (DICs), pave the way for enhanced future patient care. However, only limited information is captured regarding the integrity, traceability, and quality of the (sensitive) data elements. This lack of detail diminishes trust in the validity of the collected data. From a technical standpoint, adhering to the widely accepted FAIR (Findability, Accessibility, Interoperability, and Reusability) principles for data stewardship necessitates enriching data with provenance-related metadata. Provenance offers insights into the readiness for the reuse of a data element and serves as a supplier of data governance.

OBJECTIVE

The primary goal of this study is to augment the reusability of clinical routine data within a medical DIC for secondary utilization in clinical research. Our aim is to establish provenance traces that underpin the status of data integrity, reliability, and consequently, trust in electronic health records, thereby enhancing the accountability of the medical DIC. We present the implementation of a proof-of-concept provenance library integrating international standards as an initial step.

METHODS

We adhered to a customized road map for a provenance framework, and examined the data integration steps across the ETL (extract, transform, and load) phases. Following a maturity model, we derived requirements for a provenance library. Using this research approach, we formulated a provenance model with associated metadata and implemented a proof-of-concept provenance class. Furthermore, we seamlessly incorporated the internationally recognized Word Wide Web Consortium (W3C) provenance standard, aligned the resultant provenance records with the interoperable health care standard Fast Healthcare Interoperability Resources, and presented them in various representation formats. Ultimately, we conducted a thorough assessment of provenance trace measurements.

RESULTS

This study marks the inaugural implementation of integrated provenance traces at the data element level within a German medical DIC. We devised and executed a practical method that synergizes the robustness of quality- and health standard-guided (meta)data management practices. Our measurements indicate commendable pipeline execution times, attaining notable levels of accuracy and reliability in processing clinical routine data, thereby ensuring accountability in the medical DIC. These findings should inspire the development of additional tools aimed at providing evidence-based and reliable electronic health record services for secondary use.

CONCLUSIONS

The research method outlined for the proof-of-concept provenance class has been crafted to promote effective and reliable core data management practices. It aims to enhance biomedical data by imbuing it with meaningful provenance, thereby bolstering the benefits for both research and society. Additionally, it facilitates the streamlined reuse of biomedical data. As a result, the system mitigates risks, as data analysis without knowledge of the origin and quality of all data elements is rendered futile. While the approach was initially developed for the medical DIC use case, these principles can be universally applied throughout the scientific domain.

Ophthalmology at the Forefront of Big Data Integration in Medicine: Insights from the IRIS Registry Database

Ophthalmology stands at the vanguard of incorporating big data into medicine, as exemplified by the integration of The Intelligent Research in Sight (IRIS) Registry. This synergy cultivates patient-centered care, demonstrates real world efficacy and safety data for new therapies, and facilitates comprehensive population health insights. By evaluating the creation and utilization of the world's largest specialty clinical data registry, we underscore the transformative capacity of data-driven medical paradigms, current shortcomings, and future directions. We aim to provide a scaffold for other specialties to adopt big data integration into medicine.

Data Challenges for Externally Controlled Trials: Viewpoint

The preferred evidence of a large randomized controlled trial is difficult to adopt in scenarios, such as rare conditions or clinical subgroups with high unmet needs, and evidence from external sources, including real-world data, is being increasingly considered by decision makers. Real-world data originate from many sources, and identifying suitable real-world data that can be used to contextualize a single-arm trial, as an external control arm, has several challenges. In this viewpoint article, we provide an overview of the technical challenges raised by regulatory and health reimbursement agencies when evaluating comparative efficacy, such as identification, outcome, and time selection challenges. By breaking down these challenges, we provide practical solutions for researchers to consider through the approaches of detailed planning, collection, and record linkage to analyze external data for comparative efficacy.

Transferability of real-world data across borders for regulatory and health technology assessment decision-making

Recently, there has been increased consideration of real-world data (RWD) and real-world evidence (RWE) in regulatory and health technology assessment (HTA) decision-making. Due to challenges in identifying high-quality and relevant RWD sources, researchers and regulatory/HTA bodies may turn to RWD generated in locales outside of the locale of interest (referred to as "transferring RWD"). We therefore performed a review of stakeholder guidance as well as selected case studies to identify themes for researchers to consider when transferring RWD from one jurisdiction to another. Our review highlighted that there is limited consensus on defining decision-grade, transferred RWD; certain stakeholders have issued relevant guidance, but the recommendations are high-level and additional effort is needed to generate comprehensive guidance. Additionally, the case studies revealed that RWD transferability has not been a consistent concern for regulatory/HTA bodies and that more focus has been put on the evaluation of internal validity. To help develop transferability best practices (alongside internal validity best practices), we suggest that researchers address the following considerations in their justification for transferring RWD: treatment pathways, nature of the healthcare system, incidence/prevalence of indication, and patient demographics. We also recommend that RWD transferability should garner more attention as the use of imported RWD could open doors to high-quality data sources and potentially reduce methodological issues that often arise in the use of local RWD; we thus hope this review provides a foundation for further dialogue around the suitability and utility of transferred RWD in the regulatory/HTA decision-making space.

Development and validation of a novel model for characterizing migraine outcomes within real-world data

BACKGROUND

In disease areas with 'soft' outcomes (i.e., the subjective aspects of a medical condition or its management) such as migraine or depression, extraction and validation of real-world evidence (RWE) from electronic health records (EHRs) and other routinely collected data can be challenging due to how the data are collected and recorded. In this study, we aimed to define and validate a scalable framework model to measure outcomes of migraine treatment and prevention by use of artificial intelligence (AI) algorithms within EHR data.

METHODS

Headache specialists defined descriptive features based on routinely collected clinical data. Data elements were weighted to define a 10-point scale encompassing headache severity (1-7 points) and associated features (0-3 points). A test data set was identified, and a reference standard was manually produced by trained annotators. Automation (i.e., AI) was used to extract features from the unstructured data of patient encounters and compared to the reference standard. A threshold of 70% close agreement (within 1 point) between the automated score and the human annotator was considered to be a sufficient extraction accuracy. The accuracy of AI in identifying features used to construct the outcome model was also evaluated and success was defined as achieving an F1 score (i.e., the weighted harmonic mean of the precision and recall) of 80% in identifying encounters.

RESULTS

Using data from 2,006 encounters, 11 features were identified and included in the model; the average F1 scores for automated extraction were 92.0% for AI applied to unstructured data. The outcome model had excellent accuracy in characterizing migraine status with an exact match for 77.2% of encounters and a close match (within 1 point) for 82.2%, compared with manual extraction scores-well above the 70% match threshold set prior to the study.

CONCLUSION

Our findings indicate the feasibility of technology-enabled models for validated determination of soft outcomes such as migraine progression using the data elements typically captured in the real-world clinical setting, providing a scalable approach to credible EHR-based clinical studies.

Considerations for the Use of Machine Learning Extracted Real-World Data to Support Evidence Generation: A Research-Centric Evaluation Framework

A vast amount of real-world data, such as pathology reports and clinical notes, are captured as unstructured text in electronic health records (EHRs). However, this information is both difficult and costly to extract through human abstraction, especially when scaling to large datasets is needed. Fortunately, Natural Language Processing (NLP) and Machine Learning (ML) techniques provide promising solutions for a variety of information extraction tasks such as identifying a group of patients who have a specific diagnosis, share common characteristics, or show progression of a disease. However, using these ML-extracted data for research still introduces unique challenges in assessing validity and generalizability to different cohorts of interest. In order to enable effective and accurate use of ML-extracted real-world data (RWD) to support research and real-world evidence generation, we propose a research-centric evaluation framework for model developers, ML-extracted data users and other RWD stakeholders. This framework covers the fundamentals of evaluating RWD produced using ML methods to maximize the use of EHR data for research purposes.