What Does It Take to Transform Real-World Data Into Real-World Evidence?

Digital Healthcare Innovative Services in Times of Crisis: A Literature Review

Objectives: The transformation of healthcare systems during crises, particularly demonstrated during the COVID-19 pandemic, emphasizes the urgent need for effective research methodologies to evaluate digital healthcare innovations. These methodologies are essential in addressing the rapid shift in healthcare service delivery modalities, responding to unprecedented challenges that have revealed both opportunities and barriers within the digital ecosystem. Methods: For this review, research was carried out on the Medline/PubMed, Scopus, and Google Scholar databases to locate articles published from 2015 to 2024, using the search terms digital health, digital technologies, management implications, and digital platforms. The inclusion criteria referred to studies that were directly related to the topic, available in the English language, and published in peer-reviewed scientific journals. The exclusion parameters were as follows: (a) articles not relevant to the topic as defined in the purpose of the review, (b) systematic reviews and meta-analyses, and (c) articles published in a language other than English. Results: Key findings indicate that, while digital health technologies have the potential to mitigate healthcare disparities, they often exacerbate existing inequities, especially among vulnerable populations lacking consistent access to technology. Furthermore, the shift towards digital platforms has revealed significant gaps in workforce training and support, which are essential for effective implementation. Conclusions: This review underscores the financial implications, with expenditures rising significantly due to the increased use of digital services, reflecting a broader trend noted in studies of related health conditions. Moreover, discussions on public health governance suggest a critical need for democratic frameworks to support such digital transformations effectively.

Real-World Evidence Application in Translational Medicine: Making Use of Prescription Claims to Inform Drug-Drug Interactions of a New Psoriasis Treatment

Patients with psoriasis often take multiple medications due to comorbidities, raising concerns about drug-drug interactions (DDIs) during the development of new medicines. DDI risk assessments of a new small molecule showed risks of CYP3A4 autoinduction and being a sensitive CYP3A4 substrate. We conducted a real-world evidence (RWE) claims analysis to assess the frequency of prescription claims for up to 12 months from the date of the initial psoriasis diagnosis for drugs that may interact with CYP3A4 substrates. We used 2013 to 2018 patient data from the US Merative MarketScan Research Database. Among patients diagnosed with psoriasis, less than 1% had a claim for a moderate/strong inducer, but up to 15% had a claim for moderate/strong inhibitor. Most prescriptions for CYP3A4 inhibitors or inducers included antibiotics and anticonvulsants. While CYP3A4 inducers were rarely used, those treated received more than >90 days treatment. Then, these RWE data were used to inform the early translational medicine strategy for the new investigational drug by strategically integrating DDI evaluations into a first-in-human healthy volunteer trial prior to studies in patients with psoriasis. The resulting DDI substudy showed that the investigational small molecule did not induce midazolam clearance but was sensitive to CYP3A inhibition, leading to the decision to exclude concomitant use of strong CYP3A4 inducers or inhibitors from clinical trials.

Development of a definition for real-world evidence using the Jandhyala method for observing consensus opinion among medical affairs pharmaceutical physicians

OBJECTIVE

Low rates of inclusion of real-world evidence (RWE) during regulation may arise from lack of clarity and consensus on its definition. A conceptually mature definition of RWE may have pragmatic utility, increasing its inclusion during regulation. The aim was to develop a definition of RWE to promote inclusion in regulatory submissions and assess its conceptual maturity.

METHODS

Thirteen medical affairs pharmaceutical physicians completed two qualitative online surveys to generate items needed in a definition of RWE. Items that reached a consensus index of > 50% (CI > = 0.51) were retained in the final definition. The maturity of the definition was assessed using concept analysis.

RESULTS

After attrition, 11 participants completed the study and generated 18 items to be included in a definition of RWE. All items reached the consensus threshold and were included. The definition was conceptually mature on three of the four dimensions: the potential for a consensual definition across stakeholders, a description of its characteristics and clear preconditions and outcomes. Further research is needed to delineate the boundaries of RWE.

CONCLUSIONS

A definition of RWE was generated that may increase its inclusion during medicines regulation, especially with further refinement from regulators and other stakeholders.

Clinical Pharmacology Applications of Real-World Data and Real-World Evidence in Drug Development and Approval-An Industry Perspective

Since the 21st Century Cures Act was signed into law in 2016, real-world data (RWD) and real-world evidence (RWE) have attracted great interest from the healthcare ecosystem globally. The potential and capability of RWD/RWE to inform regulatory decisions and clinical drug development have been extensively reviewed and discussed in the literature. However, a comprehensive review of current applications of RWD/RWE in clinical pharmacology, particularly from an industry perspective, is needed to inspire new insights and identify potential future opportunities for clinical pharmacologists to utilize RWD/RWE to address key drug development questions. In this paper, we review the RWD/RWE applications relevant to clinical pharmacology based on recent publications from member companies in the International Consortium for Innovation and Quality in Pharmaceutical Development (IQ) RWD Working Group, and discuss the future direction of RWE utilization from a clinical pharmacology perspective. A comprehensive review of RWD/RWE use cases is provided and discussed in the following categories of application: drug-drug interaction assessments, dose recommendation for patients with organ impairment, pediatric plan development and study design, model-informed drug development (e.g., disease progression modeling), prognostic and predictive biomarkers/factors identification, regulatory decisions support (e.g., label expansion), and synthetic/external control generation for rare diseases. Additionally, we describe and discuss common sources of RWD to help guide appropriate data selection to address questions pertaining to clinical pharmacology in drug development and regulatory decision making.

A novel method for continuous measurements of clinical practice guideline adherence

Introduction

Clinical practice guidelines (hereafter 'guidelines') are crucial in providing evidence-based recommendations for physicians and multidisciplinary teams to make informed decisions regarding diagnostics and treatment in various diseases, including cancer. While guideline implementation has been shown to reduce (unwanted) variability and improve outcome of care, monitoring of adherence to guidelines remains challenging. Real-world data collected from cancer registries can provide a continuous source for monitoring adherence levels. In this work, we describe a novel structured approach to guideline evaluation using real-world data that enables continuous monitoring. This method was applied to endometrial cancer patients in the Netherlands and implemented through a prototype web-based dashboard that enables interactive usage and supports various analyses.

Method

The guideline under study was parsed into clinical decision trees (CDTs) and an information standard was drawn up. A dataset from the Netherlands Cancer Registry (NCR) was used and data items from both instruments were mapped. By comparing guideline recommendations with real-world data an adherence classification was determined. The developed prototype can be used to identify and prioritize potential topics for guideline updates.

Results

CDTs revealed 68 data items for recording in an information standard. Thirty-two data items from the NCR were mapped onto information standard data items. Four CDTs could sufficiently be populated with NCR data.

Conclusion

The developed methodology can evaluate a guideline to identify potential improvements in recommendations and the success of the implementation strategy. In addition, it is able to identify patient and disease characteristics that influence decision-making in clinical practice. The method supports a cyclical process of developing, implementing and evaluating guidelines and can be scaled to other diseases and settings. It contributes to a learning healthcare cycle that integrates real-world data with external knowledge.

[Evidence-based health care with pharmaceuticals for rare diseases: the role of digitalisation]

Knowledge generation in the field of drug development for people with rare diseases (RDs) faces particular difficulties. This paper will show what improvements are expected from increasing digitalisation from the perspective of three healthcare institutions: the Federal Institute for Drugs and Medical Devices, the Institute for Quality and Efficiency in Health Care and the Federal Joint Committee.First, the potential of digitalisation to increase the efficiency of clinical development and regulatory decision-making through earlier collaboration of all stakeholders is proposed. Subsequently, it is argued that digitalisation should be used to reduce barriers to the implementation of care-associated randomised controlled trials, including those based on registries. High-quality registry studies should not only be started after approval but during the approval process, so that the evidence necessary for therapy decisions is available promptly after approval. Finally, it is stated that improving the evidence base through qualitative improvement of the data sources and their linkages directly benefits patients. Usable evidence that can be generated over a longer period of time - also beyond approval - and contribute to decisions within healthcare system ensures effective drug provision.The institutions agree that high-quality indication registries should be developed as product-independent, standing infrastructures so that high-quality data can be accessed early in the development of medicines for RD.

Artificial Neural Network vs. Pharmacometric Model for Population Prediction of Plasma Concentration in Real-World Data: A Case Study on Valproic Acid

We compared the predictive performance of an artificial neural network to traditional pharmacometric modeling for population prediction of plasma concentrations of valproate in real-world data. We included individuals aged 65 years or older with epilepsy who redeemed their first prescription of valproate after the diagnosis of epilepsy and had at least one valproate plasma concentration measured. A long short-term memory neural network (LSTM) was developed using the training data set to fit the LSTM and the test data set to validate the model. Predictions from the LSTM were compared with those obtained from the population predictions from a pharmacometric model by Birnbaum et al. which had the best predictive performance for population predictions of valproate concentrations in Danish databases. We used the cutoff of ± 20 mg/L of prediction error to define good predictions. A total of 1,252 individuals were included in the study. The LSTM fitted using the training data set had poor predictive performance in the test data set, but better than that of the pharmacometric model. The proportion of individuals with at least one predicted concentration within ± 20 mg/L of observed concentration was largest in case of the LSTM (64.4%, 95% confidence interval (CI): 58.4-70.2%) compared with the pharmacometric model by Birnbaum et al. (49.8%, 95% CI: 47.0-52.6%). LSTM shows better predictive performance to predict valproate plasma concentrations compared with a traditional pharmacometric model in the investigated setting with real-world data in older patients with epilepsy where information on exact timepoints for both dosing and plasma concentration measurement are missing.

The methodology of a "living" COVID-19 registry development in a clinical context

OBJECTIVE

The aim of this study was to describe an innovative methodology of a registry development, constantly updated for the scientific assessment and analysis of the health status of the population with COVID-19.

STUDY DESIGN AND SETTING

A methodological study design to develop a multi-site, Living COVID-19 Registry of COVID-19 patients admitted in Fondazione Don Gnocchi centres started in March 2020.

RESULTS

The integration of the living systematic reviews and focus group methodologies led to a development of a registry which includes 520 fields filled in for 748 COVID-19 patients recruited from 17 Fondazione Don Gnocchi centres. The result is an evidence and experience-based registry, according to the evolution of a new pathology which was not known before outbreak of March 2020 and with the aim of building knowledge to provide a better quality of care for COVID-19 patients.

CONCLUSION

A Living COVID-19 Registry is an open, living and up to date access to large-scale patient-level data sets that could help identifying important factors and modulating variable for recognising risk profiles and predicting treatment success in COVID-19 patients hospitalized. This innovative methodology might be used for other registries, to be sure which the data collected is an appropriate means of accomplishing the scientific objectives planned.

CLINICAL TRIAL REGISTRATION NUMBER

not applicable.

Linking Provider Specialty and Outpatient Diagnoses in Medicare Claims Data: Data Quality Implications

BACKGROUND

With increasing use of real world data in observational health care research, data quality assessment of these data is equally gaining in importance. Electronic health record (EHR) or claims datasets can differ significantly in the spectrum of care covered by the data.

OBJECTIVE

In our study, we link provider specialty with diagnoses (encoded in International Classification of Diseases) with a motivation to characterize data completeness.

METHODS

We develop a set of measures that determine diagnostic span of a specialty (how many distinct diagnosis codes are generated by a specialty) and specialty span of a diagnosis (how many specialties diagnose a given condition). We also analyze ranked lists for both measures. As use case, we apply these measures to outpatient Medicare claims data from 2016 (3.5 billion diagnosis-specialty pairs). We analyze 82 distinct specialties present in Medicare claims (using Medicare list of specialties derived from level III Healthcare Provider Taxonomy Codes).

RESULTS

A typical specialty diagnoses on average 4,046 distinct diagnosis codes. It can range from 33 codes for medical toxicology to 25,475 codes for internal medicine. Specialties with large visit volume tend to have large diagnostic span. Median specialty span of a diagnosis code is 8 specialties with a range from 1 to 82 specialties. In total, 13.5% of all observed diagnoses are generated exclusively by a single specialty. Quantitative cumulative rankings reveal that some diagnosis codes can be dominated by few specialties. Using such diagnoses in cohort or outcome definitions may thus be vulnerable to incomplete specialty coverage of a given dataset.

CONCLUSION

We propose specialty fingerprinting as a method to assess data completeness component of data quality. Datasets covering a full spectrum of care can be used to generate reference benchmark data that can quantify relative importance of a specialty in constructing diagnostic history elements of computable phenotype definitions.

Integrating real-world data and modeling to project changes in femoral neck bone mineral density and fracture risk in premenopausal women

Decline of bone mineral density (BMD) during menopause is related to increased risk of fractures in postmenopausal women, however, this relationship in premenopausal women has not been established. To quantify this relationship, real-world data (RWD) from the National Health and Nutrition Examination Survey (NHANES), and longitudinal data from the elagolix phase III clinical trials were modeled across a wide age range, and covariates were evaluated. The natural changes in femoral neck BMD (FN-BMD) were well-described by a bi-exponential relationship with first-order BMD formation (k₁ ) and resorption (k₂ ) rate constants. Body mass index (BMI) and race (i.e., Black) were significant predictors indicating that patients with high BMI or Black race experience a relatively lower BMD loss. Simulations suggest that untreated premenopausal women with uterine fibroids (UFs) from elagolix phase III clinical trials (median age 43 years [minimum 25-maximum 53]) lose 0.6% FN-BMD each year up to menopausal age. For clinical relevance, the epidemiological FRAX model was informed by the simulation results to predict the 10-year risk of major osteoporotic fracture (MOF). Premenopausal women with UFs, who received placebo only in the elagolix phase III trials, have a projected FN-BMD of 0.975 g/cm² at menopause, associated with a 10-year risk of MOF of 2.3%. Integration of modeling, RWD, and clinical trials data provides a quantitative framework for projecting long-term postmenopausal risk of fractures, based on natural history of BMD changes in premenopausal women. This framework enables quantitative evaluation of the future risk of MOF for women receiving medical therapies (i.e., GnRH modulators) that adversely affect BMD.

Validity of Chronic Venous Disease Diagnoses and Epidemiology Using Validated Electronic Health Records From Primary Care: A Real-World Data Analysis

PURPOSE

The aim of this study was to evaluate the validity of lower limb chronic venous disease (CVD) diagnoses entered in a large electronic health record database in primary care in Catalonia, Spain; to investigate the reliability of these data for research purposes; and to estimate the prevalence and incidence of CVD, chronic venous insufficiency (CVI), and venous leg ulcer (VLU).

DESIGN

Real-world data analysis based on a large electronic health record database in primary care in Catalonia, Spain.

METHODS

We used a primary care research database (Information System for the Development of Research in Primary Care [SIDIAP]), which contains anonymous data on some 5.8 million people from 279 primary care centers, accounting for more than 80% of the Catalonian population and 15% of the Spanish population. We evaluated the validity of the ICD-10 codes for CVD in SIDIAP for 200 adult patients through the responses of 20 primary care physicians to a questionnaire.

FINDINGS

The positive predictive value of CVD in SIDIAP was 89.95% (95% confidence interval [CI] 84.99-93.40). The prevalence rates for CVD, CVI, and VLU were 9.54% (95% CI 9.51-9.56), 3.87%, and 0.33%, respectively. The incidence rates for CVD, CVI, and VLU were 7.91/1,000 person-years (95% CI 7.82-8.00), 3.37/1,000 person-years (95% CI 3.31-3.43), and 0.23/1,000 person-years (95% CI 0.21-0.24), respectively.

CONCLUSIONS

The Catalonian SIDIAP database contains valid CVD diagnoses. The prevalence and incidence rates found using real-world data are low compared with those in the literature, possibly because CVD is an underdiagnosed entity.

CLINICAL RELEVANCE

Real-world data can inform clinicians on lower limb venous health in a population, show changes as individuals age, and reveal aspects where healthcare can be improved.

Demonstrating that Real World Evidence Is Fit-For-Purpose to Support Labeling: Parallels to Patient Reported Outcomes in the Pursuit of Labeling Claims

INTRODUCTION

In December 2021, U.S. Food & Drug Administration (FDA) will issue guidance on the use of real-world evidence (RWE) to support new indications or expanded product labeling. While difficult to foresee what FDA will require, learnings can be gleaned from previous paradigm shifts at FDA, such as for patient reported outcomes (PROs) in 2006-2009.

METHODS

We contrast published requirements for justifying PROs as fit-for-purpose for a specific labeling claim with a potential approach to justify RWE as fit-for-purpose to support expanded labeling or a new indication.

RESULTS

PRO labeling claims require a PRO Evidence Dossier that includes: specific wording of claim, clinical trial hypothesis structure and endpoint model, and justification that the PRO is relevant and meaningful to patients in the target population (content validity) with adequate psychometric properties. FDA's 2018 RWE Framework outlined critical considerations for using RWE to support regulatory decisions, including data quality, relevancy, provenance, and transparency. Strong parallels exist between the evidence required to justify that PROs are fit-for-purpose to support specific labeling claims and evidence to justify RWE as fit-for-purpose for specific research questions and labeling. Early discussion with FDA is encouraged.

CONCLUSION

Drawing on parallels with use of PROs in labeling, RWE for regulatory purposes should be evaluated within the context of specific labeling or indication, specific study design and analysis plans, and the data attributes of data source. Sponsors seeking a new indication or labeling expansion based on RWE should justify that a specific data source and specific study design are fit-for-purpose.

Are Novel, Nonrandomized Analytic Methods Fit for Decision Making? The Need for Prospective, Controlled, and Transparent Validation

Real-world data and patient-level data from completed randomized controlled trials are becoming available for secondary analysis on an unprecedented scale. A range of novel methodologies and study designs have been proposed for their analysis or combination. However, to make novel analytical methods acceptable for regulators and other decision makers will require their testing and validation in broadly the same way one would evaluate a new drug: prospectively, well-controlled, and according to a pre-agreed plan. From a European regulators' perspective, the established methods qualification advice procedure with active participation of patient groups and other decision makers is an efficient and transparent platform for the development and validation of novel study designs.

The present and future scope of real-world evidence research in diabetes: What questions can and cannot be answered and what might be possible in the future?

The last decade has witnessed an exponential growth in the opportunities to collect and link health-related data from multiple resources, including primary care, administrative, and device data. The availability of these "real-world," "big data" has fuelled also an intense methodological research into methods to handle them and extract actionable information. In medicine, the evidence generated from "real-world data" (RWD), which are not purposely collected to answer biomedical questions, is commonly termed "real-world evidence" (RWE). In this review, we focus on RWD and RWE in the area of diabetes research, highlighting their contributions in the last decade; and give some suggestions for future RWE diabetes research, by applying well-established and less-known tools to direct RWE diabetes research towards better personalized approaches to diabetes care. We underline the essential aspects to consider when using RWD and the key features limiting the translational potential of RWD in generating high-quality and applicable RWE. Only if viewed in the context of other study designs and statistical methods, with its pros and cons carefully considered, RWE will exploit its full potential as a complementary or even, in some cases, substitutive source of evidence compared to the expensive evidence obtained from randomized controlled trials.