Transparency in real-world evidence (RWE) studies to build confidence for decision-making: Reporting RWE research in diabetes

Advancing the Use of Longitudinal Electronic Health Records: Tutorial for Uncovering Real-World Evidence in Chronic Disease Outcomes

Managing chronic diseases requires ongoing monitoring of disease activity and therapeutic responses to optimize treatment plans. With the growing availability of disease-modifying therapies, it is crucial to investigate comparative effectiveness and long-term outcomes beyond those available from randomized clinical trials. We introduce a comprehensive pipeline for generating reproducible and generalizable real-world evidence on disease outcomes by leveraging electronic health record data. The pipeline first generates scalable disease outcomes by linking electronic health record data with registry data containing a small sample of labeled outcomes. It then applies causal analysis using these scalable outcomes to evaluate therapies for chronic diseases. The implementation of the pipeline is illustrated in a case study based on multiple sclerosis. Our approach addresses challenges in real-world evidence generation for disease activity of chronic conditions, specifically the lack of direct observations on key outcomes and biases arising from imperfect or incomplete data. We present advanced machine learning techniques such as semisupervised and ensemble methods to impute missing outcome data, further incorporating steps for calibrated causal analyses and bias correction.

Transparency in the secondary use of health data: assessing the status quo of guidance and best practices

We evaluated what guidance exists in the literature to improve the transparency of studies that make secondary use of health data. To find peer-reviewed papers, we searched PubMed and Google Scholar. To find institutional documents, we used our personal expertise to draft a list of health organizations and searched their websites. We quantitatively and qualitatively coded different types of research transparency: registration, methods reporting, results reporting, data sharing and code sharing. We found 56 documents that provide recommendations to improve the transparency of studies making secondary use of health data, mainly in relation to study registration (n = 27) and/or methods reporting (n = 39). Only three documents made recommendations on data sharing or code sharing. Recommendations for study registration and methods reporting mainly came in the form of structured documents like registration templates and reporting guidelines. Aside from the recommendations aimed directly at researchers, we also found recommendations aimed at the wider research community, typically on how to improve research infrastructure. Limitations or challenges of improving transparency were rarely mentioned, highlighting the need for more nuance in providing transparency guidance for studies that make secondary use of health data.

Evaluating and improving real-world evidence with Targeted Learning

BACKGROUND

The Targeted Learning roadmap provides a systematic guide for generating and evaluating real-world evidence (RWE). From a regulatory perspective, RWE arises from diverse sources such as randomized controlled trials that make use of real-world data, observational studies, and other study designs. This paper illustrates a principled approach to assessing the validity and interpretability of RWE.

METHODS

We applied the roadmap to a published observational study of the dose-response association between ritodrine hydrochloride and pulmonary edema among women pregnant with twins in Japan. The goal was to identify barriers to causal effect estimation beyond unmeasured confounding reported by the study's authors, and to explore potential options for overcoming the barriers that robustify results.

RESULTS

Following the roadmap raised issues that led us to formulate alternative causal questions that produced more reliable, interpretable RWE. The process revealed a lack of information in the available data to identify a causal dose-response curve. However, under explicit assumptions the effect of treatment with any amount of ritodrine versus none, albeit a less ambitious parameter, can be estimated from data.

CONCLUSIONS

Before RWE can be used in support of clinical and regulatory decision-making, its quality and reliability must be systematically evaluated. The TL roadmap prescribes how to carry out a thorough, transparent, and realistic assessment of RWE. We recommend this approach be a routine part of any decision-making process.

Emulating Target Trials With Real-World Data to Inform Health Technology Assessment: Findings and Lessons From an Application to Emergency Surgery

OBJECTIVES

International health technology assessment (HTA) agencies recommend that real-world data (RWD) are used in some circumstances to add to the evidence base about the effectiveness and cost-effectiveness of health interventions. The target trial framework applies the design principles of randomized-controlled trials to RWD and can help alleviate inevitable concerns about bias and design flaws with nonrandomized studies. This article aimed to tackle the lack of guidance and exemplar applications on how this methodology can be applied to RWD to inform HTA decision making.

METHODS

We use Hospital Episode Statistics data from England on emergency hospital admissions from 2010 to 2019 to evaluate the cost-effectiveness of emergency surgery for 2 acute gastrointestinal conditions. We draw on the case study to describe the main challenges in applying the target trial framework alongside RWD and provide recommendations for how these can be addressed in practice.

RESULTS

The 4 main challenges when applying the target trial framework to RWD are (1) defining the study population, (2) defining the treatment strategies, (3) establishing time zero (baseline), and (4) adjusting for unmeasured confounding. The recommendations for how to address these challenges, mainly around the incorporation of expert judgment and use of appropriate methods for handling unmeasured confounding, are illustrated within the case study.

CONCLUSIONS

The recommendations outlined in this study could help future studies seeking to inform HTA decision processes. These recommendations can complement checklists for economic evaluations and design tools for estimating treatment effectiveness in nonrandomized studies.

A Road Map for Peer Review of Real-World Evidence Studies on Safety and Effectiveness of Treatments

The growing acceptance of real-world evidence (RWE) in clinical and regulatory decision-making, coupled with increasing availability of health care data and advances in automated analytic approaches, has contributed to a marked expansion of RWE studies of diabetes and other diseases. However, a recent spate of high-profile retractions highlights the need for improvements in the conduct of RWE research as well as in the associated peer review and editorial processes. We review best pharmacoepidemiologic practices and common pitfalls regarding design, measurement, analysis, data validity, appropriateness, and generalizability of RWE studies. To enhance RWE study assessments, we propose that journal editors require 1) study authors to complete RECORD-PE, a reporting guideline for pharmacoepidemiological studies on routinely collected data, 2) availability of predetermined study protocols and analysis plans, 3) inclusion of pharmacoepidemiologists on the peer review team, and 4) provision of detail on data provenance, characterization, and custodianship to facilitate assessment of the data source. We recognize that none of these steps guarantees a high-quality research study. Collectively, however, they permit an informed assessment of whether the study was adequately designed and conducted and whether the data source used was fit for purpose.

Use of Real-World Data in Population Science to Improve the Prevention and Care of Diabetes-Related Outcomes

The past decade of population research for diabetes has seen a dramatic proliferation of the use of real-world data (RWD) and real-world evidence (RWE) generation from non-research settings, including both health and non-health sources, to influence decisions related to optimal diabetes care. A common attribute of these new data is that they were not collected for research purposes yet have the potential to enrich the information around the characteristics of individuals, risk factors, interventions, and health effects. This has expanded the role of subdisciplines like comparative effectiveness research and precision medicine, new quasi-experimental study designs, new research platforms like distributed data networks, and new analytic approaches for clinical prediction of prognosis or treatment response. The result of these developments is a greater potential to progress diabetes treatment and prevention through the increasing range of populations, interventions, outcomes, and settings that can be efficiently examined. However, this proliferation also carries an increased threat of bias and misleading findings. The level of evidence that may be derived from RWD is ultimately a function of the data quality and the rigorous application of study design and analysis. This report reviews the current landscape and applications of RWD in clinical effectiveness and population health research for diabetes and summarizes opportunities and best practices in the conduct, reporting, and dissemination of RWD to optimize its value and limit its drawbacks.

Generate Analysis-Ready Data for Real-world Evidence: Tutorial for Harnessing Electronic Health Records With Advanced Informatic Technologies

Although randomized controlled trials (RCTs) are the gold standard for establishing the efficacy and safety of a medical treatment, real-world evidence (RWE) generated from real-world data has been vital in postapproval monitoring and is being promoted for the regulatory process of experimental therapies. An emerging source of real-world data is electronic health records (EHRs), which contain detailed information on patient care in both structured (eg, diagnosis codes) and unstructured (eg, clinical notes and images) forms. Despite the granularity of the data available in EHRs, the critical variables required to reliably assess the relationship between a treatment and clinical outcome are challenging to extract. To address this fundamental challenge and accelerate the reliable use of EHRs for RWE, we introduce an integrated data curation and modeling pipeline consisting of 4 modules that leverage recent advances in natural language processing, computational phenotyping, and causal modeling techniques with noisy data. Module 1 consists of techniques for data harmonization. We use natural language processing to recognize clinical variables from RCT design documents and map the extracted variables to EHR features with description matching and knowledge networks. Module 2 then develops techniques for cohort construction using advanced phenotyping algorithms to both identify patients with diseases of interest and define the treatment arms. Module 3 introduces methods for variable curation, including a list of existing tools to extract baseline variables from different sources (eg, codified, free text, and medical imaging) and end points of various types (eg, death, binary, temporal, and numerical). Finally, module 4 presents validation and robust modeling methods, and we propose a strategy to create gold-standard labels for EHR variables of interest to validate data curation quality and perform subsequent causal modeling for RWE. In addition to the workflow proposed in our pipeline, we also develop a reporting guideline for RWE that covers the necessary information to facilitate transparent reporting and reproducibility of results. Moreover, our pipeline is highly data driven, enhancing study data with a rich variety of publicly available information and knowledge sources. We also showcase our pipeline and provide guidance on the deployment of relevant tools by revisiting the emulation of the Clinical Outcomes of Surgical Therapy Study Group Trial on laparoscopy-assisted colectomy versus open colectomy in patients with early-stage colon cancer. We also draw on existing literature on EHR emulation of RCTs together with our own studies with the Mass General Brigham EHR.

Glycaemic Control in People with Type 2 Diabetes Treated with Insulin Degludec: A Real-World, Prospective Non-interventional Study-UPDATES Saudi Arabia

INTRODUCTION

Insulin degludec (degludec) has proven benefits in type 2 diabetes (T2D), in terms of improved glycaemic control, low risk of hypoglycaemia, and flexibility in dosing time. This prospective non-interventional UPDATES study aimed to investigate whether results obtained from randomised clinical trials and other real-world studies with degludec are generalisable to patients with T2D in routine clinical practice in Saudi Arabia.

METHODS

Eligible adults (n = 561) with T2D received degludec for 26-34 weeks, at physicians' discretion and in accordance with local routine clinical practice. The primary endpoint was mean change in HbA_1c from baseline to end of study (EOS). Secondary endpoints included mean change from baseline to EOS in fasting plasma glucose (FPG), daily insulin dose and rate of hypoglycaemia.

RESULTS

At baseline, mean age, HbA_1c and FPG were 55.7 years, 9.4% and 185.6 mg/dL, respectively. Mean (standard error [SE]) changes from baseline to EOS (crude analysis) were statistically significant for HbA_1c (- 1.1 [0.08] %-points, 95% CI - 1.29, - 0.98; P < 0.0001), FPG (- 39.1 [3.42] mg/dL, 95% CI - 45.9, - 32.4; P < 0.0001) and total daily insulin dose (+ 4.7 [1.6] units, 95% CI 1.63, 7.86; P = 0.003, insulin-experienced population). In exploratory analysis of patients switching from insulin glargine U100 or U300 to degludec, similar reductions were seen in HbA_1c and FPG. The rate of hypoglycaemia was significantly reduced with degludec versus previous treatment, with no apparent or unexpected safety and tolerability issues. The number of insulin-experienced patients utilising resources associated with severe hypoglycaemia was also reduced. Most patients (95.5%) were willing to continue treatment at EOS, and expressed a preference for degludec over their previous regimen (93.0%).

CONCLUSION

Patients with T2D treated with degludec in routine clinical practice in Saudi Arabia experienced clinically significant improvements in glycaemic control and a lower rate of hypoglycaemia compared with baseline, with no new safety concerns reported.

CLINICAL TRIAL REGISTRATION

NCT03785522.

Real-world data: a brief review of the methods, applications, challenges and opportunities

Background

The increased adoption of the internet, social media, wearable devices, e-health services, and other technology-driven services in medicine and healthcare has led to the rapid generation of various types of digital data, providing a valuable data source beyond the confines of traditional clinical trials, epidemiological studies, and lab-based experiments.

Methods

We provide a brief overview on the type and sources of real-world data and the common models and approaches to utilize and analyze real-world data. We discuss the challenges and opportunities of using real-world data for evidence-based decision making This review does not aim to be comprehensive or cover all aspects of the intriguing topic on RWD (from both the research and practical perspectives) but serves as a primer and provides useful sources for readers who interested in this topic.

Results and Conclusions

Real-world hold great potential for generating real-world evidence for designing and conducting confirmatory trials and answering questions that may not be addressed otherwise. The voluminosity and complexity of real-world data also call for development of more appropriate, sophisticated, and innovative data processing and analysis techniques while maintaining scientific rigor in research findings, and attentions to data ethics to harness the power of real-world data.

Does coprescribing nonsteroidal anti-inflammatory drugs and oral anticoagulants increase the risk of major bleeding, stroke and systemic embolism?

AIMS

To examine the risk of gastrointestinal (GI) bleeding, major bleeding, stroke and systemic embolism associated with prescribing nonsteroidal anti-inflammatory drugs (NSAIDs) to adults receiving oral anticoagulant (OAC) therapy.

METHODS

We conducted a population-based cohort study in adults receiving OAC therapy using linked primary care (Clinical Practice Research Datalink GOLD) and hospital (Hospital Episodes Statistics) electronic health records. We used cause-specific Cox regression models with time-dependent NSAID treatment in a propensity score matched population to estimate the increased risk of GI bleeding, stroke, major bleeding and systemic embolism associated with NSAID use.

RESULTS

The matched cohort contained 3177 patients with OAC therapy alone and 3177 with at least 1 concomitant NSAID prescription. Compared with OAC therapy alone, concomitant prescription of NSAIDs with OACs was associated with increased risk of GI bleeding (hazard ratio [HR] 3.01, 95% confidence interval [CI] 1.63 to 5.55), stroke (HR 2.71, 95% CI 1.48 to 4.96) and major bleeding (HR 2.77, 95% CI 1.84 to 4.19). The association with systemic embolism did not reach statistical significance (HR 3.02, 95% CI 0.82 to 11.07). Sensitivity analyses indicated that the results were robust to changes in exclusion criteria and the choice of potential confounding variables.

CONCLUSION

When OACs are coprescribed with NSAIDs, the risk of adverse bleeding events increases and, simultaneously, the protective effect of OACs to prevent strokes reduces. There is a need for interventions that reduce hazardous prescribing of NSAIDs in people receiving OAC therapy.

Assessing Payers' Preferences for Real-World Evidence in the United States: A Discrete Choice Experiment

OBJECTIVES

To rank the US payers' preferences for attributes of real-world evidence (RWE) studies in the context of chronic disease and to quantify trade-offs among them.

METHODS

We conducted a discrete choice experiment in which 180 employees from payer organizations were tasked to choose between 2 RWE studies assuming they were assessing evidence to inform formulary decisions for chronic disease treatment. Each RWE study was characterized by 7 attributes with 3 levels each: very informative, moderately informative, and not measured. We used a D-optimal main-effects design. Survey data were fitted to a conditional logit model to obtain a relative measure of the ranking of importance for each attribute.

RESULTS

Clinical outcomes were the most preferred attribute. It was 4.68 times as important as productivity outcomes-the least preferred attribute. It was followed by health-related quality of life (2.78), methodologic rigor (2.09), resource utilization (1.71), and external validity (1.56).

CONCLUSIONS

This study provides a quantification of the value payers place on key RWE attributes. Across attributes, payers have higher preferences for clinical and health-related quality of life outcomes than the other attributes. Between attributes' levels, payers prefer high levels of information in clinical outcomes and methodologic rigor but are indifferent in other attributes. Our results bridge the gap between the information that payers seek and the attributes that RWE studies prioritize and effectively guide future research design.

Conducting Real-world Evidence Studies on the Clinical Outcomes of Diabetes Treatments

Real-world evidence (RWE), the understanding of treatment effectiveness in clinical practice generated from longitudinal patient-level data from the routine operation of the healthcare system, is thought to complement evidence on the efficacy of medications from randomized controlled trials (RCTs). RWE studies follow a structured approach. (1) A design layer decides on the study design, which is driven by the study question and refined by a medically informed target population, patient-informed outcomes, and biologically informed effect windows. Imagining the randomized trial we would ideally perform before designing an RWE study in its likeness reduces bias; the new-user active comparator cohort design has proven useful in many RWE studies of diabetes treatments. (2) A measurement layer transforms the longitudinal patient-level data stream into variables that identify the study population, the pre-exposure patient characteristics, the treatment, and the treatment-emergent outcomes. Working with secondary data increases the measurement complexity compared to primary data collection that we find in most RCTs. (3) An analysis layer focuses on the causal treatment effect estimation. Propensity score analyses have gained in popularity to minimize confounding in healthcare database analyses. Well-understood investigator errors, like immortal time bias, adjustment for causal intermediates, or reverse causation, should be avoided. To increase reproducibility of RWE findings, studies require full implementation transparency. This article integrates state-of-the-art knowledge on how to conduct and review RWE studies on diabetes treatments to maximize study validity and ultimately increased confidence in RWE-based decision making.

Insulin glargine 300 units/mL for the treatment of individuals with type 2 diabetes in the real world: A review of the DELIVER programme

Evidence from randomized controlled trials (RCTs) has shown that second-generation basal insulin (BI) analogues, insulin glargine 300 U/mL (Gla-300) and insulin degludec (IDeg), provide similar glycaemic control, with a lower risk of hypoglycaemia compared with the first-generation BI analogue insulin glargine 100 U/mL (Gla-100) in people with type 2 diabetes (T2D). However, the highly selected participants and frequent follow-up of RCTs may not be truly representative of real-life clinical practice. It is important to assess the safety and effectiveness of these second-generation BI analogues in real-life clinical practice settings. The DELIVER programme utilized electronic healthcare records from the United States to compare clinical outcomes in people with T2D who received either Gla-300 or other BI analogues in real-world clinical practice. This review provides a concise overview of the results of the DELIVER studies. Overall, Gla-300 provided similar antihyperglycaemic effectiveness and a lower risk of hypoglycaemia versus the first-generation BI analogues Gla-100 and insulin detemir in people with T2D who had switched BIs. In those who were insulin-naïve, initiation with Gla-300 versus Gla-100 was associated with significantly better antihyperglycaemic effectiveness and similar or lower hypoglycaemic risk. Both glycaemic control and hypoglycaemia risk were also shown to be similar with Gla-300 and IDeg, in people who had switched BIs and in those who were insulin-naïve. In addition, the DELIVER 2 study reported that people with T2D who switched to Gla-300 had reduced healthcare resource utilization, with an overall saving of US$1439 per person per year compared with those who switched to another BI analogue. Overall, the real-world DELIVER programme showed that the glycaemic control with a low risk of hypoglycaemia observed with Gla-300 in RCTs was also seen in standard clinical practice.

Real-world evidence of bariatric surgery and cardiovascular benefits using electronic health records data: A lesson in bias

AIM

To reproduce and correct studies on bariatric surgery and the reduction in major adverse cardiovascular events (MACE) among patients with obesity and type 2 diabetes (T2D).

METHODS

We used electronic healthcare records (EHR) from in and outpatient facilities around the United States to identify a cohort of patients with T2D, aged 18 to 80 years and with a body mass index (BMI) of 30 kg/m² or higher undergoing bariatric surgery. We compared against hip/knee arthroplasty to establish an active comparison group that reduced bias from differential information and confounding. The main outcome was six-point MACE. Pre-exposure characteristics were adjusted in propensity score (PS) models with 1:2 matching plus high-dimensional PS matching.

RESULTS

After a range of exclusions, the final cohort included 344 bariatric surgery patients (65% female; mean age 58 years) and 551 PS-matched patients undergoing arthroplasty (65% female; 59 years). Median follow-up was 2.5 years in both groups. Bariatric surgery patients showed a sustained 20% weight reduction and an HbA1c reduction by 1% point. We found no benefits of bariatric surgery for six-point MACE (HR = 0.99; 95% CI 0.76-1.30). We observed known increases in risks for vitamin B₁₂ deficiency anaemia (HR = 3.06; 1.10-8.49) and cholelithiasis (HR = 1.72; 0.94-3.13).

CONCLUSIONS

This real-world evidence study found reductions in HbA1c and BMI following bariatric surgery similar to trials, and no meaningful cardiovascular benefit compatible with the underpowered trials but in contrast to earlier EHR studies. We showed how information bias typical in EHR analyses and confounding may cause substantial bias.

Rules Based Data Quality Assessment on Claims Database

Data quality problems in coded clinical and administrative data have persisted ever since diagnoses and procedures were first coded and used for healthcare billing. These data are used in clinical decision-making introducing a route for iatrogenesis. As we share data on regional Health Information Exchanges (HIEs) and include them in electronic health records the potential for harm may be increased. To study this problem we applied rules-based data quality checks that have been previously tested on Electronic Health Records (EHR) data on a limited set of aggregated claims data. Medicaid claims data was used exclusively. CMS has clear guidelines for claims submitted for Medicaid patients and penalties are incurred for erroneous claims, which should ensure a high quality data source, however reports of low and varying sensitivity, specificity, positive and negative predictive value of coded diagnoses are common. To identify data quality defects in claims data in a state All Payer Claims Dataset (APCD) we applied and evaluated a recently developed rules-based data quality assessment and monitoring system for Electronic Health Record (EHR) data to test effectiveness in claims data. These rules, that are feasible for “All Payer Claims data” and Medicaid data are identified, applied and the Data Quality issue results are produced.