Replacing RCTs with real world data for regulatory decision making: a self-fulfilling prophecy?

Facilitating clinical trials in hip fracture in the UK : the role and potential of the National Hip Fracture Database and routinely collected data

Aims

The aim of this study was to evaluate the suitability, against an accepted international standard, of a linked hip fracture registry and routinely collected administrative dataset in England to embed and deliver randomized controlled trials (RCTs).

Methods

First, a bespoke cohort of individuals sustaining hip fractures between 2011 and 2016 was generated from the National Hip Fracture Database (NHFD) and linked to individual Hospital Episode Statistics (HES) records and mortality data. Second, in order to explore the availability and distribution of outcomes available in linked HES-Office of National Statistics (ONS) data, a more contemporary cohort with incident hip fracture was identified within HES between January 2014 and December 2018. Distributions of the outcomes within the HES-ONS dataset were reported using standard statistical summaries; descriptive characteristics of the NHFD and linked HES-ONS dataset were reported in line with the Clinical Trials Transformation Initiative recommendations for registry-enabled trials.

Results

Case ascertainment of the NHFD likely exceeds 94%. The assessment of the robustness, relevance, and reliability of the datasets was favourable. Outcomes from the HES-ONS dataset were concordant with other contemporaneous prospective cohort studies with bespoke data collection frameworks.

Conclusion

Our findings support the feasibility of the NHFD and HES-ONS to support a registry-embedded, data-enabled RCT.

Review of Challenges in Performing Real-World Evidence Studies for Nonprescription Products

In recent years, regulatory authorities have signaled a willingness to consider real-world evidence (RWE) data to support applications for new claims and indications for pharmaceuticals. Historically, RWE studies have been the domain of prescription drugs, driven by the fact that clinical data on patients are routinely captured in medical records, claims databases, registries, etc. However, RWE reports of nonprescription drugs and supplements are relatively sparse due to methodological gaps in this area. The objective of this narrative review is to identify which RWE methodologies have been used to study nonprescription products. A total of 49 articles were included based on literature searches. Label comprehension studies, used to support prescription-to-nonprescription switches, are useful in determining how nonprescription products will be used; however, they provide no actual clinical data. The most common RWE studies of nonprescription products were cross-sectional surveys, which investigated a broad range of indications and were conducted in an array of settings, including online, by phone, point-of-sale (pharmacy), outpatient clinics, and shopping malls. However, while this type of study is effective for identifying use patterns and attitudes in the general population, recall bias limits the ability to collect safety and effectiveness data. Studies of electronic medical records and claims databases are hampered by incomplete or absent capturing of data on nonprescription products. As a result, most RWE studies to date have provided limited useful information. Although case reports and expert opinion should not be discounted, in the absence of other information they provide few actual data. Novel approaches using smartphone apps and artificial intelligence may provide new opportunities to collect RWE for nonprescription products, but these areas of research are in their infancy. Overall, there is a need to develop standards for execution of RWE studies of nonprescription products in terms of endpoints, study design, and study quality.

Methodological expectations for demonstration of health product effectiveness by observational studies

The issue of assessing the effectiveness of health technologies (drugs, devices, etc.) through observational studies is becoming increasingly important as registration and market access agencies consider them in their evaluation process. In this context, observational studies must be able to provide real demonstrations of a level of reliability comparable to those produced by the conventional randomized controlled trial (RCT) approach. The objective of the roundtable was to establish the acceptability criteria for an observational study (non-randomized, non-interventional study) to be able to provide these demonstrations, and possibly serve as a confirmatory study for registration and market access authorities, the construction of therapeutic strategies or the development of recommendations. In order to do this, the study must be a real confirmatory study respecting the hypothetical-deductive approach and guaranteeing the absence of HARKing and p-hacking by attesting to the establishment of a protocol and a statistical analysis plan, recorded before any inferential analysis. It must also be part of a formalized approach to causal inference and demonstrate that it correctly identifies the causal estimand sought. The study should ensure that there is no residual confusion bias by taking into account all confounding factors affecting the comparison, which should be determined by a formal approach (such as a graphical causality approach, DAGs). Residual confusion bias diagnoses by forgery and nullification analysis should be non-existent. The study shall be at low risk of bias, in particular selection bias, among others by using a target test emulation design. Overall type I error risk should be strictly controlled. The absence of selective publication of results and selection bias should be ensured.

Decades in the Making: The Evolution of Digital Health Research Infrastructure Through Synthetic Data, Common Data Models, and Federated Learning

Traditionally, medical research is based on randomized controlled trials (RCTs) for interventions such as drugs and operative procedures. However, increasingly, there is a need for health research to evolve. RCTs are expensive to run, are generally formulated with a single research question in mind, and analyze a limited dataset for a restricted period. Progressively, health decision makers are focusing on real-world data (RWD) to deliver large-scale longitudinal insights that are actionable. RWD are collected as part of routine care in real time using digital health infrastructure. For example, understanding the effectiveness of an intervention could be enhanced by combining evidence from RCTs with RWD, providing insights into long-term outcomes in real-life situations. Clinicians and researchers struggle in the digital era to harness RWD for digital health research in an efficient and ethically and morally appropriate manner. This struggle encompasses challenges such as ensuring data quality, integrating diverse sources, establishing governance policies, ensuring regulatory compliance, developing analytical capabilities, and translating insights into actionable strategies. The same way that drug trials require infrastructure to support their conduct, digital health also necessitates new and disruptive research data infrastructure. Novel methods such as common data models, federated learning, and synthetic data generation are emerging to enhance the utility of research using RWD, which are often siloed across health systems. A continued focus on data privacy and ethical compliance remains. The past 25 years have seen a notable shift from an emphasis on RCTs as the only source of practice-guiding clinical evidence to the inclusion of modern-day methods harnessing RWD. This paper describes the evolution of synthetic data, common data models, and federated learning supported by strong cross-sector collaboration to support digital health research. Lessons learned are offered as a model for other jurisdictions with similar RWD infrastructure requirements.

Evaluating the Prognostic and Clinical Validity of the Fall Risk Score Derived From an AI-Based mHealth App for Fall Prevention: Retrospective Real-World Data Analysis

Background

Falls pose a significant public health concern, with increasing occurrence due to the aging population, and they are associated with high mortality rates and risks such as multimorbidity and frailty. Falls not only lead to physical injuries but also have detrimental psychological and social consequences, negatively impacting quality of life. Identifying individuals at high risk for falls is crucial, particularly for those aged ≥60 years and living in residential care settings; current professional guidelines favor personalized, multifactorial fall risk assessment approaches for effective fall prevention.

Objective

This study aimed to explore the prognostic validity of the Fall Risk Score (FRS), a multifactorial-based metric to assess fall risk (using longitudinal real-world data), and establish the clinical relevance of the FRS by identifying threshold values and the minimum clinically important differences.

Methods

This retrospective cohort study involved 617 older adults (857 observations: 615 of women, 242 of men; mean age 83.3, SD 8.7 years; mean gait speed 0.49, SD 0.19 m/s; 622 using walking aids) residing in German residential care facilities and used the LINDERA mobile health app for fall risk assessment. The study focused on the association between FRS at the initial assessment (T1) and the normalized number of falls at follow-up (T2). A quadratic regression model and Spearman correlation analysis were utilized to analyze the data, supported by descriptive statistics and subgroup analyses.

Results

The quadratic model exhibited the lowest root mean square error (0.015), and Spearman correlation analysis revealed that a higher FRS at T1 was linked to an increased number of falls at T2 (ρ=0.960, P<.001). Subgroups revealed significant strong correlations between FRS at T1 and falls at T2, particularly for older adults with slower gait speeds (ρ=0.954, P<.001) and those using walking aids (ρ=0.955, P<.001). Threshold values revealed that an FRS of 45%, 32%, and 24% corresponded to the expectation of a fall within 6, 12, and 24 months, respectively. Distribution-based minimum clinically important difference values were established, providing ranges for small, medium, and large effect sizes for FRS changes.

Conclusions

The FRS exhibits good prognostic validity for predicting future falls, particularly in specific subgroups. The findings support a stratified fall risk assessment approach and emphasize the significance of early and personalized intervention. This study contributes to the knowledge base on fall risk, despite limitations such as demographic focus and potential assessment interval variability.

Factors affecting the feasibility of post-authorisation RCTs for conditionally authorised anticancer medicines: a multistakeholder perspective from a qualitative focus group study

OBJECTIVE

The collection of comprehensive data from post-authorisation trials for conditionally authorised anticancer medicines is frequently delayed. This raises questions about the feasibility of post-authorisation randomised controlled trials (RCTs) that aim to address remaining uncertainties. Therefore, this study explored factors that facilitate or impede the feasibility of post-authorisation RCTs from the perspective of stakeholders directly involved in the design, medical-ethical approval, and conduct of these RCTs.

DESIGN

We conducted four qualitative focus groups (FGs).

SETTING

FG discussions focused on the oncology setting in European context.

PARTICIPANTS

Twenty-eight European patients, physicians, medical ethicists and pharmaceutical industry representatives participated in the FGs.

INTERVENTION

Respondents were informed about the topic and the purpose of the FGs before and at the start of FG discussions. An FG script was used to guide the discussion, which was informed by 14 semi-structured interviews with various stakeholders.

RESULTS

We identified factors with the potential to impact feasibility related to trial design, trial conduct, factors external to a trial and post-authorisation interaction with regulators. Factors that may be particularly relevant for the post-authorisation setting include the choice of relevant endpoints and the inclusion of a fair comparator (trial design), strategies to increase patients' and physicians' willingness to participate (trial conduct), and external factors relating to a medicine's commercial availability, the presence of competing medicines and trials and the perceptions about clinical equipoise. Post-authorisation interaction with regulators about how to obtain comprehensive data was deemed necessary in cases where a post-authorisation RCT seems infeasible.

CONCLUSIONS

Based on the identified factors, our findings suggest that patient recruitment and retention could be assessed more in-depth during regulatory feasibility assessments at the time of granting conditional marketing authorisation and that sponsors and regulators should better inform patients and physicians about the remaining uncertainties for conditionally authorised medicines and the necessity for post-authorisation RCTs. By enhancing the evaluation of trial feasibility, timely completion of post-authorisation RCTs may be facilitated to resolve the remaining uncertainties within a reasonable timeframe.

Aversion to pragmatic randomised controlled trials: three survey experiments with clinicians and laypeople in the USA

OBJECTIVES

Pragmatic randomised controlled trials (pRCTs) are essential for determining the real-world safety and effectiveness of healthcare interventions. However, both laypeople and clinicians often demonstrate experiment aversion: preferring to implement either of two interventions for everyone rather than comparing them to determine which is best. We studied whether clinician and layperson views of pRCTs for COVID-19, as well as non-COVID-19, interventions became more positive during the pandemic, which increased both the urgency and public discussion of pRCTs.

DESIGN

Randomised survey experiments.

SETTING

Geisinger, a network of hospitals and clinics in central and northeastern Pennsylvania, USA; Amazon Mechanical Turk, a research participant platform used to recruit online participants residing across the USA. Data were collected between August 2020 and February 2021.

PARTICIPANTS

2149 clinicians (the types of people who conduct or make decisions about conducting pRCTs) and 2909 laypeople (the types of people who are included in pRCTs as patients). The clinician sample was primarily female (81%), comprised doctors (15%), physician assistants (9%), registered nurses (54%) and other medical professionals, including other nurses, genetic counsellors and medical students (23%), and the majority of clinicians (62%) had more than 10 years of experience. The layperson sample ranges in age from 18 to 88 years old (mean=38, SD=13) and the majority were white (75%) and female (56%).

OUTCOME MEASURES

Participants read vignettes in which a hypothetical decision-maker who sought to improve health could choose to implement intervention A for all, implement intervention B for all, or experimentally compare A and B and implement the superior intervention. Participants rated and ranked the appropriateness of each decision. Experiment aversion was defined as the degree to which a participant rated the experiment below their lowest-rated intervention.

RESULTS

In a survey of laypeople administered during the pandemic, we found significant aversion to experiments involving catheterisation checklists and hypertension drugs unrelated to the treatment of COVID-19 (Cohen's d=0.25-0.46, p<0.001). Similarly, among both laypeople and clinicians, we found significant aversion to most (comparing different checklist, proning and mask protocols; Cohen's d=0.17-0.56, p<0.001) but not all (comparing school reopening protocols; Cohen's d=0.03, p=0.64) non-pharmaceutical COVID-19 experiments. Interestingly, we found the lowest experiment aversion to pharmaceutical COVID-19 experiments (comparing new drugs and new vaccine protocols for treating the novel coronavirus; Cohen's d=0.04-0.12, p=0.12-0.55). Across all vignettes and samples, 28%-57% of participants expressed experiment aversion, whereas only 6%-35% expressed experiment appreciation by rating the trial higher than their highest-rated intervention.

CONCLUSIONS

Advancing evidence-based medicine through pRCTs will require anticipating and addressing experiment aversion among patients and healthcare professionals.

STUDY REGISTRATION

http://osf.io/6p5c7/.

[Design of a clinical study to demonstrate the therapeutic benefit of another wound treatment product]

BACKGROUND

Due to new legal requirements, a patient-relevant benefit for other wound treatment products (sPW) must be demonstrated through clinical studies if reimbursement at the expense of the statutory health insurance is sought in the non-inpatient sector.

METHOD

An interdisciplinary group with expertise in clinical wound care has developed general recommendations for the design of suitable studies. In addition to regulatory documents, previous studies that have already been recognized as proof of benefit in other areas served as a basis.

RESULTS

Randomized controlled trials that cover at least the most common types of chronic wounds (arterial, venous, diabetic or pressure sore) are recommended as the best method for gathering evidence. Despite the heterogeneous etiology of chronic wounds, the results should also be transferable to other wound types. The test intervention does not usually consist of the sPW alone, but of a combined wound treatment that follows a treatment plan that is as clearly defined as possible. In the comparison group, all wound treatment options (besides the sPW) must also be available and used according to a similar predefined treatment plan. Depending on the intended purpose and treatment goal, complete wound closure should, if possible, be recorded as the cardinal - although not always as the primary - endpoint. In justified cases, e.g. in the case of intermediate use as part of phase-appropriate wound therapy for chronic wounds, a significant reduction in the wound area can also be considered for benefit assessment. Quality of life (e.g. pain) can also justify a benefit and can therefore be recorded as a primary outcome parameter in clinical trials. The duration of the clinical trial should be adapted to the central endpoints, the medical or nursing goal of wound care and the intended purpose of the sPW. A benefit does not always arise from microbiological, physiological, laboratory or histological parameters or imaging findings.

Can requests for real-world evidence by the French HTA body be planned? An exhaustive retrospective case-control study of medicinal products appraisals from 2016 to 2021

OBJECTIVES

In France, decisions for pricing and reimbursement for medicinal products are based on appraisals performed by the National authority for health (Haute Autorité de Santé (HAS)). During the appraisal process, additional real-world evidence can be requested as "Post-Registration Studies" (PRS) when there are uncertainties in evidence that could be resolved by additional data collection. To facilitate PRS planning, a retrospective exploratory analysis was conducted to identify the characteristics of medicinal products associated with a PRS request.

METHODS

This analysis encompassed all appraisals finalized between January 1, 2016 and December 31, 2021 and compared products for which the appraisal led to a PRS request with those that did not.

RESULTS

Six hundred positive opinions for reimbursement were identified, with a PRS request present in 17 percent (n = 103) of cases. The independent characteristics associated with a PRS request were a mild or moderate clinical benefit score, a major to moderate or minor clinical added value score, previous availability under an early access program, and certain therapeutic areas (neurology, pulmonology, and endocrinology). These findings suggest two different profiles of PRS requests: (i) products for which there is uncertainty in the size of the clinical benefit and (ii) innovative products for which a substantial benefit is expected but uncertainties persist.

CONCLUSIONS

These results will assist health technology developers to better anticipate data generation to promptly address uncertainties identified by HAS. It may also help HAS and other assessment agencies to work together to improve postlaunch evidence generation according to the characteristics of the medicinal products.

Do efficacy results obtained from randomized controlled trials translate to effectiveness data from observational studies for relapsing-remitting multiple sclerosis?

BACKGROUND

Randomized controlled trials are considered the gold standard in regulatory decision making, as observational studies are known to have important methodological limitations. However, real-world evidence may be helpful in specific situations. This review investigates how the effect estimates obtained from randomized controlled trials compare to those obtained from observational studies, using drug therapy for relapsing-remitting multiple sclerosis as an example.

STUDY DESIGN AND SETTING

A systematic review of randomized controlled trials and observational studies was conducted. The primary outcome was the annualized relapse rate. Using (network) meta-analysis together with posterior predictive distributions, the drug-specific rate ratios from the network of randomized controlled trials were compared with those from the network of observational studies.

RESULTS

Effect estimates from 26 observational studies showed greater magnitudes and were less precise compared to estimates obtained from 21 randomized controlled trials. Twenty of the 28 treatment comparisons between designs had similar rate ratios. Seven inconsistencies in observed rate ratios could be attributed to two specific disease-modifying therapies.

CONCLUSION

In this case study, estimates from observational studies predominantly agreed with estimates from randomized controlled trials given their posterior predictive distributions. Multiple observational studies together may therefore supplement additional pivotal randomized controlled trials in relapsing-remitting multiple sclerosis, for instance facilitating the extrapolation of trial results to the broader patient population.

R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 14

In this latest update we highlight: a publication from the US FDA regarding the definitions of real-world data (RWD) and real-world evidence (RWE); a publication from academic researchers on a demonstration project for target trial emulation; a publication from the National Institute of Health and Care Excellence (NICE) on the 1 year anniversary of their RWE framework; and a publication from NICE and Flatiron Health on the utility of US RWD for initial UK health technology assessment decision making.

The role of quantitative bias analysis for nonrandomized comparisons in health technology assessment: recommendations from an expert workshop

The use of treatment effects derived from nonrandomized studies (NRS) in health technology assessment (HTA) is growing. NRS carry an inherently greater risk of bias than randomized controlled trials (RCTs). Although bias can be mitigated to some extent through appropriate approaches to study design and analysis, concerns around data availability and quality and the absence of randomization mean residual biases typically render the interpretation of NRS challenging. Quantitative bias analysis (QBA) methods are a range of methods that use additional, typically external, data to understand the potential impact that unmeasured confounding and other biases including selection bias and time biases can have on the results (i.e., treatment effects) from an NRS. QBA has the potential to support HTA bodies in using NRS to support decision-making by quantifying the magnitude, direction, and uncertainty of biases. However, there are a number of key aspects of the use of QBA in HTA which have received limited discussion. This paper presents recommendations for the use of QBA in HTA developed using a multi-stakeholder workshop of experts in HTA with a focus on QBA for unmeasured confounding.

What is the role of randomised trials in implementation science?

BACKGROUND

There is a consistent demand for implementation science to inform global efforts to close the gap between evidence and practice. Key evaluation questions for any given implementation strategy concern the assessment and understanding of effects. Randomised trials are generally accepted as offering the most trustworthy design for establishing effectiveness but may be underused in implementation science.

MAIN BODY

There is a continuing debate about the primacy of the place of randomised trials in evaluating implementation strategies, especially given the evolution of more rigorous quasi-experimental designs. Further critiques of trials for implementation science highlight that they cannot provide 'real world' evidence, address urgent and important questions, explain complex interventions nor understand contextual influences. We respond to these critiques of trials and highlight opportunities to enhance their timeliness and relevance through innovative designs, embedding within large-scale improvement programmes and harnessing routine data. Our suggestions for optimising the conditions for randomised trials of implementation strategies include strengthening partnerships with policy-makers and clinical leaders to realise the long-term value of rigorous evaluation and accelerating ethical approvals and decluttering governance procedures for lower risk studies.

CONCLUSION

Policy-makers and researchers should avoid prematurely discarding trial designs when evaluating implementation strategies and work to enhance the conditions for their conduct.

Gaps in the evidence underpinning high-risk medical devices in Europe at market entry, and potential solutions

AIM

To determine the level of evidence for innovative high-risk medical devices at market entry.

METHODS

We reviewed all Belgian healthcare payer (RIZIV-INAMI) assessor reports on novel implants or invasive medical devices (n = 18, Class IIb-III) available between 2018 to mid-2019 on applications submitted for inclusion on their reimbursement list. We also conducted a review of the literature on evidence gaps and an analysis of relevant legal and ethical frameworks within the European context.

FINDINGS

Conformity assessment of medical devices is based on performance, safety, and an acceptable risk-benefit balance. Information submitted for obtaining CE marking is confidential and legally protected, limiting access to clinical evidence. Seven out of the 18 RIZIV-INAMI assessor reports (39%) included a randomized controlled trial (RCT) using the novel device, whilst 2 applications (11%) referred to an RCT that used a different device. The population included was inappropriate or unclear for 3 devices (17%). Only half of the applications presented evidence on quality of life or functioning and 2 (11%) presented overall survival data. Four applications (22%) included no data beyond twelve months. The findings from the literature demonstrated similar problems with the study design and the clinical evidence.

DISCUSSION AND CONCLUSIONS

CE marking does not indicate that a device is effective, only that it complies with the law. The lack of transparency hampers evidence-based decision making. Despite greater emphasis on clinical benefit for the patient, the provisions of the European Medical Device Regulation (MDR) are not yet fully aligned with international ethical standards for clinical research. The MDR fails to address key issues, such as the lack of access to data submitted for CE marking and a failure to require evidence of clinical effectiveness. Indeed, a first report shows no improvement in the clinical evidence for implantable devices generated under the MDR. Thus, patients may continue to be exposed to ineffective or unsafe novel devices. The Health Technology Assessment Regulation plans for Joint Scientific Consultations for specific high-risk devices before companies begin their pivotal clinical investigations. The demanded comparative evidence should facilitate payer decisions. Nevertheless, there is also a need for legislation requiring comparative RCTs assessing patient-relevant outcomes for high-risk devices to ensure implementation, including development and implementation of common specifications for study designs.

R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 12

In this latest update we highlight the final results from the RCT-DUPLICATE initiative, the publication of guidance from Haute Autorité de Santé (HAS), the joint viewpoint from the Institute for Quality and Efficiency in HealthCare (IQWIG) and the Belgian HealthCare Knowledge Center, and a position from the European Organization for Research and Treatment of Cancer (EORTC). Finally, we discuss how the NICE RWE framework has been implemented to allow consideration of RWE external control arms.

[Genetic tumour diagnostics and personalised medicine from a systems perspective]

High-quality studies are necessary and feasible in personalised medicine in order to evaluate the benefits across the entire treatment chain of biomarker tests and resulting treatments in regard to patient-relevant endpoints. With the introduction of genome sequencing in oncology, a considerable number of new treatment concepts with mostly low-quality evidence can be expected. High quality requirements, interdisciplinary cooperation structures, knowledge-generating care and the connection of patient care at the expense of the statutory health insurance funds, with research at the expense of the manufacturers or public funding, are necessary.