Using Real-World Data in Health Technology Assessment (HTA) Practice: A Comparative Study of Five HTA Agencies

Acceptability of Using Real-World Data to Estimate Relative Treatment Effects in Health Technology Assessments: Barriers and Future Steps

OBJECTIVES

Evidence about the comparative effects of new treatments is typically collected in randomized controlled trials (RCTs). In some instances, RCTs are not possible, or their value is limited by an inability to capture treatment effects over the longer term or in all relevant population subgroups. In these cases, nonrandomized studies (NRS) using real-world data (RWD) are increasingly used to complement trial evidence on treatment effects for health technology assessment (HTA). However, there have been concerns over a lack of acceptability of this evidence by HTA agencies. This article aims to identify the barriers to the acceptance of NRS and steps that may facilitate increases in the acceptability of NRS in the future.

METHODS

Opinions of the authorship team based on their experience in real-world evidence research in academic, HTA, and industry settings, supported by a critical assessment of existing studies.

RESULTS

Barriers were identified that are applicable to key stakeholder groups, including HTA agencies (eg, the lack of comprehensive methodological guidelines for using RWD), evidence generators (eg, avoidable deviations from best practices), and external stakeholders (eg, data controllers providing timely access to high-quality RWD). Future steps that may facilitate future acceptability of NRS include improvements in the quality, integration, and accessibility of RWD, wider use of demonstration projects to highlight the value and applicability of nonrandomized designs, living, and more detailed HTA guidelines, and improvements in HTA infrastructure relating to RWD.

CONCLUSION

NRS can represent a crucial source of evidence on treatment effects for use in HTA when RCT evidence is limited.

Real-world evidence in the reassessment of oncology therapies: payer perceptions from five countries

Aim: This study explored the perceived value of real-world evidence (RWE) in the reassessment of oncology therapies by collecting the perspectives of health technology assessment/payer decision-makers. Materials & methods: A web-based survey was conducted using the Market Access Transformation Rapid Payer Response online portal. 30 participants from France, Germany, Spain, the UK and the USA were recruited based on their expertise. Results: Participants agreed that the most common uses of RWE are to confirm efficacy and safety results from randomized controlled trials and to reevaluate the projected utilization of an oncology therapy. We found variability in other reported uses of RWE. Conclusion: The organizations developing RWE should ensure that their plans recognize the heterogeneity in payer perceptions.

Real-World Evidence of Ribociclib Plus Aromatase Inhibitors as First-Line Treatment in Advanced Breast Cancer: The BrasiLEEira Study

PURPOSE

Cyclin inhibitors plus endocrine therapy represent the reference standard for hormone receptor-positive (HR+)/human epidermal growth factor receptor 2-negative (HER2-) locally advanced or metastatic breast cancer (ABC). Efficacy results on hard end points such as overall survival come from well-designed randomized clinical trials (RCTs). However, a limitation of RCTs is the low external results validity, and their extrapolation to a broader population may not be appropriate. Real-world studies can overcome these limitations, also increasing the reliability of RCTs.

MATERIALS AND METHODS

The BrasiLEEira was an observational, longitudinal, retrospective, multicenter study to evaluate the effectiveness and safety of ribociclib plus nonsteroidal aromatase inhibitors in Brazilian women age 18 years or older with HR+/HER2- ABC. The study was approved by the institutional review boards of all 11 hospitals. Data were collected anonymously from medical records using an electronic case report form designed by an independent academic research organization, which conducted the study considering all recommendations of international guidelines. The primary end point was 1-year progression-free survival (PFS) rate. Secondary end points included mortality, dose reduction, and safety.

RESULTS

The mean age of 76 patients was 57 years, and 28.9% were Black/Brown. The most prevalent comorbidity was arterial hypertension (34.7%). About 26.0% had endocrine-resistant disease, and 54.1% had more than three metastatic sites. The PFS rate was 77.6%. Three patients died (3.9%). Dose reductions occurred in 37.7% of patients. The most common adverse event was neutropenia (68.4%).

CONCLUSION

The high-quality evidence from the BrasiLEEira study corroborates the RCTs' findings, expanding its validity to a broader spectrum and underrepresented population who may benefit from ribociclib treatment.

Generating and using real-world data: A worthwhile uphill battle

The precision oncology paradigm challenges the feasibility and data generalizability of traditional clinical trials. Consequently, an unmet need exists for practical approaches to test many subgroups, evaluate real-world drug value, and gather comprehensive, accessible datasets to validate novel biomarkers. Real-world data (RWD) are increasingly recognized to have the potential to fill this gap in research methodology. Established applications of RWD include informing disease epidemiology, pharmacovigilance, and healthcare quality assessment. Currently, concerns regarding RWD quality and comprehensiveness, privacy, and biases hamper their broader application. Nonetheless, RWD may play a pivotal role in supplementing clinical trials, enabling conditional reimbursement and accelerated drug access, and innovating trial conduct. Moreover, purpose-built RWD repositories may support the extension or refinement of drug indications and facilitate the discovery and validation of new biomarkers. This perspective explores the potential of leveraging RWD to advance oncology, highlights its benefits and challenges, and suggests a path forward in this evolving field.

Cross-sectional analysis of use of real-world data in single technology appraisals of oncological medicine by the National Institute for Health and Care Excellence in 2011-2021

OBJECTIVES

This study aims to identify how real-world data (RWD) have been used in single technology appraisals (STAs) of cancer drugs by the National Institute for Health and Care Excellence (NICE).

DESIGN

Cross-sectional study of NICE technology appraisals of cancer drugs for which guidance was issued between January 2011 and December 2021 (n=229). The appraisals were reviewed following a published protocol to extract the data about the use of RWD. The use of RWD was analysed by reviewing the specific ways in which RWD were used and by identifying different patterns of use.

PRIMARY OUTCOME MEASURE

The number of appraisals where RWD are used in the economic modelling.

RESULTS

Most appraisals used RWD in their economic models. The parametric use of RWD was commonly made in the economic models (76% of the included appraisals), whereas non-parametric use was less common (41%). Despite widespread use of RWD, there was no dominant pattern of use. Three sources of RWD (registries, administrative data, chart reviews) were found across the three important parts of the economic model (choice of comparators, overall survival and volume of treatment).

CONCLUSIONS

NICE has had a long-standing interest in the use of RWD in STAs. A systematic review of oncology appraisals suggests that RWD have been widely used in diverse parts of the economic models. Between 2011 and 2021, parametric use was more commonly found in economic models than non-parametric use. Nonetheless, there was no clear pattern in the way these data were used. As each appraisal involves a different decision problem and the ability of RWD to provide the information required for the economic modelling varies, appraisals will continue to differ with respect to their use of RWD.

Tools for assessing quality of studies investigating health interventions using real-world data: a literature review and content analysis

OBJECTIVES

We aimed to identify existing appraisal tools for non-randomised studies of interventions (NRSIs) and to compare the criteria that the tools provide at the quality-item level.

DESIGN

Literature review through three approaches: systematic search of journal articles, snowballing search of reviews on appraisal tools and grey literature search on websites of health technology assessment (HTA) agencies.

DATA SOURCES

Systematic search: Medline; Snowballing: starting from three articles (D'Andrea et al, Quigley et al and Faria et al); Grey literature: websites of European HTA agencies listed by the International Network of Agencies for Health Technology Assessment. Appraisal tools were searched through April 2022.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We included a tool, if it addressed quality concerns of NRSIs and was published in English (unless from grey literature). A tool was excluded, if it was only for diagnostic, prognostic, qualitative or secondary studies.

DATA EXTRACTION AND SYNTHESIS

Two independent researchers searched, screened and reviewed all included studies and tools, summarised quality items and scored whether and to what extent a quality item was described by a tool, for either methodological quality or reporting.

RESULTS

Forty-nine tools met inclusion criteria and were included for the content analysis. Concerns regarding the quality of NRSI were categorised into 4 domains and 26 items. The Research Triangle Institute Item Bank (RTI Item Bank) and STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) were the most comprehensive tools for methodological quality and reporting, respectively, as they addressed (n=20; 17) and sufficiently described (n=18; 13) the highest number of items. However, none of the tools covered all items.

CONCLUSION

Most of the tools have their own strengths, but none of them could address all quality concerns relevant to NRSIs. Even the most comprehensive tools can be complemented by several items. We suggest decision-makers, researchers and tool developers consider the quality-item level heterogeneity, when selecting a tool or identifying a research gap.

OSF REGISTRATION NUMBER

OSF registration DOI (https://doi.org/10.17605/OSF.IO/KCSGX).

Comparative effectiveness and cost-effectiveness of natalizumab and fingolimod in rapidly evolving severe relapsing-remitting multiple sclerosis in the United Kingdom

AIM

To evaluate the real-world comparative effectiveness and the cost-effectiveness, from a UK National Health Service perspective, of natalizumab versus fingolimod in patients with rapidly evolving severe relapsing-remitting multiple sclerosis (RES-RRMS).

METHODS

Real-world data from the MSBase Registry were obtained for patients with RES-RRMS who were previously either naive to disease-modifying therapies or had been treated with interferon-based therapies, glatiramer acetate, dimethyl fumarate, or teriflunomide (collectively known as BRACETD). Matched cohorts were selected by 3-way multinomial propensity score matching, and the annualized relapse rate (ARR) and 6-month-confirmed disability worsening (CDW6M) and improvement (CDI6M) were compared between treatment groups. Comparative effectiveness results were used in a cost-effectiveness model comparing natalizumab and fingolimod, using an established Markov structure over a lifetime horizon with health states based on the Expanded Disability Status Scale. Additional model data sources included the UK MS Survey 2015, published literature, and publicly available sources.

RESULTS

In the comparative effectiveness analysis, we found a significantly lower ARR for patients starting natalizumab compared with fingolimod (rate ratio [RR] = 0.65; 95% confidence interval [CI], 0.57-0.73) or BRACETD (RR = 0.46; 95% CI, 0.42-0.53). Similarly, CDI6M was higher for patients starting natalizumab compared with fingolimod (hazard ratio [HR] = 1.25; 95% CI, 1.01-1.55) and BRACETD (HR = 1.46; 95% CI, 1.16-1.85). In patients starting fingolimod, we found a lower ARR (RR = 0.72; 95% CI, 0.65-0.80) compared with starting BRACETD, but no difference in CDI6M (HR = 1.17; 95% CI, 0.91-1.50). Differences in CDW6M were not found between the treatment groups. In the base-case cost-effectiveness analysis, natalizumab dominated fingolimod (0.302 higher quality-adjusted life-years [QALYs] and £17,141 lower predicted lifetime costs). Similar cost-effectiveness results were observed across sensitivity analyses.

CONCLUSIONS

This MSBase Registry analysis suggests that natalizumab improves clinical outcomes when compared with fingolimod, which translates to higher QALYs and lower costs in UK patients with RES-RRMS.

Modelling Informal Carers' Health-Related Quality of Life: Challenges for Economic Evaluation

There has been increasing interest in including carers' health-related qualify of life (HRQoL) in decision models, but currently there is no best practice guidance as to how to do so. Models thus far have typically assumed that carers' HRQoL can be predicted from patient health states, as we illustrate with three examples of disease-modifying treatments. However, this approach limits the mechanisms that influence carers' HRQoL solely to patient health and may not accurately reflect carers' outcomes. In this article, we identify and discuss challenges associated with modelling intervention effects on carers' HRQoL: attaching carer utilities to patient disease states, the size of the caring network, aggregation of carer and patient HRQoL, patient death, and modelling longer-term carer HRQoL. We review and critique potential alternatives to modelling carers' HRQoL in decision models: trial-based analyses, qualitative consideration, cost-consequence analysis, and multicriteria decision analysis, noting that each of these also has its own challenges. We provide a framework of issues to consider when modelling carers' HRQoL and suggest how these can be addressed in current practice and future research.

Testing the "RCT augmentation" methodology: A trial simulation study to guide the broadening of trials eligibility criteria and inform on effectiveness

Background

Exclusion criteria that are treatment effect modifiers (TEM) decrease RCTs results generalisability and the potentials of effectiveness estimation. In "augmented RCTs", a small proportion of otherwise-excluded patients are included to allow for effectiveness estimation. In Hodgkin Lymphoma (HL) RCTs, older age and comorbidity are common exclusion criteria, while also TEM. We simulated HL RCTs augmented with age or comorbidity, and explored in each scenario the impact of augmentation on effectiveness estimation accuracy.

Methods

Simulated data with a population of HL individuals initiating drug A or B was generated. There were drug-age and drug-comorbidity interactions in the simulated data, with a greater magnitude of the former compared to the latter. Multiple augmented RCTs were simulated by randomly selecting patients with increasing proportions of older, or comorbid patients. Treatment effect size was expressed using the between-group Restricted Mean Survival Time (RMST) difference at 3 years. For each augmentation proportion, a model estimating the "real-world" treatment effect (effectiveness) was fitted and the estimation error measured (Root Mean Square Error, RMSE).

Results

In simulated RCTs including none (0%), or the real-world proportion (30%) of older patients, the interquartile range of RMST difference was 0.4-0.5 years and 0.2-0.3 years, respectively, and RMSE were 0.198 years (highest possible error) and 0.056 years (lowest), respectively. Augmenting RCTs with 5% older patients decreased estimation error substantially (RMSE = 0.076 years). Augmentation with comorbid patients proved less useful for effectiveness estimation.

Conclusion

In augmented RCTs aiming to inform the effectiveness of drugs, augmentation should concern in priority those exclusion criteria of suspected important TEM magnitude, so as to minimie the proportion of augmentation necessary for good effectiveness estimations.

Real-world evidence in lung and hematologic oncology health technology appraisals: a review of six assessment agencies

Aim: To assess the use and acceptability of real-world evidence (RWE) in lung and hematologic cancer appraisals. Materials & methods: A review of appraisals published by National Institute for Health and Care Excellence (NICE) in the UK was conducted. A total of 20 case studies employing RWE were identified and compared across five additional health technology assessment agencies: Scottish Medicines Consortium (SMC) (Scotland), CADTH (Canada), INESSS (Quebec), HAS (France) and IQWiG (Germany). Results: Of 80 RWE references from 20 case studies from NICE, 67 were identified in the respective CADTH submissions, 46 in IQWiG, 37 in INESSS, 37 in HAS, and 33 in SMC. NICE had the highest RWE acceptance rate (90%), followed by HAS (88%), SMC (82%), INESSS (73%), IQWiG (68%), and CADTH (67%). Conclusion: RWE was generally accepted by respective committees, allowing improved access to innovative treatments.

Using Real-World Data in the Health Technology Assessment of Pharmaceuticals: Strengths, Difficulties, and a Pragmatic Way Forward

In the past decade, there have been increasing calls for greater use of real-world evidence (RWE) and data (RWD), with the explicit goal of enabling faster provision of effective medicines to patients in need. The push for decision makers to accept RWE is especially noticeable in the pursuit of regulatory approval, but RWE, particularly when used to estimate the relative effectiveness of interventions, is not always readily accepted by agencies responsible for reimbursement and pricing of new pharmaceuticals and, to a varying degree, is not accepted across jurisdictions. This lack of trust hampers the use of RWE despite a very large and growing literature base on the principles of how RWE should be used. In this article, we suggest an important part of the explanation of why this situation has arisen and make suggestions for its alleviation. Given that problems commonly arise that are particular to the question being asked and the data sources being used, general guidance on the principles of how to use RWD cannot cover all eventualities. Therefore, we are suggesting the creation of an archive, or repository, to record uses of RWD in support of decisions by funding bodies or their advisors. This article introduces a proposed, structured classification of decision types using RWE, around which evidence can be assembled in a curated source (RWD/RWE taxonomy) and thus facilitate judgments on when evidence is "good enough." This article is part of a series in a special issue of this journal that looks at the barriers to optimal use of RWE in health technology assessment and how to overcome them. We begin significantly to populate our "taxonomy" with examples in an accompanying article. We also propose recommendations for international standards of evaluating the acceptability of RWD governance practices.

The Real-World Evidence Workstream in EUreccA 2025: How the Task Was Addressed

This is one of a series of articles that consider the barriers to optimal use of real-world evidence (RWE) in health technology assessment (HTA) as well as ways to overcome them. The work was carried out as part of EUreccA 2025 (European Initiative for New Reimbursement and Access Approaches 2025), in particular with the RWE workstream embodied within that collaboration. The starting premises of this workstream were as follows: (1) the acceptance of RWE by HTA agencies and payers in the assessment of drugs is suboptimal and variable between jurisdictions, and (2) if that were not the case, the path of new pharmaceuticals to patients could be quicker and less expensive. Elsewhere in this issue we set out the conclusions we had reached in the EUreccA RWE workstream. In this article, we set out the methodology used to conduct the totality of the EureccA 2025 RWE workstream effort, which led us to those conclusions. The main results, strengths, and limitations of the individual parts are discussed further in separate articles in this supplement. Through scoping work, we generated 4 key topics within which to identify and address the barriers to optimal RWE use in HTA. Through pragmatic literature searches, stakeholder engagement, and case studies, we suggest ways in which the problems identified may be addressed as a contribution to progress in this area.

Across-Country Variations of Real-World Data and Evidence for Drugs: A 5-European-Country Study

OBJECTIVES

This study aimed to describe the role of real-world data (RWD) and real-world evidence (RWE) in health technology assessment (HTA) in 5 European countries and to identify the hurdles to the acceptance of RWE and suggest directions toward its more effective use.

METHODS

Authors from France, Germany, Italy, and Sweden used a common template to extract evidence. For England, the Cancer Drugs Fund was described and analyzed as a particular model for the use of RWD to provide evidence for coverage decisions and managed entry agreements.

RESULTS

In all countries except Germany, HTA bodies acknowledged the relevance of RWD/RWE to address postlaunch uncertainties. In Germany, evidence from randomized controlled trials remains the gold standard, and evidence based on RWD is generally rejected. Multiple sources of RWD exist, but the quality, the immediate relevance of existing sources, and their interoperability limit their adaptation to the specifics of a given drug. This leads to skepticism about the validity of the evidence. Timing is also a key issue: the production of evidence may not be synchronized with the HTA and pricing bodies' agendas. The Cancer Drugs Fund case emphasizes that a strong partnership among all stakeholders and a pragmatic use of existing data, alongside clinical evidence provided by companies, are key success factors.

CONCLUSIONS

A continuous investment in national health information systems is a key issue for providing valid RWE. Processes and aids to guide the acceptability and usage of RWE derived from pairing between sources and questions are essential.

Real-world study of overall survival with palbociclib plus aromatase inhibitor in HR+/HER2- metastatic breast cancer

Data on real-world effectiveness of cyclin-dependent kinase 4/6 inhibitor combination therapy versus endocrine therapy alone are limited. The Flatiron Health Analytic Database was used to assess overall survival (OS) in patients with hormone receptor–positive/human epidermal growth factor receptor 2–negative (HR+/HER2−) metastatic breast cancer (MBC) treated with first-line palbociclib plus an aromatase inhibitor (AI) versus an AI alone in routine US clinical practice. In total, 2888 patients initiated treatment during February 3, 2015–March 31, 2020, with a potential ≥6-month follow-up (cutoff date, September 30, 2020). After stabilized inverse probability treatment weighting, median OS (95% CI) is significantly longer among palbociclib versus AI recipients (49.1 [45.2–57.7] versus 43.2 [37.6–48.0] months; hazard ratio, 0.76 [95% CI, 0.65–0.87]; P < 0.0001). Progression-free survival (95% CI) is 19.3 (17.5–20.7) versus 13.9 (12.5–15.2) months, respectively (hazard ratio, 0.70 [95% CI, 0.62–0.78]; P < 0.0001). These data support first-line palbociclib plus an AI treatment for HR+/HER2− MBC.

(Trial number NCT05361655).

Real-world evidence in health technology assessment of high-risk medical devices: Fit for purpose?

Health technology assessment (HTA) of medical devices (MDs) increasingly rely on real-world evidence (RWE). The aim of this study was to evaluate the type and the quality of the evidence used to assess the (cost-)effectiveness of high risk MDs (Class III) by HTA agencies in Europe (four European HTA agencies and EUnetHTA), with particular focus on RWE. Data were extracted from HTA reports on the type of evidence demonstrating (cost-)effectiveness, and the quality of observational studies of comparative effectiveness using the Good Research for Comparative Effectiveness principles. 25 HTA reports were included that incorporated 28 observational studies of comparative effectiveness. Half of the studies (46%) took important confounding and/or effect modifying variables into account in the design and/or analyses. The most common way of including confounders and/or effect modifiers was through multivariable regression analysis. Other methods, such as propensity score matching, were rarely employed. Furthermore, meaningful analyses to test key assumptions were largely omitted. Resulting recommendations from HTA agencies on MDs is therefore (partially) based on evidence which is riddled with uncertainty. Considering the increasing importance of RWE it is important that the quality of observational studies of comparative effectiveness are systematically assessed when used in decision-making.

Barriers to Use Artificial Intelligence Methodologies in Health Technology Assessment in Central and East European Countries

The aim of this paper is to identify the barriers that are specifically relevant to the use of Artificial Intelligence (AI)-based evidence in Central and Eastern European (CEE) Health Technology Assessment (HTA) systems. The study relied on two main parallel sources to identify barriers to use AI methodologies in HTA in CEE, including a scoping literature review and iterative focus group meetings with HTx team members. Most of the other selected articles discussed AI from a clinical perspective (n = 25), and the rest are from regulatory perspective (n = 13), and transfer of knowledge point of view (n = 3). Clinical areas studied are quite diverse—from pediatric, diabetes, diagnostic radiology, gynecology, oncology, surgery, psychiatry, cardiology, infection diseases, and oncology. Out of all 38 articles, 25 (66%) describe the AI method and the rest are more focused on the utilization barriers of different health care services and programs. The potential barriers could be classified as data related, methodological, technological, regulatory and policy related, and human factor related. Some of the barriers are quite similar, especially concerning the technologies. Studies focusing on the AI usage for HTA decision making are scarce. AI and augmented decision making tools are a novel science, and we are in the process of adapting it to existing needs. HTA as a process requires multiple steps, multiple evaluations which rely on heterogenous data. Therefore, the observed range of barriers come as a no surprise, and experts in the field need to give their opinion on the most important barriers in order to develop recommendations to overcome them and to disseminate the practical application of these tools.

Reported Challenges in Health Technology Assessment of Complex Health Technologies

OBJECTIVES

With complex health technologies entering the market, methods for health technology assessment (HTA) may require changes. This study aimed to identify challenges in HTA of complex health technologies.

METHODS

A survey was sent to European HTA organizations participating in European Network for HTA (EUnetHTA). The survey contained open questions and used predefined potentially complex health technologies and 7 case studies to identify types of complex health technologies and challenges faced during HTA. The survey was validated, tested for reliability by an expert panel, and pilot tested before dissemination.

RESULTS

A total of 22 HTA organizations completed the survey (67%). Advanced therapeutic medicinal products (ATMPs) and histology-independent therapies were considered most challenging based on the predefined complex health technologies and case studies. For the case studies, more than half of the reported challenges were "methodological," equal in relative effectiveness assessments as in cost-effectiveness assessments. Through the open questions, we found that most of these challenges actually rooted in data unavailability. Data were reported as "absent," "insufficient," "immature," or "low quality" by 18 of 20 organizations (90%), in particular data on quality of life. Policy and organizational challenges and challenges because of societal or political pressure were reported by 8 (40%) and 4 organizations (20%), respectively. Modeling issues were reported least often (n = 2, 4%).

CONCLUSIONS

Most challenges in HTA of complex health technologies root in data insufficiencies rather than in the complexity of health technologies itself. As the number of complex technologies grows, the urgency for new methods and policies to guide HTA decision making increases.

Target Trial Emulation for Transparent and Robust Estimation of Treatment Effects for Health Technology Assessment Using Real-World Data: Opportunities and Challenges

Evidence about the relative effects of new treatments is typically collected in randomised controlled trials (RCTs). In many instances, evidence from RCTs falls short of the needs of health technology assessment (HTA). For example, RCTs may not be able to capture longer-term treatment effects, or include all relevant comparators and outcomes required for HTA purposes. Information routinely collected about patients and the care they receive have been increasingly used to complement RCT evidence on treatment effects. However, such routine (or real-world) data are not collected for research purposes, so investigators have little control over the way patients are selected into the study or allocated to the different treatment groups, introducing biases for example due to selection or confounding. A promising approach to minimise common biases in non-randomised studies that use real-world data (RWD) is to apply design principles from RCTs. This approach, known as 'target trial emulation' (TTE), involves (1) developing the protocol with respect to core study design and analysis components of the hypothetical RCT that would answer the question of interest, and (2) applying this protocol to the RWD so that it mimics the data that would have been gathered for the RCT. By making the 'target trial' explicit, TTE helps avoid common design flaws and methodological pitfalls in the analysis of non-randomised studies, keeping each step transparent and accessible. It provides a coherent framework that embeds existing analytical methods to minimise confounding and helps identify potential limitations of RWD and the extent to which these affect the HTA decision. This paper provides a broad overview of TTE and discusses the opportunities and challenges of using this approach in HTA. We describe the basic principles of trial emulation, outline some areas where TTE using RWD can help complement RCT evidence in HTA, identify potential barriers to its adoption in the HTA setting and highlight some priorities for future work.

Gene Therapy Evidence Generation and Economic Analysis: Pragmatic Considerations to Facilitate Fit-for-Purpose Health Technology Assessment

Gene therapies (GTs) are considered to be a paradigm-shifting class of treatments with the potential to treat previously incurable diseases or those with significant unmet treatment needs. However, considerable challenges remain in their health technology assessment (HTA), mainly stemming from the inability to perform robust clinical trials to convince decision-makers to pay the high prices for the potential long-term treatment benefits provided. This article aims to review the recommendations that have been published for evidence generation and economic analysis for GTs against the feasibility of their implementation within current HTA decision analysis frameworks. After reviewing the systematically identified literature, we found that questions remain on the appropriateness of GT evidence generation, considering that additional, broader values brought by GTs seem insufficiently incorporated within proposed analytic methods. In cases where innovative methods are proposed, HTA organizations remain highly conservative and resistant to change their reference case and decision analysis framework. Such resistances are largely attributed to the substantial evidence uncertainty, resource-consuming administration process, and the absence of consensus on the optimized methodology to balance all the advantages and potential pitfalls of GTs.

Health technology assessment processes: a North-South comparison of the evaluation and recommendation of health technologies in Canada and Chile

Purpose

Health systems are progressively stressed by health spending, which is partially explained by the increase in the cost of health technologies. Countries have defined processes to prioritize interventions to be covered. This study aims to compare for the first time health technology assessment (HTA) processes in Canada and Chile, to explain the factors driving these decisions.

Design/methodology/approach

This is a health policy analysis comparing HTA processes in Canada and Chile. An analysis of publicly available documents in Canada (for CADTH) and Chile (for the Ministry of Health (MoH)) was carried out. A recognized political science framework (the 3-I framework) was used to explain the similarities and differences in both countries. The comparison of processes was disaggregated into eligibility and evaluation processes.

Findings

CADTH has different programmes for different types of drugs (with two separate expert committees), whereas the MoH has a unified process. Although CADTH’s recommendations have a federal scope, the final coverage is a provincial decision. In Chile, the recommendation has a national scope. In both cases, past recommendations influence the scope of the evaluation. Pharmaceutical companies and patient associations are important interest groups in both countries. Whereas manufacturers and tumour groups are able to submit applications to CADTH, the Chilean MoH prioritizes applications submitted by patient associations.

Originality/value

Institutions, interests and ideas play important roles in driving HTA decisions in Canada and Chile, which is demonstrated in this novel analysis. This paper provides a unique comparison to highly relevant policy processes in HTA, which is often a research area dominated by effectiveness and cost-effectiveness studies.

Haematology Outcomes Network in Europe (HONEUR)-A collaborative, interdisciplinary platform to harness the potential of real-world data in hematology

INTRODUCTION

There remains a need to optimize treatments and improve outcomes among patients with hematologic malignancies. The timely synthesis and analysis of real-world data could play a key role.

OBJECTIVES

The Haematology Outcomes Network in Europe (HONEUR) is a federated data network (FDN) that aims to overcome the challenges of heterogenous data collected from different registries, hospitals, and other databases in different countries. It has the functionality required to analyze data from various sources in a time efficient manner, while preserving local data security and governance. With this, research studies can be performed that can increase knowledge and understanding of the management of patients with hematologic malignancies.

METHODS

HONEUR uses the Observational Medical Outcomes Partnership (OMOP) common data model, which allows analysis scripts to be run by multiple sites using their own data, ultimately generating aggregated results. Furthermore, distributed analytics can be used to run statistical analyses across multiple sites, as if data were pooled. The external governance model ensures high-quality standards, while data ownership is retained locally. Twenty partners from nine countries are now participating, with data from more than 26 000 patients available for analysis. Research questions that can be addressed through HONEUR include assessments of natural disease history, treatment patterns, and clinical effectiveness.

CONCLUSIONS

The HONEUR FDN marks an important step forward in increasing the value of information routinely captured by individual hospitals, registries and other database holders, thus enabling larger-scale studies to be undertaken rapidly and efficiently.

Assessing Payers' Preferences for Real-World Evidence in the United States: A Discrete Choice Experiment

OBJECTIVES

To rank the US payers' preferences for attributes of real-world evidence (RWE) studies in the context of chronic disease and to quantify trade-offs among them.

METHODS

We conducted a discrete choice experiment in which 180 employees from payer organizations were tasked to choose between 2 RWE studies assuming they were assessing evidence to inform formulary decisions for chronic disease treatment. Each RWE study was characterized by 7 attributes with 3 levels each: very informative, moderately informative, and not measured. We used a D-optimal main-effects design. Survey data were fitted to a conditional logit model to obtain a relative measure of the ranking of importance for each attribute.

RESULTS

Clinical outcomes were the most preferred attribute. It was 4.68 times as important as productivity outcomes-the least preferred attribute. It was followed by health-related quality of life (2.78), methodologic rigor (2.09), resource utilization (1.71), and external validity (1.56).

CONCLUSIONS

This study provides a quantification of the value payers place on key RWE attributes. Across attributes, payers have higher preferences for clinical and health-related quality of life outcomes than the other attributes. Between attributes' levels, payers prefer high levels of information in clinical outcomes and methodologic rigor but are indifferent in other attributes. Our results bridge the gap between the information that payers seek and the attributes that RWE studies prioritize and effectively guide future research design.

Real-World Efficacy of Treatment with Benralizumab, Dupilumab, Mepolizumab and Reslizumab for Severe Asthma: A Systematic Review and Meta-analysis

BACKGROUND

Severe asthma is a major cause of morbidity. Some patients may benefit from biological therapies. Most evaluations of these treatments are derived from randomised controlled trials (RCTs), but few patients are eligible for these trials. Studies involving more diverse groups of participants exist but there is a lack of precise pooled estimates.

OBJECTIVE

This systematic review aims to evaluate the real-world efficacy of recently and nearly licensed biological therapies for severe asthma to assess the generalisability of the RCT data.

METHODS

Clinical outcomes including exacerbation rate, oral corticosteroid (OCS) usage, forced expiratory volume in 1 second (FEV₁ ) and fractional exhaled nitric oxide (FeNO) were examined. Studies were assessed for risk of bias using the Critical Appraisal Skills Programme (CASP) checklist tool. The certainty of evidence was assessed using GRADE.

RESULTS

A total of 21 studies examining biologicals in real-world settings were identified, they mostly focused on benralizumab and mepolizumab. The introduction of biologicals reduced the annualised exacerbation rate significantly by -3.79 (95% CI -4.53, -3.04), -3.17 (95% CI -3.74, -2.59) and -6.72 (95% CI -8.47, -4.97) with benralizumab, mepolizumab and reslizumab respectively. Likewise, improvements were observed in FEV₁ (0.17 L 95% CI 0.11, 0.24) and FeNO (-14.23 ppb 95% CI -19.71, -8.75) following treatment with mepolizumab. After treatment with benralizumab there was an increase in FEV₁ (0.21 L 95% CI 0.08, 0.34).

CONCLUSIONS

These data demonstrate that anti-IL5 biologicals may improve the clinical outcomes of patients with severe asthma in a clinic environment with similar effect sizes to RCTs. The data were mainly retrospective and unadjusted, so estimated effect sizes may not be reliable. More data is needed to acquire accurate effect estimates in different subpopulations of patients.

Real World Data in Health Technology Assessment of Complex Health Technologies

The available evidence on relative effectiveness and risks of new health technologies is often limited at the time of health technology assessment (HTA). Additionally, a wide variety in real-world data (RWD) policies exist among HTA organizations. This study assessed which challenges, related to the increasingly complex nature of new health technologies, make the acceptance of RWD most likely. A questionnaire was disseminated among 33 EUnetHTA member HTA organizations. The questions focused on accepted data sources, circumstances that allowed for RWD acceptance and barriers to acceptance. The questionnaire was validated and tested for reliability by an expert panel, and pilot-tested before dissemination via LimeSurvey. Twenty-two HTA organizations completed the questionnaire (67%). All reported accepting randomized clinical trials. The most accepted RWD source were patient registries (19/22, 86%), the least accepted were editorials and expert opinions (8/22, 36%). With orphan treatments or companion diagnostics, organizations tended to be most likely to accept RWD sources, 4.3–3.2 on a 5-point Likert scale, respectively. Additional circumstances were reported to accept RWD (e.g., a high disease burden). The two most important barriers to accepting RWD were lacking necessary RWD sources and existing policy structures. European HTA organizations seem positive toward the (wider) use of RWD in HTA of complex therapies. Expanding the use of patient registries could be potentially useful, as a large share of the organizations already accepts this source. However, many barriers still exist to the widespread use of RWD. Our results can be used to prioritize circumstances in which RWD might be accepted.

Can we use existing guidance to support the development of robust real-world evidence for health technology assessment/payer decision-making?

Advances in the digitization of health systems and expedited regulatory approvals of innovative treatments have led to increased potential for the use of real-world data (RWD) to generate real-world evidence (RWE) to complement evidence from clinical trials. However, health technology assessment (HTA) bodies and payers have concerns about the ability to generate RWE of sufficient quality to be pivotal evidence of relative treatment effectiveness. Consequently, there is a growing need for HTA bodies and payers to develop guidance for the industry and other stakeholders about the use of RWD/RWE to support access, reimbursement, and pricing. We therefore sought to (i) understand barriers to the use of RWD/RWE by HTA bodies and payers; (ii) review potential solutions in the form of published guidance; and (iii) review findings with selected HTA/payer bodies. Four themes considered key to shaping the generation of robust RWE for HTA bodies and payers were identified as: (i) data (availability, governance, and quality); (ii) methodology (design and analytics); (iii) trust (transparency and reproducibility); and (iv) policy and partnerships. A range of guidance documents were found from trusted sources that could address these themes. These were discussed with HTA experts. This commentary summarizes the potential guidance solutions available to help resolve issues faced by HTA decision-makers in the adoption of RWD/RWE. It shows that there is alignment among stakeholders about the areas that need improvement in the development of RWE and that the key priority to move forward is better collaboration to make data usable for multiple purposes.

Big Data and Real-World Data based Cost-Effectiveness Studies and Decision-making Models: A Systematic Review and Analysis

Background: Big data and real-world data (RWD) have been increasingly used to measure the effectiveness and costs in cost-effectiveness analysis (CEA). However, the characteristics and methodologies of CEA based on big data and RWD remain unknown. The objectives of this study were to review the characteristics and methodologies of the CEA studies based on big data and RWD and to compare the characteristics and methodologies between the CEA studies with or without decision-analytic models. Methods: The literature search was conducted in Medline (Pubmed), Embase, Web of Science, and Cochrane Library (as of June 2020). Full CEA studies with an incremental analysis that used big data and RWD for both effectiveness and costs written in English were included. There were no restrictions regarding publication date. Results: 70 studies on CEA using RWD (37 with decision-analytic models and 33 without) were included. The majority of the studies were published between 2011 and 2020, and the number of CEA based on RWD has been increasing over the years. Few CEA studies used big data. Pharmacological interventions were the most frequently studied intervention, and they were more frequently evaluated by the studies without decision-analytic models, while those with the model focused on treatment regimen. Compared to CEA studies using decision-analytic models, both effectiveness and costs of those using the model were more likely to be obtained from literature review. All the studies using decision-analytic models included sensitivity analyses, while four studies no using the model neither used sensitivity analysis nor controlled for confounders. Conclusion: The review shows that RWD has been increasingly applied in conducting the cost-effectiveness analysis. However, few CEA studies are based on big data. In future CEA studies using big data and RWD, it is encouraged to control confounders and to discount in long-term research when decision-analytic models are not used.

Current challenges for assessing the long-term clinical benefit of cancer immunotherapy: a multi-stakeholder perspective

Immuno-oncologics (IOs) differ from chemotherapies as they prime the patient’s immune system to attack the tumor, rather than directly destroying cancer cells. The IO mechanism of action leads to durable responses and prolonged survival in some patients. However, providing robust evidence of the long-term benefits of IOs at health technology assessment (HTA) submission presents several challenges for manufacturers. The aim of this article was to identify, analyze, categorize, and further explore the key challenges that regulators, HTA agencies, and payers commonly encounter when assessing the long-term benefits of IO therapies. Insights were obtained from an international, multi-stakeholder steering committee (SC) and expert panels comprising of payers, economists, and clinicians. The selected individuals were tasked with developing a summary of challenges specific to IOs in demonstrating their long-term benefits at HTA submission. The SC and expert panels agreed that standard methods used to assess the long-term benefit of anticancer drugs may have limitations for IO therapies. Three key areas of challenges were identified: (1) lack of a disease model that fully captures the mechanism of action and subsequent patient responses; (2) estimation of longer-term outcomes, including a lack of agreement on ideal methods of survival analyses and extrapolation of survival curves; and (3) data limitations at the time of HTA submission, for which surrogate survival end points and real-world evidence could prove useful. A summary of the key challenges facing manufacturers when submitting evidence at HTA submission was developed, along with further recommendations for manufacturers in what evidence to produce. Despite almost a decade of use, there remain significant challenges around how best to demonstrate the long-term benefit of checkpoint inhibitor-based IOs to HTA agencies, clinicians, and payers. Manufacturers can potentially meet or mitigate these challenges with a focus on strengthening survival analysis methodology. Approaches to doing this include identifying reliable biomarkers, intermediate and surrogate end points, and the use of real-world data to inform and validate long-term survival projections. Wider education across all stakeholders—manufacturers, payers, and clinicians—in considering the long-term survival benefit with IOs is also important.

Measuring the Effectiveness of Real-World Evidence to Ensure Appropriate Impact

The value of real-world evidence (RWE) in medicines regulation and health technology assessment has been increasingly emphasized. Nevertheless, although RWE is increasingly used, there has been limited systematic evidence of its value. A recent study that examined the role and impact of RWE in regulatory assessments conducted through the European Medicines Agency provided such evidence. Results of the study demonstrated RWE was important to decision making, particularly for certain questions such as the quantification of adverse events, the evaluation of risk minimization measures, and the assessment of product usage. The study suggested, however, that in many of the assessments further RWE would have been valuable and concluded that RWE has, as yet, played a limited role in hypothesis generation and in the assessment of medication effectiveness. This study had been possible only because of the transparency of the European Medicines Agency decision making. Ensuring transparency of RWE evidence collection, study design and conduct, and of decision making based on this evidence will facilitate further development of the uses and value of RWE. Keywords: benefit-risk assessment; medicines regulation; real-world evidence; regulatory decision making.

Real-world outcomes associated with new cancer medicines approved by the Food and Drug Administration and European Medicines Agency: A retrospective cohort study

Purpose

Real-World Data (RWD) studies are increasingly used to support regulatory approvals, reimbursement decisions, and changes in clinical practice for novel cancer drugs. However, few studies have systematically appraised their quality or compared outcomes to pivotal trials.

Methods

All RWD studies (2010–2019) for drugs approved by the Food and Drug Administration (FDA) and European Medicines Agency (EMA) from 2010 to 2015 for solid organ tumours in the non-curative setting were identified. Quality assessment was undertaken using the Newcastle Ottawa Scale. Survival differences between each RWD study and the pivotal trial were determined using a related sample Wilcoxon signed-rank test.

Results

293 RWD studies for 45 of the 57 drug indications approved by the FDA/EMA were identified. The most common tumour types were prostate cancer (29%, n = 86) and melanoma (15%, n = 43). A quarter of the studies had industry funding. No high-quality studies were identified, and 78% were low quality. Comparative survival analysis between RWD and pivotal trials was possible for 224 studies (37 drug indications). Differences in median survival between the RWD studies and their corresponding trial ranged from −32 months to 21 months (IQR –4·2 months to 1·6 months). Low-quality studies were more likely to report superior survival outcomes (23%) compared to higher quality studies (8%) (p = 0.02).

Conclusion

RWD study quality for novel cancer drugs is low and of insufficient rigour to inform reimbursement decisions and clinical practice. RWD studies seeking publication should provide a completed quality assessment tool on submission. Greater investment in properly designed RWD studies is required.

•

Study provides a systematic appraisal of FDA/EMA approved drugs in real-world practice.

•

Most novel FDA/EMA cancer drugs have real-world data (RWD) studies, but the quality is low.

•

Variability in survival outcomes exists, and findings should be applied cautiously.

•

Most RWD studies reported inferior survival outcomes compared to the pivotal trial.

•

Pre-publication critical appraisal checklists should be used for RWD studies.

The use of nonrandomized evidence to estimate treatment effects in health technology assessment

Health technology assessment (HTA) is increasingly informed by nonrandomized studies, but there is limited guidance from HTA bodies on expectations around evidence quality and study conduct. We developed recommendations to support the appropriate use of such evidence based on a pragmatic literature review and a workshop involving 16 experts from eight countries as part of the EU's Horizon-2020 IMPACT-HTA program (work package six). To ensure HTA processes remain rigorous and robust, HTA bodies should demand clear, extensive and structured reporting of nonrandomized studies, including an in-depth assessment of the risk of bias. In recognition of the additional uncertainty imparted by nonrandomized designs in estimates of treatment effects, HTA bodies should strengthen early scientific advice and engage in collaborative efforts to improve use of real-world data.

Low rates of serious complications and further procedures following surgery for base of thumb osteoarthritis: analysis of a national cohort of 43 076 surgeries

OBJECTIVES

To determine the incidence of further procedures and serious adverse events (SAEs) requiring admission to hospital following elective surgery for base of thumb osteoarthritis (BTOA), and the patient factors associated with these outcomes.

DESIGN

Population based cohort study.

SETTING

National Health Service using the national Hospital Episode Statistics data set linked to mortality records over a 19-year period (01 April 1998-31 March 2017).

PARTICIPANTS

43 076 primary surgeries were followed longitudinally in secondary care until death or migration on 37 329 patients over 18 years of age.

MAIN OUTCOME MEASURES

Incidence of further thumb base procedures (including revision surgery or intra-articular steroid injection) at any time postoperatively, and local wound complications and systemic events (myocardial infarction, stroke, respiratory tract infection, venous thromboembolic events, urinary tract infection or renal failure) within 30 and 90 days. To identify patient factors associated with outcome, Fine and Gray model regression analysis was used to adjust for the competing risk of mortality in addition to age, overall comorbidity and socioeconomic status.

RESULTS

Over the 19 years, there was an increasing trend in surgeries undertaken. The rate of further thumb base procedures after any surgery was 1.39%; the lowest rates after simple trapeziectomy (1.12%), the highest rates after arthroplasty (3.84%) and arthrodesis (3.5%). When matched for age, comorbidity and socioeconomic status, those undergoing arthroplasty and arthrodesis were 2.5 times more likely to undergo a further procedure (subHR 2.51 (95% CI 1.81 to 3.48) and 2.55 (1.91 to 3.40)) than those undergoing simple trapeziectomy. Overall complication rates following surgery were 0.22% for serious local complications and 0.58% for systemic events within 90 days of surgery.

CONCLUSIONS

The number of patients proceeding to BTOA surgery has increased over the last 19 years, with a low rate of further thumb base procedures and SAEs after surgery overall registered. Arthrodesis and arthroplasty had a significantly higher revision rate.

TRIAL REGISTRATION NUMBER

NCT03573765.

Healthcare Funding Decisions and Real-World Benefits: Reducing Bias by Matching Untreated Patients

Governments and health insurers often make funding decisions based on health gains from randomised controlled trials. These decisions are inherently uncertain because health gains in trials may not translate to practice owing to differences in the population, treatment use and setting. Post-market analysis of real-world data can provide additional evidence but estimates from standard matching methods may be biased when unobserved characteristics explain whether a patient is treated and their outcomes. We propose a new untreated matching approach that can reduce this bias. Our approach utilises the outcomes of contemporaneous untreated patients to improve the matching of treated and historical control patients. We assess the performance of this new approach compared to standard matching using a simulation study and demonstrate the steps required using a funding decision for prostate cancer treatments in Australia. Our simulation study shows that our new matching approach eliminates nearly all bias when unobserved treatment selection is related to outcomes, and outperforms standard matching in most scenarios. In our empirical example, standard matching overestimated survival by 15% (95% confidence interval 2-34) compared to our untreated matching approach. The health gains estimated using our approach were slightly lower than expected based on the trial evidence, but we also found evidence that in practice prescribers ceased prior therapies earlier, treated a more vulnerable population and continued treatment for longer. Our untreated matching approach offers researchers a new tool for reducing uncertainty in healthcare funding decisions using real-world data.

QALYs and medical costs saved from prevention of a cancer: Analysis of nation-wide real-world data of Taiwan with lifetime horizon

BACKGROUND/PURPOSE

To quantify savings of loss-of-QALE (quality-adjusted life expectancy) and lifetime medical costs from prevention of different cancers.

METHODS

We collected nation-wide data on 808,700 new cancer cases of 14 different organ systems and followed them from 1998 to 2014 in Taiwan. We also collected 13,005 cancer patients from a medical center and 47,320 repeated measurements of quality of life (QoL) of EQ-5D to obtain utility values and multiplied them with the corresponding survival rates to calculate QALE. With Kaplan-Meier estimation to survival function to the end of follow-up, we extrapolated to lifetime through a rolling over algorithm on the logit transform of the survival ratio between the index cohort and age-, sex, and calendar year matched referents simulated from vital statistics. Lifetime costs for each cancer were estimated by multiplying survival with average monthly costs after adjustment with annual discount rate. The loss-of-QALE was estimated by the difference in QALE between the index cancer cohort and corresponding referents.

RESULTS

The dynamic changes and weighted averages of the QoL utility values of 14 different cancers ranged from 0.82 to 0.95. Successful prevention of liver, lung, esophagus, or nasopharynx cancer would save more than 10 quality-adjusted life years and more than 21,000 USD per case for both genders. Since the saving of loss-of-QALE was adjusted for different age, sex, and calendar-year distributions, it could be used in cost effectiveness evaluation.

CONCLUSION

Savings of loss-of-QALE and lifetime costs could be used for comparison of prevention, diagnosis, treatment and rehabilitation from a lifetime horizon.

Common Problems, Common Data Model Solutions: Evidence Generation for Health Technology Assessment

There is growing interest in using observational data to assess the safety, effectiveness, and cost effectiveness of medical technologies, but operational, technical, and methodological challenges limit its more widespread use. Common data models and federated data networks offer a potential solution to many of these problems. The open-source Observational and Medical Outcomes Partnerships (OMOP) common data model standardises the structure, format, and terminologies of otherwise disparate datasets, enabling the execution of common analytical code across a federated data network in which only code and aggregate results are shared. While common data models are increasingly used in regulatory decision making, relatively little attention has been given to their use in health technology assessment (HTA). We show that the common data model has the potential to facilitate access to relevant data, enable multidatabase studies to enhance statistical power and transfer results across populations and settings to meet the needs of local HTA decision makers, and validate findings. The use of open-source and standardised analytics improves transparency and reduces coding errors, thereby increasing confidence in the results. Further engagement from the HTA community is required to inform the appropriate standards for mapping data to the common data model and to design tools that can support evidence generation and decision making.

Cost-Effectiveness of Direct Non-Vitamin K Oral Anticoagulants Versus Vitamin K Antagonists for the Management of Patients with Non-Valvular Atrial Fibrillation Based on Available "Real-World" Evidence: The Italian National Health System Perspective

BACKGROUND AND OBJECTIVE

The increasing availability of real-world evidence (RWE) about safety and effectiveness of direct non-vitamin K oral anticoagulants (DOACs) for the management of atrial fibrillation (AF) offers the opportunity to better understand the clinical and economic implications of DOACs versus vitamin K antagonists (VKAs). The objective of this study was to compare the economic implications of DOACs and VKAs using data from real-world evidence in patients with AF.

METHODS

A Markov model simulating the lifetime course of patients diagnosed with non-valvular AF was used to evaluate the cost-effectiveness of DOACs (i.e., rivaroxaban, dabigatran and apixaban) versus VKAs from the Italian National Health System (INHS) perspective. The model was made up of data from the literature and a meta-analysis of RWE on the incidence of stroke/systemic embolism (SE), major bleeding (MB), intracranial haemorrhage (ICH) and all-cause mortality (ACM); direct costs included drug costs, costs for drug monitoring, and management of events from official national lists. One-way and probabilistic sensitivity analyses (PSA) were used to assess the robustness of the results.

RESULTS

Results from the meta-analysis showed that apixaban had a high probability of being the most effective for stroke/SE, MB and ACM. Despite their higher acquisition costs, the cost-effectiveness analysis showed all DOACs involved a saving when compared with VKAs, with per-patient savings ranging between €4647 (rivaroxaban) to €6086 (apixaban). Moreover, all DOACs indicated a gain both in quality-adjusted life-years and life-years. According to PSA, findings related to apixaban were consistent, while for dabigatran and rivaroxaban PSA revealed a higher degree of uncertainty.

CONCLUSIONS

The beneficial effect of DOACs on containing events showed in RWE had the potential to offset drug-related costs, thus improving the sustainability of treatment for non-valvular AF in daily clinical practice.

Real-world evidence: perspectives on challenges, value, and alignment of regulatory and national health technology assessment data collection requirements

OBJECTIVE

This study aims to assess stakeholder perceptions on the challenges and value of real-world evidence (RWE) post approval, the differences in regulatory and health technology assessment (HTA) real-world data (RWD) collection requirements under the German regulation for more safety in drug supply (GSAV), and future alignment opportunities to create a complementary framework for postapproval RWE requirements.

METHODS

Eleven semistructured interviews were conducted purposively with pharmaceutical industry experts, regulatory authorities, health technology assessment bodies (HTAbs), and academia. The interview questions focused on the role of RWE post approval, the added value and challenges of RWE, the most important requirements for RWD collection, experience with registries as a source of RWD, perceptions on the GSAV law, RWE requirements in other countries, and the differences between regulatory and HTA requirements and alignment opportunities. The interviews were recorded, transcribed, and translated for coding in Nvivo to summarize the findings.

RESULTS

All experts agree that RWE could close evidence gaps by showing the actual value of medicines in patients under real-world conditions. However, experts acknowledged certain challenges such as: (i) heterogeneous perspectives and differences in outcome measures for RWE generation and (ii) missing practical experience with RWD collected through mandatory registries within the German benefit assessment due to an unclear implementation of the GSAV.

CONCLUSIONS

This study revealed that all stakeholder groups recognize the added value of RWE but experience conflicting demands for RWD collection. Harmonizing requirements can be achieved through common postlicensing evidence generation (PLEG) plans and joint scientific advice to address uncertainties regarding evidence needs and to optimize drug development.

Low rate of subsequent surgery and serious complications following intra-articular steroid injection for base of thumb osteoarthritis: national cohort analysis

Objectives

Intra-articular steroid injection is commonly used to treat base of thumb osteoarthritis (BTOA), despite a lack of large-scale data on safety and effectiveness. We estimate the incidence of serious complications and further procedures following BTOA injection, including the risk of post-operative serious surgical site infection for subsequent operative intervention.

Methods

Hospital Episode Statistics data linked to mortality records from 1 April 1998 to 31 March 2017 were used to identify all BTOA injections undertaken in adults in the National Health Service secondary care in England. Patients were followed up longitudinally until death or 31 March 2017. A multivariable regression with a Fine and Gray model adjusting for the competing risk of mortality in addition to age, sex and socioeconomic deprivation was used to identify factors associated with progression to further procedure. Secondary outcomes included serious complications after injection and subsequent surgical site infection.

Results

A total of 19 120 primary injections were performed during the 19-year period in 18 356 patients. Of these 76.5% were female; mean age 62 years (s.d. 10.6); 50.48% underwent further procedure; 22.40% underwent surgery. Median time to further intervention was 412 days (IQR 110–1945). Female sex was associated with increased risk of proceeding to surgery. Serious complication rate following injection was 0.04% (0.01–0.08) within 90 days. Of those proceeding to surgery 0.16% (0.06–0.34) presented with a wound infection within 30 days and 90 days, compared with an overall post-operative wound infection rate of 0.03% (0.02–0.05).

Conclusions

Very low rates of serious complications were identified following BTOA injections performed in secondary care; only one in five patients proceeded to subsequent surgery.

Clinical trial registration

clinicaltrials.gov, https://www.clinicaltrials.gov, NCT03573765

Serious postoperative complications and reoperation after carpal tunnel decompression surgery in England: a nationwide cohort analysis

BACKGROUND

Carpal tunnel decompression surgery to treat carpal tunnel syndrome is a common procedure, yet data on safety and effectiveness of the operation in the general population remain scarce. We aimed to estimate the incidence of reoperation and serious postoperative complications (requiring admission to hospital or further surgery) following carpal tunnel decompression in routine clinical practice and to identify the patient factors associated with these adverse outcomes.

METHODS

We did a nationwide cohort analysis including all carpal tunnel decompression surgeries in patients aged 18 years or older, done in the National Health Service in England between April 1, 1998, and March 31, 2017, using the Hospital Episode Statistics dataset linked to mortality records. Patients were followed-up until death or until the end of the study (March 31, 2017). Primary outcomes were the overall incidence of carpal tunnel decompression reoperation and serious postoperative complications (surgical site infection or dehiscence, or neurovascular or tendon injury, requiring admission to hospital or further surgery) within 30 days and 90 days after surgery. Multivariable Cox regression analysis was used to identify factors influencing complications and reoperation, and the Fine and Gray method was used to adjust for the competing risk of mortality. This study is registered with ClinicalTrials.gov, NCT03573765.

FINDINGS

855 832 carpal tunnel decompression surgeries were done between April 1, 1998, and March 31, 2017 (incidence rate 1·10 per 1000 person-years [95% CI 1·02-1·17]). 29 288 procedures (3·42%) led to carpal tunnel decompression reoperation (incidence rate 3·18 per 1000 person-years [95% CI 3·12-3·23]). Of the 855 832 initial surgeries, 620 procedures (0·070% [95% CI 0·067-0·078]) led to a serious complication within 30 days after surgery, and 698 procedures (0·082% [0·076-0·088]) within 90 days. Local complications within 90 days after surgery were associated with male sex (adjusted hazard ratio 2·32 [95% CI 1·74-3·09]) and age category 18-29 years (2·25 [1·10-4·62]). Male sex (adjusted subhazard ratio 1·09 [95% CI 1·06-1·13]), old age (>80 years vs 50-59 years: 1·09 [1·03-1·15]), and greater levels of comorbidity (Charlson score ≥5 vs 0: 1·25 [1·19-1·32]) and socioeconomic deprivation (most deprived 10% vs least deprived 10%: 1·18 [1·10-1·27]) were associated with increased reoperation risk.

INTERPRETATION

To our knowledge, this is the largest national study on carpal tunnel decompression to date, providing strong evidence on serious postoperative complication and reoperation rates. Carpal tunnel decompression appears to be a safe operation in most patients, with an overall serious complication rate (requiring admission to hospital or further surgery) of less than 0·1%.

FUNDING

Versus Arthritis; Medical Research Council; Royal College of Surgeons of England and National Joint Registry research fellowship; University of Oxford; National Institute for Health Research; and National Institute for Health Research Biomedical Research Centre, Oxford.

Real-world effectiveness of olanzapine and risperidone in the treatment of schizophrenia in Brazil over a 16-year follow-up period; findings and implications

Introduction: Antipsychotics are widely prescribed for patients with schizophrenia. The Brazilian public health system provides these patients free of charge to patients and it is pertinent to evaluate their benefits.Objective: To evaluate the effectiveness of olanzapine and risperidone in the treatment of patients with schizophrenia in the real-world and assessing risk factors for their discontinuation through a national non-concurrent cohort with 16 years of follow-up.Methods: Three SUS administrative databases were integrated by deterministic-probabilistic linkage. After patients were matched (1:1) for psychiatric hospitalization, year of receiving the antipsychotic, sex, and age, considering either olanzapine or risperidone at study entry. Kaplan-Meier was used to estimate the cumulative probabilities of discontinuation of treatment and associated factors were identified. Sensitivity analyses were performed.Results: 3416 pairs of patients were included. Olanzapine had a longer time until discontinuation of treatment (p = 0.021), and risperidone had a higher risk of discontinuation (p = 0.021). Among patients persistent for at least 24 months, there was no statistically significant difference.Conclusion: Olanzapine demonstrated superior real-world effectiveness over risperidone, in terms of survival and psychiatric hospitalization. This superiority was not sustained in all analyses.

Use of real-world evidence in economic assessments of pharmaceuticals in the United States

BACKGROUND: Despite the increasing interest in expanding the use of real-world evidence (RWE) in economic assessments of pharmaceuticals, decision makers face uncertainty about how RWE should be used. OBJECTIVE: To assess the use of RWE in economic assessments of drugs by the Institute for Clinical and Economic Review (ICER). METHODS: We reviewed cost-effectiveness and budget impact analyses in final evidence reports of pharmaceuticals published by ICER. We calculated the total number of RWE uses and the proportion of model inputs informed by RWE per report. We classified model inputs into 15 categories based on their attributes and then examined what category each RWE informed to classify the reason for RWE use. Finally, we characterized RWE by study design, data source, and sponsor type. RESULTS: We identified 33 reports, all of which used RWE; the mean RWE use per report was 12 (range = 4-26). The average proportion of model inputs informed by RWE per report was 32.7%, but this proportion had a wide range (range = 4.1%-76.9%). RWE was most commonly used for disease progression inputs (28.7%) and health care resource utilization and costs (21.1%), but was rarely used for drug-specific clinical outcomes such as effectiveness (1.5%), adverse drug event rates (0.5%), and discontinuation rates (1.2%). The most frequently used study design was a retrospective cohort (56.6%), and the most frequently used data source was registry data (41.4%). About a third (30.2%) of RWE was industry-sponsored. CONCLUSIONS: RWE has been commonly used to inform pharmaceutical value assessments conducted by ICER. However, there has been relatively limited use of RWE to inform drug-specific effectiveness, despite calls for greater inclusion of RWE in value assessments for real-world drug effectiveness. DISCLOSURES: This study was funded by the University of Washington School of Pharmacy Corporate Advisory Board Health Tech Fund (HTF). The funder had no role in management, analysis, and interpretation of the data; preparation, review, and approval of the manuscript; and the decision to submit the manuscript for publication. All authors were employed by the CHOICE Institute, University of Washington School of Pharmacy, at the time of the study. Carlson reports grants from the Institute for Clinical and Economic Review during the conduct of the study and personal fees from Bayer, Adaptive Biotechnologies, Allergan, Galderma, and ViFor Pharma, unrelated to this study. Veenstra reports personal fees from several manufacturers unrelated to this study. The other authors have nothing to disclose.

Research protocol for the validation of a new portable technology for real-time continuous monitoring of Early Warning Score (EWS) in hospital practice and for an early-stage multistakeholder assessment

INTRODUCTION

The real-time continuous monitoring of vital parameters in patients affected by multiple chronic conditions and/or COVID-19 can lead to several benefits to the Italian National Healthcare System (IT-NHS). The UBiquitous Integrated CARE (UBICARE) technology is a novel health digital platform at the validation stage in hospital setting. UBICARE might support the urgent need for digitalisation and early intervention, as well as minimise the face-to-face delivery of care in both hospital and community-based care settings. This research protocol aims to design an early-stage assessment of the multidimensional impact induced by UBICARE within the IT-NHS alongside technology validation in a hospital ward.

METHODS AND ANALYSIS

The targeted patients will be medium/high-risk hypertensive individuals as an illustrative first example of how UBICARE might bring benefits to susceptible patients. A mixed-method study will be applied to incorporate to the validation study a multistakeholder perspective, including perceived patient experiences and preferences, and facilitate technology adoption. First, semistructured interviews will be undertaken with a variety of stakeholders including clinicians, health managers and policy-makers to capture views on the likely technology utility, economic sustainability, impact of adoption in hospital practice and alternative adoption scenarios. Second, a monocentric, non-randomised and non-comparative clinical study, supplemented by the administration of standardised usability questionnaires to patients and health professionals, will validate the use of UBICARE in hospital practice. Finally, the results of the previous stages will be discussed in a multidisciplinary-facilitated workshop with IT-NHS relevant stakeholders to reconcile stakeholders' perspectives. Limitations include a non-random recruitment strategy in the clinical study, small sample size of the key stakeholders and potential stakeholder recruitment bias introduced by the research technique.

ETHICS AND DISSEMINATION

The Ethics Committee for Clinical Experimentation of Tuscany Region approved the protocol. Participation in this study is voluntary. Study results will be disseminated through peer-reviewed publications and academic conferences.

A review of global health technology assessments of non-VKA oral anticoagulants in non-valvular atrial fibrillation

BACKGROUND

This review assessed global health technology assessment (HTA) reports and recommendations of non-vitamin K oral anticoagulants (NOACs) in non-valvular atrial fibrillation (NVAF).

METHODS

NHTA agency websites were searched for HTA reports evaluating NOACs versus NOACs or vitamin K antagonists. HTA methods and information on patient involvement/access were collected and empirically analyzed.

RESULTS

The review identified 38 unique HTA reports published between 2012 and 2017 in 16 countries including 11 in Europe. NOACs that were cost-effective per local willingness-to-pay (WTP) thresholds were positively recommended for the treatment of NVAF. WTP thresholds ranged from €20,000 to 69,000. Apixaban was recommended in 10/12 (83%) countries, dabigatran in 9/13 (69%) countries, and rivaroxaban in 10/13 (76%) over warfarin. Edoxaban was recommended in 5/7 (71%) countries. Economic evaluations and recommendations comparing NOACs were sparse (two or three countries per NOAC) and generally favored apixaban and edoxaban, followed by dabigatran. Eleven HTA reports from four countries considered the patient voice (Canada [n = 3], Scotland [n = 3], England [n = 4], Brazil [n = 1]); however, only 2/11 (18%) developed recommendations based on this. Among the reports with a positive recommendation, 26/30 (87%) featured a decision that aligned with the approved regulatory label.

CONCLUSIONS

Most agencies recommended NOACs over warfarin for patients with NVAF. Few countries made statements recommending one NOAC over another. Given different WTP thresholds, a drug that is cost-effective in one market may not be in another. Therefore, the various NOAC recommendations from HTA agencies cannot be generalized across different countries.

Assessment of Studies Evaluating Incremental Costs, Effectiveness, or Cost-Effectiveness of Systemic Therapies in Breast Cancer Based on Claims Data: A Systematic Review

OBJECTIVES

Large secondary databases, such as those containing insurance claims data, are increasingly being used to compare the effects and costs of treatments in routine clinical practice. Despite their appeal, however, caution must be exercised when using these data. In this study, we aimed to identify and assess the methodological quality of studies that used claims data to compare the effectiveness, costs, or cost-effectiveness of systemic therapies for breast cancer.

METHODS

We searched Embase, the Cochrane Library, Medline, Web of Science, and Google Scholar for English-language publications and assessed the methodological quality using the Good Research for Comparative Effectiveness principles. This study was registered with the International Prospective Register of Systematic Reviews (PROSPERO) under number CRD42018103992.

RESULTS

We identified 1251 articles, of which 106 met the inclusion criteria. Most studies were conducted in the United States (74%) and Taiwan (9%) and were based on claims data sets (35%) or claims data linked to cancer registries (58%). Furthermore, most included large samples (mean 17 130 patients) and elderly patients, and they covered various outcomes (eg, survival, adverse events, resource use, and costs). Key methodological shortcomings were the lack of information on relevant confounders, the risk of immortal time bias, and the lack of information on the validity of outcomes. Only a few studies performed sensitivity analyses.

CONCLUSIONS

Many comparative studies of cost, effectiveness, and cost-effectiveness have been published in recent decades based on claims data, and the number of publications has increased over time. Despite the availability of guidelines to improve quality, methodological issues persist and are often inappropriately addressed or reported.

Characterizing the Validity and Real-World Utility of Health Technology Assessments in Healthcare: Future Directions Comment on "Problems and Promises of Health Technologies: The Role of Early Health Economic Modelling"

With their article, Grutters et al raise an important question: What do successful health technology assessments (HTAs) look like, and what is their real-world utility in decision-making? While many HTAs are published in peer-reviewed journals, many are considered proprietary and their attributes remain confidential, limiting researchers’ ability to answer these questions. Models for economic evaluations like cost-effectiveness analyses (CEAs) synthesize a wide range of evidence, are often statistically and mathematically sophisticated, and require untestable assumptions. As such, there is nearly universal agreement among researchers that enhancing transparency is an important issue in health economic modeling. However, the definition of transparency and guidelines for its implementation vary. Model registration combined with a linked database of model-based economic evaluations has been proposed as a solution, whereby registered models and their accompanying technical and nontechnical documentation are sourced into a single publicly-available repository, ideally in a standardized format to ensure consistent and complete representation of features, code, data sources, results, validation exercises, and policy recommendations. When such a repository is ultimately created, modelers will not have to reinvent the wheel for every new drug launched or new treatment pathway. These more open and transparent approaches will have substantial implications for model accuracy, reliability, and validity, improving trust and acceptance by healthcare decision-makers.

Real-world evidence use in assessments of cancer drugs by NICE

OBJECTIVE

To establish how real-world evidence (RWE) has been used to inform single technology appraisals (STAs) of cancer drugs conducted by the National Institute for Health and Care Excellence (NICE).

METHODS

STAs published by NICE from April 2011 to October 2018 that evaluated cancer treatments were reviewed. Information regarding the use of RWE to directly inform the company-submitted cost-effectiveness analysis was extracted and categorized by topic. Summary statistics were used to describe emergent themes, and a narrative summary was provided for key case studies.

RESULTS

Materials for a total of 113 relevant STAs were identified and analyzed, of which nearly all (96 percent) included some form of RWE within the company-submitted cost-effectiveness analysis. The most common categories of RWE use concerned the health-related quality of life of patients (71 percent), costs (46 percent), and medical resource utilization (40 percent). While sources of RWE were routinely criticized as part of the appraisal process, we identified only two cases where the use of RWE was overtly rejected; hence, in the majority of cases, RWE was accepted in cancer drug submissions to NICE.

DISCUSSION

RWE has been used extensively in cancer submissions to NICE. Key criticisms of RWE in submissions to NICE are seldom regarding the use of RWE in general; instead, these are typically concerned with specific data sources and the applicability of these to the decision problem. Within an appropriate context, RWE constitutes an extremely valuable source of information to inform decision making; yet the development of best practice guidelines may improve current reporting standards.

Basal thumb osteoarthritis surgery improves health state utility irrespective of technique: a study of UK Hand Registry data

We used UK Hand Registry data to study two aspects of basal thumb osteoarthritis surgery: first, whether health-related quality of life improves after surgery. Second, whether results from trials comparing simple trapeziectomy and trapeziectomy with ligament reconstruction and tendon interposition are reproducible in routine clinical practice. Prospectively collected EQ5D index and Patient Evaluation Measure part 2 data were compared at baseline and at 3, 6, and 12 months postoperatively in 1456 patients (median age 67 years; 78% female). A mixed-effects regression model was also used to determine the postoperative trajectory of these variables. There was a significant improvement in the EQ5D index (median + 0.15; (interquartile range 0 to 0.40)) and Patient Evaluation Measure (-22; (-33 to -10)) by 1 year postoperatively and with no meaningful difference between the two techniques. This study demonstrates health state utility gains after basal thumb osteoarthritis surgery regardless of surgical techniques used. Level of evidence: III.

A Narrative Review and Proposed Framework for Using Health System Data with Systematic Reviews to Support Decision-making

Systematic reviews are a necessary, but often insufficient, source of information to address the decision-making needs of health systems. In this paper, we address when and how the use of health system data might make systematic reviews more useful to decision-makers. We describe the different ways in which health system data can be used with systematic reviews, identify scenarios in which the addition of health system data may be most helpful (i.e., to improve the strength of evidence, to improve the applicability of evidence, and to inform the implementation of evidence), and discuss the importance of framing the limitations and considerations when using unpublished health system data in reviews. We developed a framework to guide the use of health system data alongside systematic reviews based on a narrative review of the literature and empirical experience. We also offer recommendations to improve the transparency of reporting when using health system data alongside systematic reviews including providing rationale for employing additional data, details on the data source, critical appraisal to understand study design biases as well as limitations in data and information quality, and how the unpublished data compares to the systematically reviewed data. Future methodological work on how best to handle internal and external validity concerns of health system data in the context of systematically reviewed data and work on developing infrastructure to do this type of work is needed.

The Lean and Agile Multi-dimensional Process (LAMP) - a new framework for rapid and iterative evidence generation to support health-care technology design and development

Introduction: Health technology assessments (HTA) are tools for policymaking and resource allocation. Early HTAs are increasingly used in design and development of new technologies. Conducting early HTAs is challenging, due to a lack of evidence and significant uncertainties in the technology and the market. A multi-disciplinary approach is considered essential. However, an operational framework that can enable the integration of multi-dimensional evidence into commercialization remains lacking.Areas covered: We developed the Lean and Agile Multi-dimensional Process (LAMP), an early HTA framework, for embedding commercial decision-making in structured evidence generation activities, divided into phases. Diverse evidence in unmet needs, user acceptance, cost-effectiveness, and market competitiveness are being generated in increasing depth. This supports the emergence of design and value propositions that align technology capabilities and clinical and user needs.Expert opinion: We have been applying LAMP to working with medical device and diagnostic industry in the UK. The framework can be adapted to suit different technologies, decision needs, time scales, and resources. LAMP offers a practical solution to the multi-disciplinary approach. Methodologists drive the process by performing evidence generation and synthesis as and by enabling interactions between manufacturers, designers, clinicians, and other key stakeholders.

Results dissemination of registered clinical trials across Polish academic institutions: a cross-sectional analysis

OBJECTIVES

To establish the rates of publication and reporting of results for interventional clinical trials across Polish academic medical centres (AMCs) completed between 2009 and 2013. We aim also to compare the publication and reporting success between adult and paediatric trials.

DESIGN

Cross-sectional study.

SETTING

AMCs in Poland.

PARTICIPANTS

AMCs with interventional trials registered on ClinicalTrials.gov.

MAIN OUTCOME MEASURE

Results reporting on ClinicalTrials.gov and publishing via journal publication.

RESULTS

We identified 305 interventional clinical trials registered on ClinicalTrials.gov, completed between 2009 and 2013 and affiliated with at least one AMC. Overall, 243 of the 305 trials (79.7%) had been published as articles or posted their summary results on ClinicalTrials.gov. Results were posted within a year of study completion and/or published within 2 years of study completion for 131 trials (43.0%). Dissemination by both posting and publishing results in a timely manner was achieved by four trials (1.3%).

CONCLUSIONS

Our cross-sectional analysis revealed that Polish AMCs fail to meet the expectation for timely disseminating the findings of all interventional clinical trials. Delayed dissemination and non-dissemination of trial results negatively affects decisions in healthcare.