Protocol for data extraction: how real-world data have been used in the National Institute for Health and Care Excellence appraisals of cancer therapy

Real-world evidence in the reassessment of oncology therapies: payer perceptions from five countries

Aim: This study explored the perceived value of real-world evidence (RWE) in the reassessment of oncology therapies by collecting the perspectives of health technology assessment/payer decision-makers.Materials & methods: A web-based survey was conducted using the Market Access Transformation Rapid Payer Response online portal. 30 participants from France, Germany, Spain, the UK and the USA were recruited based on their expertise.Results: Participants agreed that the most common uses of RWE are to confirm efficacy and safety results from randomized controlled trials and to reevaluate the projected utilization of an oncology therapy. We found variability in other reported uses of RWE.Conclusion: The organizations developing RWE should ensure that their plans recognize the heterogeneity in payer perceptions.

Cross-sectional analysis of use of real-world data in single technology appraisals of oncological medicine by the National Institute for Health and Care Excellence in 2011-2021

OBJECTIVES

This study aims to identify how real-world data (RWD) have been used in single technology appraisals (STAs) of cancer drugs by the National Institute for Health and Care Excellence (NICE).

DESIGN

Cross-sectional study of NICE technology appraisals of cancer drugs for which guidance was issued between January 2011 and December 2021 (n=229). The appraisals were reviewed following a published protocol to extract the data about the use of RWD. The use of RWD was analysed by reviewing the specific ways in which RWD were used and by identifying different patterns of use.

PRIMARY OUTCOME MEASURE

The number of appraisals where RWD are used in the economic modelling.

RESULTS

Most appraisals used RWD in their economic models. The parametric use of RWD was commonly made in the economic models (76% of the included appraisals), whereas non-parametric use was less common (41%). Despite widespread use of RWD, there was no dominant pattern of use. Three sources of RWD (registries, administrative data, chart reviews) were found across the three important parts of the economic model (choice of comparators, overall survival and volume of treatment).

CONCLUSIONS

NICE has had a long-standing interest in the use of RWD in STAs. A systematic review of oncology appraisals suggests that RWD have been widely used in diverse parts of the economic models. Between 2011 and 2021, parametric use was more commonly found in economic models than non-parametric use. Nonetheless, there was no clear pattern in the way these data were used. As each appraisal involves a different decision problem and the ability of RWD to provide the information required for the economic modelling varies, appraisals will continue to differ with respect to their use of RWD.

Using automated text classification to explore uncertainty in NICE appraisals for drugs for rare diseases

OBJECTIVE

This study examined the application, feasibility, and validity of supervised learning models for text classification in appraisals for rare disease treatments (RDTs) in relation to uncertainty, and analyzed differences between appraisals based on the classification results.

METHODS

We analyzed appraisals for RDTs (n = 94) published by the National Institute for Health and Care Excellence (NICE) between January 2011 and May 2023. We used Naïve Bayes, Lasso, and Support Vector Machine models in a binary text classification task (classifying paragraphs as either referencing uncertainty in the evidence base or not). To illustrate the results, we tested hypotheses in relation to the appraisal guidance, advanced therapy medicinal product (ATMP) status, disease area, and age group.

RESULTS

The best performing (Lasso) model achieved 83.6 percent classification accuracy (sensitivity = 74.4 percent, specificity = 92.6 percent). Paragraphs classified as referencing uncertainty were significantly more likely to arise in highly specialized technology (HST) appraisals compared to appraisals from the technology appraisal (TA) guidance (adjusted odds ratio = 1.44, 95 percent CI 1.09, 1.90, p = 0.004). There was no significant association between paragraphs classified as referencing uncertainty and appraisals for ATMPs, non-oncology RDTs, and RDTs indicated for children only or adults and children. These results were robust to the threshold value used for classifying paragraphs but were sensitive to the choice of classification model.

CONCLUSION

Using supervised learning models for text classification in NICE appraisals for RDTs is feasible, but the results of downstream analyses may be sensitive to the choice of classification model.

Effectively Leveraging RWD for External Controls: A Systematic Literature Review of Regulatory and HTA Decisions

Real-world data (RWD)-derived external controls can be used to contextualize efficacy findings for investigational therapies evaluated in uncontrolled trials. As the number of submissions to regulatory and health technology assessment (HTA) bodies using external controls rises, and in light of recent regulatory and HTA guidance on the appropriate use of RWD, there is a need to address the operational and methodological challenges impeding the quality of real-world evidence (RWE) generation and the consistency in evaluation of RWE across agencies. This systematic review summarizes publicly available information on the use of external controls to contextualize outcomes from uncontrolled trials for all indications from January 1, 2015, through August 20, 2021, that were submitted to the European Medicines Agency, the US Food and Drug Administration, and/or select major HTA bodies (National Institute for Health and Care Excellence (NICE), Haute Autorité de Santé (HAS), Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (IQWiG), and Gemeinsamer Bundesausschuss (G-BA)). By systematically reviewing submissions to regulatory and HTA bodies in the context of recent guidance, this study provides quantitative and qualitative insights into how external control design and analytic choices may be viewed by different agencies in practice. The primary operational and methodological aspects identified for discussion include, but are not limited to, engagement of regulators and HTA bodies, approaches to handling missing data (a component of data quality), and selection of real-world endpoints. Continued collaboration and guidance to address these and other aspects will inform and assist stakeholders attempting to generate evidence using external controls.

"Don't Think Twice, It's All Right": Using Additional Data to Reduce Uncertainty Regarding Oncologic Drugs Provided Through Managed Access Agreements in England

OBJECTIVES

The Cancer Drugs Fund (CDF) in England uses managed access agreements to facilitate additional data collection to address uncertainties identified in the appraisals of new drugs. This study reviews the uncertainties highlighted in the original appraisals where recommendations "to use within the CDF" were made and how additional data were used to address these uncertainties in the CDF review appraisals where final decisions on routine commissioning were made.

METHODS

The first 24 drugs exiting the 2016 CDF were included in this review. The information about uncertainty and the use of newly collected data were extracted from the original appraisals and the CDF review appraisals. The additional data used in the CDF review appraisals, distinguishing between clinical trial data and real-world data (RWD), were reviewed to assess the extent to which the additional data were able to reduce the original uncertainties.

RESULTS

The recommendation that the drug be routinely commissioned was made in 87.5% of re-appraisals. Uncertainty stemming from immaturity of the survival data in clinical trials was frequently found in appraisals. Later follow-up of clinical trials was used to address this uncertainty, whereas limited use was made of RWD. The Systemic Anti-Cancer Therapy (SACT) dataset is the most frequently used source of RWD. SACT data were mostly used in review appraisals to support the clinical outcomes based on later follow-up of trial participants and to inform modelling of subsequent treatments or treatment duration.

CONCLUSIONS

While additionally collected RWD attracted attention when the 2016 CDF was introduced, RWD have not been widely used in CDF review appraisals and (to date) have done little to reduce uncertainty. Experience with these appraisals has highlighted the importance of longer follow-up of clinical trials and the relatively limited role of RWD, in general, and of SACT data in particular.

Exploring uncertainty and use of real-world data in the National Institute for Health and Care Excellence single technology appraisals of targeted cancer therapy

OBJECTIVES

Dealing with uncertainty is one of the critical topics in health technology assessment. The greater decision uncertainty in appraisals, the less clear the clinical- and cost-effectiveness of the health technology. Although the development of targeted cancer therapies (TCTs) has improved patient health care, additional complexity has been introduced in drug appraisals due to targeting more specific populations. Real-world data (RWD) are expected to provide helpful information to fill the evidence gaps in appraisals. This study compared appraisals of TCTs with those of non-targeted cancer therapies (non-TCTs) regarding sources of uncertainty and reviewed how RWD have been used to supplement the information in these appraisals.

METHODS

This study reviews single technology appraisals (STAs) of oncology medicines performed by the National Institute for Health and Care Excellence (NICE) over 11 years up to December 2021. Three key sources of uncertainty were identified for comparison (generalisability of clinical trials, availability of direct treatment comparison, maturity of survival data in clinical trials). To measure the intensity of use of RWD in appraisals, three components were identified (overall survival, volume of treatment, and choice of comparators).

RESULTS

TCTs received more recommendations for provision through the Cancer Drugs Fund (27.7, 23.6% for non-TCT), whereas similar proportions were recommended for routine commissioning. With respect to sources of uncertainty, the external validity of clinical trials was greater in TCT appraisals (p = 0.026), whereas mature survival data were available in fewer TCT appraisals (p = 0.027). Both groups showed similar patterns of use of RWD. There was no clear evidence that RWD have been used more intensively in appraisals of TCT.

CONCLUSIONS

Some differences in uncertainty were found between TCT and non-TCT appraisals. The appraisal of TCT is generally challenging, but these challenges are neither new nor distinctive. The same sources of uncertainty were often found in the non-TCT appraisals. The uncertainty when appraising TCT stems from insufficient data rather than the characteristics of the drugs. Although RWD might be expected to play a more active role in appraisals of TCT, the use of RWD has generally been limited.

Real-World Outcomes of Systemic Therapy in Japanese Patients with Cancer (Tokushukai REAl-World Data Project: TREAD): Study Protocol for a Nationwide Cohort Study

Cohort studies using large-scale databases have become increasingly important in recent years. The Tokushukai Medical Group is a leading medical group in Japan that includes 71 general hospitals nationwide from Hokkaido to Okinawa, with a total of 18,000 beds, and a unified electronic medical record system. This retrospective cohort study aims to evaluate the real-world outcomes of systemic therapy for Japanese patients with cancer using this merit of scale. All adult patients with cancer who received systemic therapy using a centrally registered chemotherapy protocol system at 46 hospitals from April 2010 to March 2020 will be identified (~48,850 patients). Key exclusion criteria include active double cancer and inadequate data extraction. Data will be obtained through electronic medical records, diagnosis procedure combination data, medical prescription data, and the national cancer registration system that includes sociodemographic variables, diagnostic and laboratory tests, concomitant drug prescriptions, cost, and overall survival. Kaplan–Meier estimates will be calculated for time-to-event analyses. Stratified/conventional Cox proportional hazards regression analyses will be conducted to examine the relationships between overall survival and related factors. Our findings provide important insights for future research directions, policy initiatives, medical guidelines, and clinical decision-making.