Evidence Review Group approaches to the critical appraisal of manufacturer submissions for the NICE STA process: a mapping study and thematic analysis

Adaptive Health Technology Assessment: A Scoping Review of Methods

OBJECTIVES

Health technology assessment (HTA) is an established mechanism for explicit priority setting to support universal health coverage. However, full HTA requires significant time, data, and capacity for each intervention, which limits the number of decisions it can inform. Another approach systematically adapts full HTA methods by leveraging HTA evidence from other settings. We call this "adaptive" HTA (aHTA), although in settings where time is the main constraint, it is also called "rapid HTA."

METHODS

The objectives of this scoping review were to identify and map existing aHTA methods, and to assess their triggers, strengths, and weaknesses. This was done by searching HTA agencies' and networks' websites, and the published literature. Findings have been narratively synthesized.

RESULTS

This review identified 20 countries and 1 HTA network with aHTA methods in the Americas, Europe, Africa, and South-East Asia. These methods have been characterized into 5 types: rapid reviews, rapid cost-effectiveness analyses, rapid manufacturer submissions, transfers, and de facto HTA. Three characteristics "trigger" the use of aHTA instead of full HTA: urgency, certainty, and low budget impact. Sometimes, an iterative approach to selecting methods guides whether to do aHTA or full HTA. aHTA was found to be faster and more efficient, useful for decision makers, and to reduce duplication. Nevertheless, there is limited standardization, transparency, and measurement of uncertainty.

CONCLUSIONS

aHTA is used in many settings. It has potential to improve the efficiency of any priority-setting system, but needs to be better formalized to improve uptake, particularly for nascent HTA systems.

Conducting Value for Money Analyses for Non-randomised Interventional Studies Including Service Evaluations: An Educational Review with Recommendations

This article provides an educational review covering the consideration of conducting ‘value for money’ analyses as part of non-randomised study designs including service evaluations. These evaluations represent a vehicle for producing evidence such as value for money of a care intervention or service delivery model. Decision makers including charities and local and national governing bodies often rely on evidence from non-randomised data and service evaluations to inform their resource allocation decision-making. However, as randomised data obtained from randomised controlled trials are considered the ‘gold standard’ for assessing causation, the use of this alternative vehicle for producing an evidence base requires careful consideration. We refer to value for money analyses, but reflect on methods associated with economic evaluations as a form of analysis used to inform resource allocation decision-making alongside a finite budget. Not all forms of value for money analysis are considered a full economic evaluation with implications for the information provided to decision makers. The type of value for money analysis to be conducted requires considerations such as the outcome(s) of interest, study design, statistical methods to control for confounding and bias, and how to quantify and describe uncertainty and opportunity costs to decision makers in any resulting value for money estimates. Service evaluations as vehicles for producing evidence present different challenges to analysts than what is commonly associated with research, randomised controlled trials and health technology appraisals, requiring specific study design and analytic considerations. This educational review describes and discusses these considerations, as overlooking them could affect the information provided to decision makers who may make an ‘ill-informed’ decision based on ‘poor’ or ‘inaccurate’ information with long-term implications. We make direct comparisons between randomised controlled trials relative to non-randomised data as vehicles for assessing causation; given ‘gold standard’ randomised controlled trials have limitations. Although we use UK-based decision makers as examples, we reflect on the needs of decision makers internationally for evidence-based decision-making specific to resource allocation. We make recommendations based on the experiences of the authors in the UK, reflecting on the wide variety of methods available, used as documented in the empirical literature. These methods may not have been fully considered relevant to non-randomised study designs and/or service evaluations, but could improve and aid the analysis conducted to inform the relevant value for money decision problem.

Comparing the ICERs in Medicine Reimbursement Submissions to NICE and PBAC-Does the Presence of an Explicit Threshold Affect the ICER Proposed?

OBJECTIVES

The English National Institute for Health and Care Excellence (NICE) and the Australian Pharmaceutical Benefits Advisory Committee (PBAC) require evidence that a new medicine represents value for money before being publicly funded. NICE has an explicit threshold for cost effectiveness, whereas PBAC does not. We compared the initial incremental cost-effectiveness ratios (ICERs) presented by manufacturers in matched submissions to each decision-making body, with the aim of exploring the impact of an explicit threshold on these ICERs.

METHODS

Data were extracted from matched submissions from 2005 to 2015. The ICERs in these submissions were compared within each pair and with respect to a cost-effectiveness threshold.

RESULTS

Fifty-eight pairs of matched submissions were identified. The median difference between the ICERs ($2635/quality-adjusted life year [QALY]) was significantly greater than zero (Wilcoxon signed-rank test, P = 0.0299), indicating that the proposed ICERs in the submissions to NICE were higher than those in the matched submissions to PBAC. On 93% of occasions, NICE ICERs were within -$17,772 to +$48,422 of the corresponding PBAC ones (Bland-Altman analysis), demonstrating poor agreement. When an implicit threshold of AUD$50,000/QALY was assumed for PBAC decision making, only eight pairs of submissions had discordant ICERs falling above or below the respective threshold.

CONCLUSIONS

The significantly higher ICERs in the submissions to NICE than those to PBAC may be a consequence of NICE's explicit willingness-to-pay threshold, and/or other health system factors. Industry may be assuming an implicit threshold for PBAC when constructing their ICERs despite the lack of acknowledgement of such a threshold.

Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries

BACKGROUND

Although health technology assessment (HTA) systems base their decision making process either on economic evaluations or comparative clinical benefit assessment, a central aim of recent approaches to value measurement, including value based assessment and pricing, points towards the incorporation of supplementary evidence and criteria that capture additional dimensions of value.

OBJECTIVE

To study the practices, processes and policies of value-assessment for new medicines across eight European countries and the role of HTA beyond economic evaluation and clinical benefit assessment.

METHODS

A systematic (peer review and grey) literature review was conducted using an analytical framework examining: (1) 'Responsibilities and structure of HTA agencies'; (2) 'Evidence and evaluation criteria considered in HTAs'; (3) 'Methods and techniques applied in HTAs'; and (4) 'Outcomes and implementation of HTAs'. Study countries were France, Germany, England, Sweden, Italy, Netherlands, Poland and Spain. Evidence from the literature was validated and updated through two rounds of feedback involving primary data collection from national experts.

RESULTS

All countries assess similar types of evidence; however, the specific criteria/endpoints used, their level of provision and requirement, and the way they are incorporated (e.g. explicitly vs. implicitly) varies across countries, with their relative importance remaining generally unknown. Incorporation of additional 'social value judgements' (beyond clinical benefit assessment) and economic evaluation could help explain heterogeneity in coverage recommendations and decision-making.

CONCLUSION

More comprehensive and systematic assessment procedures characterised by increased transparency, in terms of selection of evaluation criteria, their importance and intensity of use, could lead to more rational evidence-based decision-making, possibly improving efficiency in resource allocation, while also raising public confidence and fairness.

Issues Related to the Frequency of Exploratory Analyses by Evidence Review Groups in the NICE Single Technology Appraisal Process

BACKGROUND

Evidence Review Groups (ERGs) critically appraise company submissions as part of the National Institute for Health and Care Excellence (NICE) Single Technology Appraisal (STA) process. As part of their critique of the evidence submitted by companies, the ERGs undertake exploratory analyses to explore uncertainties in the company's model. The aim of this study was to explore pre-defined factors that might influence or predict the extent of ERG exploratory analyses.

OBJECTIVE

The aim of this study was to explore predefined factors that might influence or predict the extent of ERG exploratory analyses.

METHODS

We undertook content analysis of over 400 documents, including ERG reports and related documentation for the 100 most recent STAs (2009-2014) for which guidance has been published. Relevant data were extracted from the documents and narrative synthesis was used to summarise the extracted data. All data were extracted and checked by two researchers.

RESULTS

Forty different companies submitted documents as part of the NICE STA process. The most common disease area covered by the STAs was cancer (44%), and most ERG reports (n = 93) contained at least one exploratory analysis. The incidence and frequency of ERG exploratory analyses does not appear to be related to any developments in the appraisal process, the disease area covered by the STA, or the company's base-case incremental cost-effectiveness ratio (ICER). However, there does appear to be a pattern in the mean number of analyses conducted by particular ERGs, but the reasons for this are unclear and potentially complex.

CONCLUSIONS

No clear patterns were identified regarding the presence or frequency of exploratory analyses, apart from the mean number conducted by individual ERGs. More research is needed to understand this relationship.

Modeling Treatment Sequences in Pharmacoeconomic Models

As the number of interventions available in a therapeutic area increases, the relevant decision questions in health technology assessment (HTA) expand to compare treatment sequences instead of discrete treatments and identify optimal sequences or position for a particular treatment in a sequence. The objective of this work was to review approaches used to model treatment sequences and provide practical guidance on conceptualizing whether and how to model sequences in health economic models. Economic models including treatment sequencing assessed by the National Institute for Health and Care Excellence were reviewed, as these assessments generally provide both policy relevance and comprehensive model detail. We identified 40 treatment-sequence models in the following disease areas: oncology (13), autoimmune (7), cardiovascular (6), neurology/mental health (4), infectious disease (2), diabetes (2), and other (6). Modeling techniques included discrete event simulation (6), individual state-transition (3), decision tree (3) and, most commonly, cohort state-transition with tracking states (28). In most cases, treatment sequencing had been incorporated to reflect either clinical practice or clinical trial design. In other cases, it was used to assess where in a treatment sequence a new treatment should be placed, or to evaluate the addition of more efficacious treatment options to a current treatment sequence. Important considerations for determining how to best model sequences include the number of treatment options, patient heterogeneity, key outcomes, and event risk (time-varying or constant). The biggest challenge is the scarcity of clinical data, as clinical trials do not commonly evaluate different treatment sequences.

The use of exploratory analyses within the National Institute for Health and Care Excellence single technology appraisal process: an evaluation and qualitative analysis

BACKGROUND

As part of the National Institute for Health and Care Excellence (NICE) single technology appraisal (STA) process, independent Evidence Review Groups (ERGs) critically appraise the company submission. During the critical appraisal process the ERG may undertake analyses to explore uncertainties around the company's model and their implications for decision-making. The ERG reports are a central component of the evidence considered by the NICE Technology Appraisal Committees (ACs) in their deliberations.

OBJECTIVE

The aim of this research was to develop an understanding of the number and type of exploratory analyses undertaken by the ERGs within the STA process and to understand how these analyses are used by the NICE ACs in their decision-making.

METHODS

The 100 most recently completed STAs with published guidance were selected for inclusion in the analysis. The documents considered were ERG reports, clarification letters, the first appraisal consultation document and the final appraisal determination. Over 400 documents were assessed in this study. The categories of types of exploratory analyses included fixing errors, fixing violations, addressing matters of judgement and the ERG-preferred base case. A content analysis of documents (documentary analysis) was undertaken to identify and extract relevant data, and narrative synthesis was then used to rationalise and present these data.

RESULTS

The level and type of detail in ERG reports and clarification letters varied considerably. The vast majority (93%) of ERG reports reported one or more exploratory analyses. The most frequently reported type of analysis in these 93 ERG reports related to the category 'matters of judgement', which was reported in 83 (89%) reports. The category 'ERG base-case/preferred analysis' was reported in 45 (48%) reports, the category 'fixing errors' was reported in 33 (35%) reports and the category 'fixing violations' was reported in 17 (18%) reports. The exploratory analyses performed were the result of issues raised by an ERG in its critique of the submitted economic evidence. These analyses had more influence on recommendations earlier in the STA process than later on in the process.

LIMITATIONS

The descriptions of analyses undertaken were often highly specific to a particular STA and could be inconsistent across ERG reports and thus difficult to interpret.

CONCLUSIONS

Evidence Review Groups frequently conduct exploratory analyses to test or improve the economic evaluations submitted by companies as part of the STA process. ERG exploratory analyses often have an influence on the recommendations produced by the ACs.

FUTURE WORK

More in-depth analysis is needed to understand how ERGs make decisions regarding which exploratory analyses should be undertaken. More research is also needed to fully understand which types of exploratory analyses are most useful to ACs in their decision-making.

FUNDING

The National Institute for Health Research Health Technology Assessment programme.

What Guidance are Economists Given on How to Present Economic Evaluations for Policymakers? A Systematic Review

OBJECTIVES

To systematically review health economic guidelines for information on how to present health economic evaluations and consider implications for nontechnical audiences such as policymakers.

METHODS

Electronic databases and supplementary sources were searched for economic evaluation guidelines. Guidelines were critically appraised. Descriptive characteristics, standard formats, supports for nontechnical audiences, presentation approaches, and common reporting recommendations were extracted. Frequencies were tabulated and trends identified.

RESULTS

Thirty-one guidelines were included. Twenty-two guidelines include a standard reporting format with some sample tables and graphs. Common presentation approaches include well-cited tables of data sources, transparent model diagrams and descriptions, disaggregated results, and tabular and graphical displays of sensitivity analyses. Despite most guidelines being funded by policymakers, only five guidelines provided advice on presenting economic evaluations to noneconomists. However, 11 guidelines included a glossary of economic terminology for nontechnical readers. Common concepts that may require further explanation include differences in economic perspectives, appropriateness of time horizons, how economic outcomes such as quality-adjusted life-years relate to their component clinical outcomes, and choice of sensitivity analyses.

CONCLUSIONS

Health economists have consistent presentation formats and common reporting elements that should be considered when developing user-friendly explanations for general audiences. These overlap with policymakers' informational needs but may not be sufficient for understanding by nontechnical audiences. Developing presentation formats and tools that incorporate viewpoints of both economists and noneconomists will allow for better application of the results of economic evaluations and enhance the transparency and legitimacy of decision-making processes that are informed by economic evaluations.

ASSESSING SEARCHES IN NICE SINGLE TECHNOLOGY APPRAISALS: PRACTICE AND CHECKLIST

Objectives: No guidelines exist in the approach that Evidence Review Groups (ERGs) should take to appraise search methodologies in the manufacturer's submission (MS) in Single Technology Appraisals (STA). As a result, ERGs are left to appraise searches using their own approach. This study investigates the limitations of manufacturers' search methodologies as critiqued by ERGs in published STA reports and to provide a recommended checklist.

Methods: Limitations from search critiques in 83 ERG reports published in the NIHR Web site between 2006 and May 2011 were extracted. The limitations were grouped into themes. Comparisons were made between limitations reported in the clinical effectiveness versus cost-effectiveness searches.

Results: Twelve themes were identified, six relating to the search strategy, source, limits, filters, translation, reporting, and missing studies. The search strategy theme contained the most limitations. Missing studies were frequently found by the ERG group in the clinical effectiveness searches. The omission of searches by manufacturers for unpublished and ongoing trials was frequently reported by the ERG. By contrast, failure of the manufacturer to report strategies was the most common limitation in the cost-effectiveness searches. Themes with the most frequent limitations in both types of searches are search strategy, reporting and source.

Conclusions: It is recommended that a checklist that has reporting, source and search strategy elements be used in the appraisal of manufacturer's searches during the STA process.

Improving the Accuracy and Comparability of Model-Based Economic Evaluations of Health Technologies for Reimbursement Decisions

Increasingly, decision analytic models are used within economic evaluations of health technologies (e.g., pharmaceuticals) submitted to national reimbursement bodies in countries like Australia and UK, where such models play a fundamental role in informing public funding decisions. Concerns regarding the accuracy of model outputs and hence the credibility of national reimbursement decisions are frequently raised. We propose a framework for developing reference models for specific diseases to inform economic evaluations of health technologies and their appraisal. The structure of a reference model reflects the natural history of the condition under study and defines the clinical events to be represented, the relationships between the events, and the effect of patient characteristics on the probability and timing of events. We contend that the use of reference models will improve the accuracy and comparability of public funding decisions. This can lead to the more efficient allocation of public funds.

A systematic review of coverage decision-making on health technologies-evidence from the real world

OBJECTIVE

Quantitative analysis of real-world coverage decision-making offers insights into the revealed preferences of appraisal committees. Aim of this review was to structure empirical evidence of coverage decisions made in practice based on the components 'methods and evidence', 'criteria and standards', 'decision outcome' and 'processes'.

METHODS

Several electronic databases, key journals and decision committees' websites were searched for publications between 1993 and June 2011. Inclusion criteria were the analysis of past decisions and application of quantitative methods. Each study was categorized by the scope of decision-making and the components covered by the variables used in quantitative analysis.

RESULTS

Thirty-two studies were identified. Many focused on pharmaceuticals, the UK NICE or the Australian PBAC. The components were covered comprehensively, but heterogeneously. Seventy-two variables were identified of which the following were more prevalent: specifications of the decision outcome; the indications considered for appraisal, identification of incremental cost-effectiveness ratios, appropriateness of evaluation methods, type of economic or clinical evidence used for assessment, and the decision date.

CONCLUSIONS

Research was dominated by analysis of decision outcomes and appraisal criteria. Although common approaches were identified, the complexity of coverage decision-making - reflected by the heterogeneity of identified variables - will continue to challenge empirical research.

A qualitative study of manufacturers' submissions to the UK NICE single technology appraisal process

OBJECTIVES

As part of the National Institute for Health and Clinical Excellence (NICE) Single Technology Appraisal (STA) process, manufacturers present submissions outlining the clinical and cost-effectiveness of new technologies. These submissions are critically appraised by Evidence Review Groups (ERGs), who produce a report, which forms part of the evidence considered by the NICE Appraisal Committees. The purpose of this research was first to identify common issues and concerns identified by the ERGs in their analyses of manufacturers' submissions (MS). The aim was then to use these as a basis to develop feedback for manufacturers.

DESIGN

A qualitative study using a content analysis approach to examine two sources of evidence, the first 30 ERG reports and 21 clarification letters associated with these STAs.

SETTING

UK HTA programme.

PRIMARY AND SECONDARY OUTCOME MEASURES

Common issues and concerns in MS.

RESULTS

There were positive comments regarding the quality of the MS, many of which were clearly written. The majority, however, were generally of poor quality and issues and concerns identified across the ERG reports and clarification letters included: criticisms related to the data being used especially data employed in the cost-effectiveness model, failure to perform a necessary analysis and poor reporting of processes used in the MS. Aspects of the decision problem were also often poorly or inadequately addressed by manufacturers. The majority of points raised for clarification related to the economic data analysis. Internal inconsistencies between the clinical and economic sections of the submission were frequently highlighted. These were used as the basis for the development of 12 suggestions for manufacturers.

CONCLUSIONS

Much can be done to improve the quality of MS in the NICE STA process. Suggestions include the need for clear and transparent reporting of methods and analyses.

The National Institute for Health and Clinical Excellence Single Technology Appraisal process: lessons from the first 4 years

OBJECTIVES

The National Institute for Health and Clinical Excellence (NICE) Single Technology Appraisal (STA) process in the United Kingdom was established in 2005 in order to provide guidance on new technologies as close to their launch as possible. The NICE recommended timeframe for completion of an STA is 34 weeks. The purpose of this study was to map the first 95 STAs to collect information on a range of issues including timelines and appraisal decisions.

METHODS

A mapping tool was devised to collect information from the NICE Web site. Data were analyzed by calculating frequencies. Simple descriptive statistics were applied where appropriate.

RESULTS

Ninety-five STAs were included in the analysis. Almost one-third (30/95) initially identified topics did not go on to be appraised often due to licensing issues. Timelines were measured for 29 completed STAs. Eight (28%) of these were completed by 37 weeks and 20 (69%) by 42 weeks. When STAs with appeals were excluded, 31% (8/26) were completed by 37 weeks and 85% (22/26) by 42 weeks. The incremental cost-effectiveness ratios reported by manufacturers were consistently lower than those estimated by the evidence review groups. In all, 76% (38/50) of the completed STAs resulted in an approval.

CONCLUSIONS

The NICE Web site enabled access to almost all necessary information, although electronic documents were sometimes difficult to locate. One-third of the referred topics were suspended or terminated. The NICE STA process is slower than initially anticipated and this is primarily due to events outside of NICE's direct control.

A thematic analysis of the strengths and weaknesses of manufacturers' submissions to the NICE Single Technology Assessment (STA) process

OBJECTIVES

The NICE Single Technology Appraisal (STA) process in the UK has been underway for five years. Evidence Review Groups (ERGs) critically appraise submissions from manufacturers on the clinical and cost effectiveness of new technologies. This study analysed the ERGs' assessment of the strengths and weaknesses of 30 manufacturers' submissions to the STA process.

METHODS

Thematic analysis was performed on the textual descriptions of the strengths and weakness of manufacturer submissions, as outlined by the ERGs in their reports.

FINDINGS

Various themes emerged from the data. These themes related to the processes applied in the submissions; the content of the submission (e.g. the amount and quality of evidence); the reporting of the submissions' review and analysis processes; the reliability and validity of the submissions' findings; and how far the submission had satisfied the STA process objectives.

CONCLUSIONS

STA submissions could be improved if attention were paid to transparency in the reporting, conduct and justification of review and modelling processes and analyses, as well as greater robustness in the choice of data and closer adherence to the scope or decision problem. Where this adherence is not possible, more detailed justification of the choice of evidence or data is required.