Towards a New Framework for Addressing Structural Uncertainty in Health Technology Assessment Guidelines

Unravelling the Association Between Uncertainties in Model-based Economic Analysis and Funding Recommendations of Medicines in Australia

OBJECTIVE

Health technology assessment is used extensively by the Pharmaceutical Benefits Advisory Committee (PBAC) to inform medicine funding recommendations in Australia. The PBAC often does not recommend medicines due to uncertainties in economic modelling that result in delaying access to medicines for patients. The systematic identification of which uncertainties can be reduced with alternative evidence or the collection of additional data can help inform recommendations. This study aims to characterise different types of uncertainty in economic models and empirically assess their association with the PBAC recommendations.

METHODS

A framework was developed to characterise four types of uncertainties: methodological, structural, generalisability and parameter uncertainty. The first two types were further subcategorised into parameterisable and unparameterisable uncertainty. Data on uncertainty and other factors were extracted from PBAC's Public Summary Documents of first submissions for 193 medicine (vaccine)-indication pairs including economic modelling between 2014 and 2021. Logistic regression was used to estimate the average marginal effect of each type of uncertainty on the probability of a positive recommendation.

RESULTS

The PBAC more often raised issues regarding parameter uncertainty (95%) and parameterisable structural uncertainty (83%) than generalisability uncertainty (48%) and unparameterisable methodological uncertainty (56%). The logistic regression results suggested that the PBAC was more likely to recommend a medicine without unparameterisable methodological, generalisability, and parameterisable structural uncertainty by 15.0%, 10.2 %, and 17.6%, respectively. Parameterisable methodological, unparameterisable structural and parameter uncertainty were not significantly associated with the PBAC recommendations.

CONCLUSIONS

This study identified the uncertainties that had significant associations with PBAC recommendations based on the first submission. This may help improve model quality and reduce resubmissions in the future, thus improving patients' access to medicines.

Navigating the path towards successful implementation of the EU HTA Regulation: key takeaways from the 2023 Spring Convention of the European Access Academy

BACKGROUND

The European Regulation on Health Technology Assessment (EU HTA R), effective since January 2022, aims to harmonize and improve the efficiency of common HTA across Member States (MS), with a phased implementation from January 2025. At "midterms" of the preparation phase for the implementation of the Regulation our aim was to identify and prioritize tangible action points to move forward.

METHODS

During the 2023 Spring Convention of the European Access Academy (EAA), participants from different nationalities and stakeholder backgrounds discussed readiness and remaining challenges for the Regulation's implementation and identified and prioritized action points. For this purpose, participants were assigned to four working groups: (i) Health Policy Challenges, (ii) Stakeholder Readiness, (iii) Approach to Uncertainty and (iv) Challenges regarding Methodology. Top four action points for each working group were identified and subsequently ranked by all participants during the final plenary session.

RESULTS

Overall "readiness" for the Regulation was perceived as neutral. Prioritized action points included the following: Health Policy, i.e. assess adjustability of MS laws and health policy processes; Stakeholders, i.e. capacity building; Uncertainty, i.e. implement HTA guidelines as living documents; Methodology, i.e. clarify the Population, Intervention, Comparator(s), Outcomes (PICO) identification process.

CONCLUSIONS

At "midterms" of the preparation phase, the focus for the months to come is on executing the tangible action points identified at EAA's Spring Convention. All action points centre around three overarching themes: harmonization and standardization, capacity building and collaboration, uncertainty management and robust data. These themes will ultimately determine the success of the EU HTA R in the long run.

How Sensitive is Sensitivity Analysis?: Evaluation of Pharmacoeconomic Submissions in Korea

Purpose: We aimed to describe the types of uncertainties examined in the economic evaluations submitted for reimbursement in Korea and their impact on the incremental cost-effectiveness ratio (ICER).

Method: Fifty dossiers were submitted by pharmaceutical companies to the economic subcommittee of the Pharmaceutical Benefit Coverage Advisory Committee (PBCAC) from January 2014 to December 2018. The types of uncertainties were categorized as structural and parametric, and the frequencies of the sensitivity analysis per variables were analyzed. The impact of uncertainties was measured by the percent variance of the ICER relative to that of the base case analysis.

Results: Of the 50 submissions, varying discount rate (44 submissions), followed by time horizon (38 submissions) and model assumptions (29 submissions), were most frequently used to examine structural uncertainty, while utility (42 submissions), resource use (41 submissions), and relative effectiveness (26 submissions) were used to examine parametric uncertainty. A total of 1,236 scenarios (a scenario corresponds to a case where a single variable is varied by a single range) were presented in the one-way sensitivity analyses, where parametric and structural sensitivity analyses comprised 679 and 557 scenarios, respectively. Varying drug prices had the highest impact on ICER (median variance 19.9%), followed by discount rate (12.2%), model assumptions (11.9%), extrapolation (11.8%), and time horizon (10.0%).

Conclusions: Variables related to long-term assumptions, such as model assumptions, time horizon, extrapolation, and discounting rate, were related to a high level of uncertainty. Caution should be exercised when using immature data.

A Dynamic Approach to Economic Priority Setting to Invest in Youth Mental Health and Guide Local Implementation: Economic Protocol for Eight System Dynamics Policy Models

BACKGROUND

Mental illness costs the world economy over US2.5 Bn each year, including premature mortality, morbidity, and productivity losses. Multisector approaches are required to address the systemic drivers of mental health and ensure adequate service provision. There is an important role for economics to support priority setting, identify best value investments and inform optimal implementation. Mental health can be defined as a complex dynamic system where decision makers are challenged to prospectively manage the system over time. This protocol describes the approach to equip eight system dynamics (SD) models across Australia to support priority setting and guide portfolio investment decisions, tailored to local implementation context.

METHODS

As part of a multidisciplinary team, three interlinked protocols are developed; (i) the participatory process to codesign the models with local stakeholders and identify interventions for implementation, (ii) the technical protocol to develop the SD models to simulate the dynamics of the local population, drivers of mental health, the service system and clinical outcomes, and (iii) the economic protocol to detail how the SD models will be equipped to undertake a suite of economic analysis, incorporating health and societal perspectives. Models will estimate the cost of mental illness, inclusive of service costs (health and other sectors, where necessary), quality-adjusted life years (QALYs) lost, productivity costs and carer costs. To assess the value of investing (disinvesting) in interventions, economic analysis will include return-on-investment, cost-utility, cost benefit, and budget impact to inform affordability. Economic metrics are expected to be dynamic, conditional upon changing population demographics, service system capacities and the mix of interventions when synergetic or antagonistic interactions. To support priority setting, a portfolio approach will identify best value combinations of interventions, relative to a defined budget(s). User friendly dashboards will guide decision makers to use the SD models to inform resource allocation and generate business cases for funding.

DISCUSSION

Equipping SD models to undertake economic analysis is intended to support local priority setting and help optimise implementation regarding the best value mix of investments, timing and scale. The objectives are to improve allocative efficiency, increase mental health and economic productivity.

State of the ART? Two New Tools for Risk Communication in Health Technology Assessments

PURPOSE

Outcomes of health technology assessments (HTA) are uncertain, and decision-making is associated with a risk. This risk, consisting of the probability of making a wrong decision and its impact, is rarely considered in HTA. This hampers transparent and consistent risk assessment and management. The aim of this study was to develop risk communication tools in the context of health technology decision-making under uncertainty.

METHODS

We performed a scoping review of tools for uncertainty and risk communication within HTA using citation pearl-growing. We developed two tools, drawing on existing publications on risk and uncertainty communication for inspiration. Individual semi-structured interviews with HTA stakeholders were performed to identify potential improvements in usefulness, user-friendliness, and information adequacy. Tools were amended and further evaluated in a real-world HTA and workshop with HTA stakeholders.

RESULTS

The identified risk communication tools did not include non-quantified uncertainties, and did not link to risk management strategies. We developed two tools: the Assessment of Risk Table (ART), for a summary of quantified and non-quantified uncertainties and the resulting risk assessment, and the Appraisal of Risk Chart (ARCH), for linking net benefit and risk outcomes to appropriate risk management strategies. Stakeholders appreciated the usefulness of the tools. They also highlighted that more information on local policy options was required for optimal risk management use, and HTA processes may need adapting.

CONCLUSION

The risk communication tools presented here can help assess risk, facilitate communication between analysts and decision-makers, and guide the appropriate use of available risk management strategies.

Implementation Barriers to Value of Information Analysis in Health Technology Decision Making: Results From a Process Evaluation

OBJECTIVES

Value of information (VOI) analysis can support health technology assessment decision making, but it is a long way from being standard use. The objective of this study was to understand barriers to the implementation of VOI analysis and propose actions to overcome these.

METHODS

We performed a process evaluation of VOI analysis use within decision making on tomosynthesis versus digital mammography for use in the Dutch breast cancer population screening. Based on steering committee meeting attendance and regular meetings with analysts, we developed a list of barriers to VOI use, which were analyzed using an established diffusion model. We proposed actions to address these barriers. Barriers and actions were discussed and validated in a workshop with stakeholders representing patients, clinicians, regulators, policy advisors, researchers, and the industry.

RESULTS

Consensus was reached on groups of barriers, which included characteristics of VOI analysis itself, stakeholder's attitudes, analysts' and policy makers' skills and knowledge, system readiness, and implementation in the organization. Observed barriers did not only pertain to VOI analysis itself but also to formulating the objective of the assessment, economic modeling, and broader aspects of uncertainty assessment. Actions to overcome these barriers related to organizational changes, knowledge transfer, cultural change, and tools.

CONCLUSIONS

This in-depth analysis of barriers to implementation of VOI analysis and resulting actions and tools may be useful to health technology assessment organizations that wish to implement VOI analysis in technology assessment and research prioritization. Further research should focus on application and evaluation of the proposed actions in real-world assessment processes.

Exploring the Feasibility of Comprehensive Uncertainty Assessment in Health Economic Modeling: A Case Study

OBJECTIVES

Decision makers adopt health technologies based on health economic models that are subject to uncertainty. In an ideal world, these models parameterize all uncertainties and reflect them in the cost-effectiveness probability and risk associated with the adoption. In practice, uncertainty assessment is often incomplete, potentially leading to suboptimal reimbursement recommendations and risk management. This study examines the feasibility of comprehensive uncertainty assessment in health economic models.

METHODS

A state transition model on peripheral arterial disease treatment was used as a case study. Uncertainties were identified and added to the probabilistic sensitivity analysis if possible. Parameter distributions were obtained by expert elicitation, and structural uncertainties were either parameterized or explored in scenario analyses, which were model averaged.

RESULTS

A truly comprehensive uncertainty assessment, parameterizing all uncertainty, could not be achieved. Expert elicitation informed 8 effectiveness, utility, and cost parameters. Uncertainties were parameterized or explored in scenario analyses and with model averaging. Barriers included time and resource constraints, also of clinical experts, and lacking guidance regarding some aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The team's multidisciplinary expertise and existing literature and tools were facilitators.

CONCLUSIONS

While comprehensive uncertainty assessment may not be attainable, improvements in uncertainty assessment in general are no doubt desirable. This requires the development of detailed guidance and hands-on tutorials for methods of uncertainty assessment, in particular aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The issue of benefits of uncertainty assessment versus time and resources needed remains unclear.

Building a trusted framework for uncertainty assessment in rare diseases: suggestions for improvement (Response to "TRUST4RD: tool for reducing uncertainties in the evidence generation for specialised treatments for rare diseases")

The aim of this letter to the editor is to provide a comprehensive summary of uncertainty assessment in Health Technology Assessment, with a focus on transferability to the setting of rare diseases. The authors of "TRUST4RD: tool for reducing uncertainties in the evidence generation for specialised treatments for rare diseases" presented recommendations for reducing uncertainty in rare diseases. Their article is of great importance but unfortunately suffers from a lack of references to the wider uncertainty in Health Technology Assessment and research prioritisation literature and consequently fails to provide a trusted framework for decision-making in rare diseases. In this letter to the editor we critique the authors' tool and provide pointers as to how their proposal can be strengthened. We present references to the literature, including our own tool for uncertainty assessment (TRUST; unrelated to the authors' research), apply TRUST to two assessments of orphan drugs in rare diseases and provide a broader perspective on uncertainty and risk management in rare diseases, including a detailed research agenda.

Quantifying Positive Health Externalities of Disease Control Interventions: Modeling Chikungunya and Dengue

Purpose. Health interventions can generate positive externalities not captured in traditional, single-disease cost-effectiveness analyses (CEAs), potentially biasing results. We illustrate this with the example of mosquito-borne diseases. When a particular mosquito species can transmit multiple diseases, a single-disease CEA comparing disease-specific interventions (e.g., vaccination) with interventions targeting the mosquito population (e.g., insecticide) would underestimate the insecticide's full benefits (i.e., preventing other diseases). Methods. We developed three dynamic transmission models: chikungunya, dengue, and combined chikungunya and dengue, each calibrated to disease-specific incidence and deaths in Colombia (June 2014 to December 2017). We compared the models' predictions of the incremental benefits and cost-effectiveness of an insecticide (10% efficacy), hypothetical chikungunya and dengue vaccines (40% coverage, 95% efficacy), and combinations of these interventions. Results. Model calibration yielded realistic parameters that produced close matches to disease-specific incidence and deaths. The chikungunya model predicted that vaccine would decrease the incidence of chikungunya and avert more total deaths than insecticide. The dengue model predicted that insecticide and the dengue vaccine would reduce dengue incidence and deaths, with no effect for the chikungunya vaccine. In the combined model, insecticide was more effective than either vaccine in reducing the incidence of and deaths from both diseases. In all models, the combined strategy was at least as effective as the most effective single strategy. In an illustrative CEA, the most frequently preferred strategy was vaccine in the chikungunya model, the status quo in the dengue model, and insecticide in the combined model. Limitations. There is uncertainty in the target calibration data. Conclusions. Failure to capture positive externalities can bias CEA results, especially when evaluating interventions that affect multiple diseases. Multidisease modeling is a reasonable alternative for addressing such biases.

Uncertainty and Coverage With Evidence Development: Does Practice Meet Theory?

OBJECTIVES

In theory, a successful coverage with evidence development (CED) scheme is one that addresses the most important uncertainties in a given assessment. We investigated the following: (1) which uncertainties were present during the initial assessment of 3 Dutch CED cases, (2) how these uncertainties were integrated in the initial assessments, (3) whether CED research plans included the identified uncertainties, and (4) issues with managing uncertainty in CED research and ways forward from these issues.

METHODS

Three CED initial assessment dossiers were analyzed and 16 stakeholders were interviewed. Uncertainties were identified in interviews and dossiers and were categorized in different causes: unavailability, indirectness, and imprecision of evidence. Identified uncertainties could be mentioned, described, and explored. Issues and ways forward to address uncertainty in CED schemes were discussed during the interviews.

RESULTS

Forty-two uncertainties were identified. Thirteen (31%) were caused by unavailability, 17 (40%) by indirectness, and 12 (29%) by imprecision. Thirty-four uncertainties (81%) were only mentioned, 19 (45%) were described, and the impact of 3 (7%) uncertainties on the results was explored in the assessment dossiers. Seventeen uncertainties (40%) were included in the CED research plans. According to stakeholders, research did not address the identified uncertainty, but CED research should be designed to focus on these.

CONCLUSIONS

In practice, uncertainties were neither systematically nor completely identified in the analyzed CED schemes. A framework would help to systematically identify uncertainty, and this process should involve all stakeholders. Value of information analysis, and the uncertainties that are not included in this analysis should inform CED research design.

Multivariable and Structural Uncertainty Analyses for Cost-Effectiveness Estimates: Back to the Future

BACKGROUND

In this commentary, celebrating the 20th anniversary of the journal Value in Health, I present a brief overview and illustration of the evolution over the past 20 years of the methodological literature providing guidelines for multivariable and structural uncertainty analysis for cost-effectiveness estimates.

METHODS

To illustrate the impact of the guidelines for uncertainty analyses, I show how the inclusion of multivariable and structural uncertainty analyses in cost-effectiveness analyses published in Value in Health changed over the past 20 years using publications from 1999/2000, 2007 and 2017.

RESULTS

The commentary is organized in three sections: past, focusing on the development and use of methods for multivariable uncertainty analysis; present, focusing on the growing awareness of the need for structural uncertainty analysis, suggested frameworks for structural uncertainty analysis and how it is currently implemented; and future, considering different methods for combining multivariable and structural uncertainty analyses over the next decades.

CONCLUSIONS

I conclude by suggesting how the continued evolution of uncertainty analyses in published studies and health technology assessment submissions can best take into account an important goal of cost-effectiveness analyses: to provide useful information to decision makers.

Model Structuring for Economic Evaluations of New Health Technologies

In countries such as Australia, the UK and Canada, decisions on whether to fund new health technologies are commonly informed by decision analytic models. While the impact of making inappropriate structural choices/assumptions on model predictions is well noted, there is a lack of clarity about the definition of key structural aspects, the process of developing model structure (including the development of conceptual models) and uncertainty associated with the structuring process (structural uncertainty) in guidelines developed by national funding bodies. This forms the focus of this article. Building on the reports of good modelling practice, and recognising the fundamental role of model structuring within the model development process, we specified key structural choices and provided ideas about model structuring for the future direction. This will help to further standardise guidelines developed by national funding bodies, with potential impact on transparency, comprehensiveness and consistency of model structuring. We argue that the process of model structuring and structural sensitivity analysis should be documented in a more systematic and transparent way in submissions to national funding bodies. Within the decision-making process, the development of conceptual models and presentation of all key structural choices would mean that national funding bodies could be more confident of maximising value for money when making public funding decisions.