Field testing of a multicriteria decision analysis (MCDA) framework for coverage of a screening test for cervical cancer in South Africa

Understanding innovation of health technology assessment methods: the IHTAM framework

Adequate methods are urgently needed to guarantee the good practice of health technology assessment (HTA) for technologies with novel properties. The aim of the study was to construct a conceptual framework to help understand the innovation of HTA methods (IHTAM). The construction of the IHTAM framework was based on two scoping reviews, one on the current practice of innovating methods, that is existing HTA frameworks, and one on theoretical foundations for innovating methods outside the HTA discipline. Both aimed to identify and synthesize concepts of innovation (i.e., innovation processes and roles of stakeholders in innovation). Using these concepts, the framework was developed in iterative brainstorming sessions and subsequent discussions with representatives from various stakeholder groups. The framework was constructed based on twenty documents on innovating HTA frameworks and fourteen guidelines from three scientific disciplines. It includes a generic innovation process consisting of three phases (“Identification,” “Development,” and “Implementation”) and nine subphases. In the framework, three roles that HTA stakeholders can play in innovation (“Developers,” “Practitioners,” and “Beneficiaries”) are defined, and a process on how the stakeholders innovate HTA methods is included. The IHTAM framework visualizes systematically which elements and stakeholders are important to the development and implementation of novel HTA methods. The framework could be used by all stakeholders involved in HTA innovation to learn how to engage dynamically and collaborate effectively throughout the innovation process. HTA stakeholders in practice have welcomed the framework, though additional testing of its applicability and acceptance is essential.

Multicriteria decision analysis in health care decision in oncology: a systematic review

INTRODUCTION

Multicriteria decision analysis (MCDA) has been used to inform health decisions in health technology assessments (HTA) processes. This is particularly important to complex treatment decisions in oncology.

AREAS COVERED

Five databases (PubMed, EMBASE, LILACS, Web of Science and CRD's NHS Economic Evaluation Database) were searched for studies comparing health technologies in oncology, involving the concept MCDA. The ISPOR MCDA Good Practices Guidelines were used to assess the reporting quality. Study selection, appraisal, and data extraction were performed by two reviewers. Fifteen studies were included. The main decision problem was related to health technology assessment of cancer treatments. Clinicians and public health experts were the most frequent stakeholders. The most frequently included criteria comprised therapeutic benefit, and socio-economic impact. Value measurement approach, direct rating techniques, and additive model for aggregation were used in most studies. Uncertainty analysis revealed the impact of posology and costs on the studies' results. All studies showed some level of overlapping decision criteria.

EXPERT OPINION

There is considerable diversity of methods in MCDA for healthcare decision-making in oncology. The evidence presented can serve as a resource when considering which stakeholders, criteria, and techniques to include in future MCDA studies in oncology.

Examining the association between oncology drug clinical benefit and the time to public reimbursement

Background

We examined if oncology drug indications with high clinical benefit, as measured by the American Society of Clinical Oncology Value Framework (ASCO‐VF) and European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO‐MCBS), received public reimbursement status faster than those with lower clinical benefit from the time of pan‐Canadian Oncology Drug Review (pCODR) recommendation.

Methods

Oncology drug indications submitted to pCODR between July 2011 and October 2018 were examined. Included indications had a regulatory approval date, completed the pCODR review process, received a positive pCODR recommendation, and been funded by at least one province. Trials cited for clinical efficacy were used to determine the clinical benefit (per ASCO‐VF and ESMO‐MCBS) of drug indications.

Results

Eighty‐four indications were identified, yielding 65 ASCO‐VF and 50 ESMO‐MCBS scores. The mean ASCO‐VF and ESMO‐MCBS scores were 44.9 (SD = 21.1) and 3.3 (SD = 1.0), respectively. The mean time to provincial reimbursement from pCODR recommendation was 13.2 months (SD = 9.3 months). Higher ASCO‐VF and ESMO‐MCBS scores had low correlation with shorter time to reimbursement, (ρ = −0.21) and (ρ = 0.24), respectively. In the multivariable analyses, ASCO‐VF (p = 0.40) and ESMO‐MCBS (p = 0.31) scores were not significantly associated with time to reimbursement. Province and year of pCODR recommendation were associated with time to reimbursement in both ASCO and ESMO models.

Conclusions

Oncology drug indications with higher clinical benefit do not appear to be reimbursed faster than those with low clinical benefit. This suggests the need to prioritize oncology drug indications based on clinical benefit to ensure quicker access to oncology drugs with the greatest benefits.

A New Decision Model Approach for Health Technology Assessment and A Case Study for Dialysis Alternatives in Turkey

Background: This paper presents a generic Multi-Criteria Decision Analysis (MCDA) model for Health Technology Assessment (HTA) decision-making, which can be applied to a wide range of HTA studies, regardless of the healthcare technology type under consideration. Methods: The HTA Core Model^® of EUnetHTA was chosen as a basis for the development of the MCDA model because of its common acceptance among European Union countries. Validation of MCDA4HTA was carried out by an application with the HTA study group of the Turkish Ministry of Health. The commitment of the decision-making group is completed via an online application of 10 different questionnaires. The Analytic Hierarchy Process (AHP) is used to determine the weights. Scores of the criteria in MCDA4HTA are gathered directly from the HTA report. The performance matrix in this application is run with fuzzy Technique for Order Preference by Similarity to Ideal Solution (TOPSIS), fuzzy Vise Kriterijumska Optimizacija I Kompromisno Resenje (VIKOR), and goal programming MCDA techniques. Results: Results for fuzzy VIKOR, fuzzy TOPSIS, and goal programming are 0.018, 0.309, and 0.191 for peritoneal dialysis and 0.978, 0.677, and 0.327 for hemodialysis, respectively. Conclusions: Peritoneal dialysis is found to be the best choice under the given circumstances, despite its higher costs to society. As an integrated decision-making model for HTA, MCDA4HTA supports both evidence-based decision policy and the transparent commitment of multi-disciplinary stakeholders.

PARS, a system combining semantic technologies with multiple criteria decision aiding for supporting antibiotic prescriptions

OBJECTIVE

Motivated by the well documented worldwide spread of adverse drug events, as well as the increased danger of antibiotic resistance (caused mainly by inappropriate prescribing and overuse), we propose a novel recommendation system for antibiotic prescription (PARS).

METHOD

Our approach is based on the combination of semantic technologies with MCDA (Multiple Criteria Decision Aiding) that allowed us to build a two level decision support model. Given a specific domain, the approach assesses the adequacy of an alternative/action (prescription of antibiotic) for a specific subject (patient) with an issue (bacterial infection) in a given context (medical). The goal of the first level of the decision support model is to select the set of alternatives which have the potential to be suitable. Then the second level sorts the alternatives into categories according to their adequacy using an MCDA sorting method (MR-Sort with Veto) and a structured set of description logic queries.

RESULTS

We applied this approach in the domain of antibiotic prescriptions, working closely with the EpiCura Hospital Center (BE). Its performance was compared to the EpiCura recommendation guidelines which are currently in use. The results showed that the proposed system is more consistent in its recommendations when compared with the static EpiCura guidelines. Moreover, with PARS the antibiotic prescribing workflow becomes more flexible. PARS allows the user (physician) to update incrementally and dynamically a patient's profile with more information, or to input knowledge modifications that accommodate the decision context (like the introduction of new side effects and antibiotics, the development of germs that are resistant, etc). At the end of our evaluation, we detail a number of limitations of the current version of PARS and discuss future perspectives.

Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art

BACKGROUND

Multi-criteria decision analysis (MCDA) concepts, models and tools have been used increasingly in health technology assessment (HTA), with several studies pointing out practical and theoretical issues related to its use. This study provides a critical review of published studies on MCDA in the context of HTA by assessing their methodological quality and summarising methodological challenges.

METHODS

A systematic review was conducted to identify studies discussing, developing or reviewing the use of MCDA in HTA using aggregation approaches. Studies were classified according to publication time and type, country of study, technology type and study type. The PROACTIVE-S approach was constructed and used to analyse methodological quality. Challenges and limitations reported in eligible studies were collected and summarised; this was followed by a critical discussion on research requirements to address the identified challenges.

RESULTS

129 journal articles were eligible for review, 56% of which were published in 2015-2017; 42% focused on pharmaceuticals; 36, 26 and 18% reported model applications, issues regarding MCDA implementation analyses, and proposing frameworks, respectively. Poor compliance with good methodological practice (< 25% complying studies) was found regarding behavioural analyses, discussion of model assumptions and uncertainties, modelling of value functions, and dealing with judgment inconsistencies. The five most reported challenges related to evidence and data synthesis; value system differences and participant selection issues; participant difficulties; methodological complexity and resource balance; and criteria and attributes modelling. A critical discussion on ways to address these challenges ensues.

DISCUSSION

Results highlight the need for advancement in robust methodologies, procedures and tools to improve methodological quality of MCDA in HTA studies. Research pathways include developing new model features, good practice guidelines, technologies to enable participation and behavioural research.

Prevention of cervical cancer in HIV-seropositive women from developing countries through cervical cancer screening: a systematic review

BACKGROUND

There is scanty or inconclusive evidence on which cervical cancer screening tool is effective and suitable for human immunodeficiency virus (HIV)-seropositive women. The aim of this review was to assess, synthesise and document published evidence relating to the available cervical cancer screening modalities for HIV-seropositive women in developing countries. This paper did not review the issue of human papillomavirus (HPV) prophylactic vaccine on HIV-seropositive women.

METHODS

Five electronic databases were systematically searched from inception to January 2018 for relevant published original research examining cervical cancer prevention modalities for HPV infection, abnormal cytology and direct visualisation of the cervix amongst HIV-seropositive women in developing countries. Extra studies were identified through reference list and citation tracking.

RESULTS

Due to methodological and clinical heterogeneity, a narrative synthesis was presented. Of the 2559 articles, 149 underwent full-text screening and 25 were included in the review. Included studies were of moderate quality, and no exclusions were made based on quality or bias. There is no standard cervical cancer screening test or programme for HIV-seropositive women and countries screening according to available resources and expertise. The screening methods used for HIV-seropositive women are the same for HIV-negative women, with varying clinical performance and accuracy. The main cervical cancer screening methods described for HIV-seropositive women are HPV deoxyribonucleic acid/messenger RNA (DNA/mRNA) testing (n = 16, 64.0%), visual inspection with acetic acid (VIA) (n = 13, 52.0%) and Pap smear (n = 11, 44.0%). HPV testing has a better accuracy/efficiency than other methods with a sensitivity of 80.0-97.0% and specificity of 51.0-78.0%. Sequential screening using VIA or visual inspection with Lugol's iodine (VILI) and HPV testing has shown better clinical performance in screening HIV-seropositive women.

CONCLUSION

Although cervical cancer screening exists in almost all developing countries, what is missing is both opportunistic and systematic organised population-based screenings. Cervical cancer screening programmes need to be integrated into already existing HIV services to enable early detection and treatment. There is a need to offer opportunistic and coordinated screening programmes that are provider-initiated to promote early identification of cervical precancerous lesions.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42018095702.

Can reflective multicriteria be the new paradigm for healthcare decision-making? The EVIDEM journey

Background

Multiple technologies, procedures and programs call for fairly-based decisions for prioritization of healthcare interventions. There is a diversity of perspectives of what constitutes a legitimate decision, which depends on both the process and the reasoning applied. Current approaches focus on technical aspects while methods to support alignment of decisions with the compassionate impetus of healthcare systems is lacking.

Methods

The framework was developed based on an analysis of the foundations of healthcare systems, the reasoning underlying decisions and fair processes. The concept of reflective multicriteria was created: it assumes that decisionmakers guided by a generic interpretative frame rooted in the compassionate impetus of healthcare systems, can sharpen their reasoning, raise awareness of their motivation and increase legitimacy of decisions. The initial framework was made available through a not for profit organization (the EVIDEM Collaboration, 2006–2017) to stimulate its development with thought leaders and stakeholders in an open source philosophy. Development was tailored to the real-life needs of decisionmakers and drew on several domains of knowledge including healthcare ethics, evidenced-based medicine, health economics, health technology assessment and multicriteria approaches.

Results

The 10th edition framework builds on four dimensions: (1) the universal impetus of healthcare systems, (2) reasoning, values and ethics, (3) evidence and knowledge on interventions, and (4) a transformative process. Mathematical aspects of the framework are designed to help clarify, express and share individual reasoning; this non-conventional use of numbers requires a cultural change and needs to be phased in slowly. The framework includes four tools for easy adaptation and operationalization: (a) concepts and operationalization, (b) adapt and pilot, (c) evidence matrix, (d) mathematical representation of reasoning. Application is useful throughout all types of healthcare interventions, for all levels of decision, and across the globe.

Conclusion

By clarifying their reasoning while keeping decisionmakers aware of the impetus of healthcare systems, reflective multicriteria provides an effective approach to increase the legitimacy of decisions. Beyond a tool, reflective multicriteria pioneered by EVIDEM is geared to transform our vision of the value of healthcare interventions and how they might contribute to relevant, equitable and sustainable healthcare systems.

Electronic supplementary material

The online version of this article (10.1186/s12962-018-0116-9) contains supplementary material, which is available to authorized users.

Incorporating MCDA into HTA: challenges and potential solutions, with a focus on lower income settings

Background

Multicriteria decision analysis (MCDA) has the potential to bring more structure and transparency to health technology assessment (HTA). The objective of this paper is to highlight key methodological and practical challenges facing the use of MCDA for HTA, with a particular focus on lower and middle-income countries (LMICs), and to highlight potential solutions to these challenges.

Methodological challenges

Key lessons from existing applications of MCDA to HTA are summarized, including: that the socio-technical design of the MCDA reflect the local decision problem; the criteria set properties of additive models are understood and applied; and the alternative approaches for estimating opportunity cost, and the challenges with these approaches are understood.

Practical challenges

Existing efforts to implement HTA in LMICs suggest a number of lessons that can help overcome the practical challenges facing the implementation of MCDA in LMICs, including: adapting inputs from other settings and from expert opinion; investing in technical capacity; embedding the MCDA in the decision-making process; and ensuring that the MCDA design reflects local cultural and social factors.

Conclusion

MCDA has the potential to improve decision making in LMICs. For this potential to be achieved, it is important that the lessons from existing applications of MCDA are learned.

HTA and MCDA solely or combined? The case of priority-setting in Colombia

Background

All healthcare systems face problems of justice and efficiency related to setting priorities for allocating limited financial resources. Therefore, explicit decision-making in healthcare depicted as a continuum from evidence generation to deliberation and communication of the decision made, needs to be transparent and fair. Nevertheless, priority-setting in many parts of the world remains being implicit and ad-hoc process. Health Technology Assessment (HTA) and Multi-Criteria Decision Analysis (MCDA) have emerged as policy tools to assist informed decision-making. Both, MCDA and HTA have pros and cons.

Main body

Colombia experienced an important institutional transformation after the establishment of the Health Technology Assessment Institute in 2012. This paper briefly presents the current challenges of the Colombian health system, the general features of the new health sector reform, the main characteristics of HTA in Colombia and the potential benefits and caveats of incorporating MCDA approaches into the decision-making process.

Conclusion

Structured and objective consideration of the factors that are both measurable and value-based in an open and transparent manner may be feasible through combining HTA and MCDA in contexts like Colombia. Further testing and validation of HTA and MCDA solely or combined in LMICs are needed to advance these approaches into healthcare decision-making worldwide.

Developing a decision support system to link health technology assessment (HTA) reports to the health system policies in Iran

The recent increase of 'Health Technology Assessment' (HTA)-related activities in Iran has necessitated the clarification of policy-making process based on the HTA reports. This study aimed to develop a Decision Support System (DSS) in order to adopt evidence-informed policies regarding health technologies in Iran. The study can be classified as Health Policy and Systems Research. A core panel of seven experts conducted two separate reviews of relevant literature for: 1- Determining the potential technology-related policies. 2- Listing the criteria influencing those policy decisions. The policies and criteria were separately discussed and subsequently rated for appropriateness and necessity during two expert meetings in 2013. In the next step, The 'Discrete Choice Experiment' (DCE) method was employed to develop the DSS for the final technology-related policies. Accordingly, the core panel members independently rated the appropriateness of each policy for 30 virtual technologies based on the random values assigned to all the criteria for each technology. The obtained data for each policy were separately analysed using stepwise regression model, resulting in a minimal set of independent and statistically significant criteria contributing in the experts' judgments about the appropriateness of that policy. The obtained regression coefficients were used as the relative weights of the different levels of the final criteria of any policy statement, shaping the decision support scoring tool for each policy. The study has outlined 64 policy decisions under 7 macro policy areas concerning a health technology. Also, 34 criteria used for making those policy decisions have been organized within a portfolio. DCE, using stepwise regression, resulted in 64 scoring tools shaping the DSS for all HTA-related policies. Both the results and methodology of the study may serve as a guide for policy makers (researchers), particularly in low and middle income countries, in developing decision aids for their own context-specific HTA-related policies.

Appraising the holistic value of Lenvatinib for radio-iodine refractory differentiated thyroid cancer: A multi-country study applying pragmatic MCDA

BACKGROUND

The objective of the study was to reveal through pragmatic MCDA (EVIDEM) the contribution of a broad range of criteria to the value of the orphan drug lenvatinib for radioiodine refractory differentiated thyroid cancer (RR-DTC) in country-specific contexts.

METHODS

The study was designed to enable comprehensive appraisal (12 quantitative, 7 qualitative criteria) in the current disease context (watchful waiting, sorafenib) of France, Italy and Spain. Data on the value of lenvatinib was collected from diverse stakeholders during country-specific panels and included: criteria weights (individual and social values); performance scores (judgments on evidence-collected through MCDA systematic review); qualitative impacts of contextual criteria; and verbal and written insights structured by criteria. The value contribution of each criterion was calculated and uncertainty explored.

RESULTS

Comparative effectiveness, Quality of evidence (Spain and Italy) and Disease severity (France) received the greatest weights. Four criteria contributed most to the value of lenvatinib, reflecting its superior Comparative effectiveness (16-22% of value), the severity of RR-DTC (16-22%), significant unmet needs (14-21%) and robust evidence (14-20%). Contributions varied by comparator, country and individuals, highlighting the importance of context and consultation. Results were reproducible at the group level. Impacts of contextual criteria varied across countries reflecting different health systems and cultural backgrounds. The MCDA process promoted sharing stakeholders' knowledge on lenvatinib and insights on context.

CONCLUSIONS

The value of lenvatinib was consistently positive across diverse therapeutic contexts. MCDA identified the aspects contributing most to value, revealed rich contextual insights, and helped participants express and explicitly tackle ethical trade-offs inherent to balanced appraisal and decisionmaking.

Utilization of multiple-criteria decision analysis (MCDA) to support healthcare decision-making FIFARMA, 2016

Background and objectives:   MCDA is a decision-making tool with increasing use in the healthcare sector, including HTA (Health Technology Assessment). By applying multiple criteria, including innovation, in a comprehensive, structured and explicit manner, MCDA fosters a transparent, participative, consistent decision-making process taking into consideration values of all stakeholders. This paper by FIFARMA (Latin American Federation of Pharmaceutical Industry) proposes the deliberative (partial) MCDA as a more pragmatic, agile approach, especially when newly implemented. Methods: Literature review including real-world examples of effective MCDA implementation in healthcare decision making in both the public and private sector worldwide and in LA. Results and conclusion: It is the view of FIFARMA that MCDA should strongly be considered as a tool to support HTA and broader healthcare decision making such as the contracts and tenders process in order to foster transparency, fairness, and collaboration amongst stakeholders.

Priority Setting for Universal Health Coverage: We Need Evidence-Informed Deliberative Processes, Not Just More Evidence on Cost-Effectiveness

Priority setting of health interventions is generally considered as a valuable approach to support low- and middle-income countries (LMICs) in their strive for universal health coverage (UHC). However, present initiatives on priority setting are mainly geared towards the development of more cost-effectiveness information, and this evidence does not sufficiently support countries to make optimal choices. The reason is that priority setting is in reality a value-laden political process in which multiple criteria beyond cost-effectiveness are important, and stakeholders often justifiably disagree about the relative importance of these criteria. Here, we propose the use of 'evidence-informed deliberative processes' as an approach that does explicitly recognise priority setting as a political process and an intrinsically complex task. In these processes, deliberation between stakeholders is crucial to identify, reflect and learn about the meaning and importance of values, informed by evidence on these values. Such processes then result in the use of a broader range of explicit criteria that can be seen as the product of both international learning ('core' criteria, which include eg, cost-effectiveness, priority to the worse off, and financial protection) and learning among local stakeholders ('contextual' criteria). We believe that, with these evidence-informed deliberative processes in place, priority setting can provide a more meaningful contribution to achieving UHC.

TESTING MULTI-CRITERIA DECISION ANALYSIS FOR MORE TRANSPARENT RESOURCE-ALLOCATION DECISION MAKING IN COLOMBIA

Objectives: In 2012, Colombia experienced an important institutional transformation after the establishment of the Health Technology Assessment Institute (IETS), the disbandment of the Regulatory Commission for Health and the reassignment of reimbursement decision-making powers to the Ministry of Health and Social Protection (MoHSP). These dynamic changes provided the opportunity to test Multi-Criteria Decision Analysis (MCDA) for systematic and more transparent resource-allocation decision-making.

Methods: During 2012 and 2013, the MCDA framework Evidence and Value: Impact on Decision Making (EVIDEM) was tested in Colombia. This consisted of a preparatory stage in which the investigators conducted literature searches and produced HTA reports for four interventions of interest, followed by a panel session with decision makers. This method was contrasted with a current approach used in Colombia for updating the publicly financed benefits package (POS), where narrative health technology assessment (HTA) reports are presented alongside comprehensive budget impact analyses (BIAs).

Results: Disease severity, size of population, and efficacy ranked at the top among fifteen preselected relevant criteria. MCDA estimates of technologies of interest ranged between 71 to 90 percent of maximum value. The ranking of technologies was sensitive to the methods used. Participants considered that a two-step approach including an MCDA template, complemented by a detailed BIA would be the best approach to assist decision-making in this context. Participants agreed that systematic priority setting should take place in Colombia.

Conclusions: This work may serve as the basis to the MoHSP on its interest of setting up a systematic and more transparent process for resource-allocation decision-making.

Can the EVIDEM Framework Tackle Issues Raised by Evaluating Treatments for Rare Diseases: Analysis of Issues and Policies, and Context-Specific Adaptation

BACKGROUND

The multiplicity of issues, including uncertainty and ethical dilemmas, and policies involved in appraising interventions for rare diseases suggests that multicriteria decision analysis (MCDA) based on a holistic definition of value is uniquely suited for this purpose. The objective of this study was to analyze and further develop a comprehensive MCDA framework (EVIDEM) to address rare disease issues and policies, while maintaining its applicability across disease areas.

METHODS

Specific issues and policies for rare diseases were identified through literature review. Ethical and methodological foundations of the EVIDEM framework v3.0 were systematically analyzed from the perspective of these issues, and policies and modifications of the framework were performed accordingly to ensure their integration.

RESULTS

Analysis showed that the framework integrates ethical dilemmas and issues inherent to appraising interventions for rare diseases but required further integration of specific aspects. Modification thus included the addition of subcriteria to further differentiate disease severity, disease-specific treatment outcomes, and economic consequences of interventions for rare diseases. Scoring scales were further developed to include negative scales for all comparative criteria. A methodology was established to incorporate context-specific population priorities and policies, such as those for rare diseases, into the quantitative part of the framework. This design allows making more explicit trade-offs between competing ethical positions of fairness (prioritization of those who are worst off), the goal of benefiting as many people as possible, the imperative to help, and wise use of knowledge and resources. It also allows addressing variability in institutional policies regarding prioritization of specific disease areas, in addition to existing uncertainty analysis available from EVIDEM.

CONCLUSION

The adapted framework measures value in its widest sense, while being responsive to rare disease issues and policies. It provides an operationalizable platform to integrate values, competing ethical dilemmas, and uncertainty in appraising healthcare interventions.

Balancing costs and benefits at different stages of medical innovation: a systematic review of Multi-criteria decision analysis (MCDA)

Background

The diffusion of health technologies from translational research to reimbursement depends on several factors included the results of health economic analysis. Recent research identified several flaws in health economic concepts. Additionally, the heterogeneous viewpoints of participating stakeholders are rarely systematically addressed in current decision-making. Multi-criteria Decision Analysis (MCDA) provides an opportunity to tackle these issues. The objective of this study was to review applications of MCDA methods in decisions addressing the trade-off between costs and benefits.

Methods

Using basic steps of the PRISMA guidelines, a systematic review of the healthcare literature was performed to identify original research articles from January 1990 to April 2014. Medline, PubMed, Springer Link and specific journals were searched. Using predefined categories, bibliographic records were systematically extracted regarding the type of policy applications, MCDA methodology, criteria used and their definitions.

Results

22 studies were included in the analysis. 15 studies (68 %) used direct MCDA approaches and seven studies (32 %) used preference elicitation approaches. Four studies (19 %) focused on technologies in the early innovation process. The majority (18 studies - 81 %) examined reimbursement decisions. Decision criteria used in studies were obtained from the literature research and context-specific studies, expert opinions, and group discussions. The number of criteria ranged between three up to 15. The most frequently used criteria were health outcomes (73 %), disease impact (59 %), and implementation of the intervention (40 %). Economic criteria included cost-effectiveness criteria (14 studies, 64 %), and total costs/budget impact of an intervention (eight studies, 36 %). The process of including economic aspects is very different among studies. Some studies directly compare costs with other criteria while some include economic consideration in a second step.

Conclusions

In early innovation processes, MCDA can provide information about stakeholder preferences as well as evidence needs in further development. However, only a minority of these studies include economic features due to the limited evidence. The most important economic criterion cost-effectiveness should not be included from a technical perspective as it is already a composite of costs and benefit. There is a significant lack of consensus in methodology employed by the various studies which highlights the need for guidance on application of MCDA at specific phases of an innovation.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-015-0930-0) contains supplementary material, which is available to authorized users.

Exploring the perspectives and preferences for HTA across German healthcare stakeholders using a multi-criteria assessment of a pulmonary heart sensor as a case study

BACKGROUND

Health technology assessment and healthcare decision-making are based on multiple criteria and evidence, and heterogeneous opinions of participating stakeholders. Multi-criteria decision analysis (MCDA) offers a potential framework to systematize this process and take different perspectives into account. The objectives of this study were to explore perspectives and preferences across German stakeholders when appraising healthcare interventions, using multi-criteria assessment of a heart pulmonary sensor as a case study.

METHODS

An online survey of 100 German healthcare stakeholders was conducted using a comprehensive MCDA framework (EVIDEM V2.2). Participants were asked to provide i) relative weights for each criterion of the framework; ii) performance scores for a health pulmonary sensor, based on available data synthesized for each criterion; and iii) qualitative feedback on the consideration of contextual criteria. Normalized weights and scores were combined using a linear model to calculate a value estimate across different stakeholders. Differences across types of stakeholders were explored.

RESULTS

The survey was completed by 54 participants. The most important criteria were efficacy, patient reported outcomes, disease severity, safety, and quality of evidence (relative weight >0.075 each). Compared to all participants, policymakers gave more weight to budget impact and quality of evidence. The quantitative appraisal of a pulmonary heart sensor revealed differences in scoring performance of this intervention at the criteria level between stakeholder groups. The highest value estimate of the sensor reached 0.68 (on a scale of 0 to 1, 1 representing maximum value) for industry representatives and the lowest value of 0.40 was reported for policymakers, compared to 0.48 for all participants. Participants indicated that most qualitative criteria should be considered and their impact on the quantitative appraisal was captured transparently.

CONCLUSIONS

The study identified important variations in perspectives across German stakeholders when appraising a healthcare intervention and revealed that MCDA can demonstrate the value of a specified technology for all participating stakeholders. Better understanding of these differences at the criteria level, in particular between policymakers and industry representatives, is important to focus innovation aligned with patient health and healthcare system values and constraints.

'Real-world' health care priority setting using explicit decision criteria: a systematic review of the literature

Background

Health care decision making requires making resource allocation decisions among programs, services, and technologies that all compete for a finite resource pool. Methods of priority setting that use explicitly defined criteria can aid health care decision makers in arriving at funding decisions in a transparent and systematic way. The purpose of this paper is to review the published literature and examine the use of criteria-based methods in ‘real-world’ health care allocation decisions.

Methods

A systematic review of the published literature was conducted to find examples of ‘real-world’ priority setting exercises that used explicit criteria to guide decision-making.

Results

We found thirty-three examples in the peer-reviewed and grey literature, using a variety of methods and criteria. Program effectiveness, equity, affordability, cost-effectiveness, and the number of beneficiaries emerged as the most frequently-used decision criteria. The relative importance of criteria in the ‘real-world’ trials differed from the frequency in preference elicitation exercises. Neither the decision-making method used, nor the relative economic strength of the country in which the exercise took place, appeared to have a strong effect on the type of criteria chosen.

Conclusions

Health care decisions are made based on criteria related both to the health need of the population and the organizational context of the decision. Following issues related to effectiveness and affordability, ethical issues such as equity and accessibility are commonly identified as important criteria in health care resource allocation decisions.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-015-0814-3) contains supplementary material, which is available to authorized users.

A systematic review of health technology assessment tools in sub-Saharan Africa: methodological issues and implications

BACKGROUND

Health technology assessment (HTA) is mostly used in the context of high- and middle-income countries. Many "resource-poor" settings, which have the greatest need for critical assessment of health technology, have a limited basis for making evidence-based choices. This can lead to inappropriate use of technologies, a problem that could be addressed by HTA that enables the efficient use of resources, which is especially crucial in such settings. There is a lack of clarity about which HTA tools should be used in these settings. This research aims to provide an overview of proposed HTA tools for "resource-poor" settings with a specific focus on sub-Saharan Africa (SSA).

METHODOLOGY

A systematic review was conducted using basic steps from the PRISMA guidelines. Studies that described HTA tools applicable for "resource-limited" settings were identified and critically appraised. Only papers published between 2003 and 2013 were included. The identified tools were assessed according to a checklist with methodological criteria.

RESULTS

Six appropriate tools that are applicable in the SSA setting and cover methodological robustness and ease of use were included in the review. Several tools fulfil these criteria, such as the KNOW ESSENTIALS tool, Mini-HTA tool, and Multi-Criteria Decision Analysis but their application in the SSA context remains limited. The WHO CHOICE method is a standardized decision making tool for choosing interventions but is limited to their cost-effectiveness. Most evaluation of health technology in SSA focuses on priority setting. There is a lack of HTA tools that can be used for the systematic assessment of technology in the SSA context.

CONCLUSIONS

An appropriate HTA tool for "resource-constrained" settings, and especially SSA, should address all important criteria of decision making. By combining the two most promising tools, KNOW ESSENTIALS and Multi-Criteria Decision Analysis, appropriate analysis of evidence with a robust and flexible methodology could be applied for the SSA setting.

Does technique matter; a pilot study exploring weighting techniques for a multi-criteria decision support framework

Background

There is an increased interest in the use of multi-criteria decision analysis (MCDA) to support regulatory and reimbursement decision making. The EVIDEM framework was developed to provide pragmatic multi-criteria decision support in health care, to estimate the value of healthcare interventions, and to aid in priority-setting. The objectives of this study were to test 1) the influence of different weighting techniques on the overall outcome of an MCDA exercise, 2) the discriminative power in weighting different criteria of such techniques, and 3) whether different techniques result in similar weights in weighting the criteria set proposed by the EVIDEM framework.

Methods

A sample of 60 Dutch and Canadian students participated in the study. Each student used an online survey to provide weights for 14 criteria with two different techniques: a five-point rating scale and one of the following techniques selected randomly: ranking, point allocation, pairwise comparison and best worst scaling.

Results

The results of this study indicate that there is no effect of differences in weights on value estimates at the group level. On an individual level, considerable differences in criteria weights and rank order occur as a result of the weight elicitation method used, and the ability of different techniques to discriminate in criteria importance. Of the five techniques tested, the pair-wise comparison of criteria has the highest ability to discriminate in weights when fourteen criteria are compared.

Conclusions

When weights are intended to support group decisions, the choice of elicitation technique has negligible impact on criteria weights and the overall value of an innovation. However, when weights are used to support individual decisions, the choice of elicitation technique influences outcome and studies that use dissimilar techniques cannot be easily compared. Weight elicitation through pairwise comparison of criteria is preferred when taking into account its superior ability to discriminate between criteria and respondents’ preferences.

Electronic supplementary material

The online version of this article (doi:10.1186/1478-7547-12-22) contains supplementary material, which is available to authorized users.

Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature

The objective of this study is to support those undertaking a multi-criteria decision analysis (MCDA) by reviewing the approaches adopted in healthcare MCDAs to date, how these varied with the objective of the study, and the lessons learned from this experience. Searches of EMBASE and MEDLINE identified 40 studies that provided 41 examples of MCDA in healthcare. Data were extracted on the objective of the study, methods employed, and decision makers' and study authors' reflections on the advantages and disadvantages of the methods. The recent interest in MCDA in healthcare is mirrored in an increase in the application of MCDA to evaluate healthcare interventions. Of the studies identified, the first was published in 1990, but more than half were published since 2011. They were undertaken in 18 different countries, and were designed to support investment (coverage and reimbursement), authorization, prescription, and research funding allocation decisions. Many intervention types were assessed: pharmaceuticals, public health interventions, screening, surgical interventions, and devices. Most used the value measurement approach and scored performance using predefined scales. Beyond these similarities, a diversity of different approaches were adopted, with only limited correspondence between the approach and the type of decision or product. Decision makers consulted as part of these studies, as well as the authors of the studies are positive about the potential of MCDA to improve decision making. Further work is required, however, to develop guidance for those undertaking MCDA.

Multicriteria decision analysis in oncology

BACKGROUND

There has been a growing interest in the development and application of alternative decision-making frameworks within health care, including multicriteria decision analysis (MCDA). Even though the literature includes several reviews on MCDA methods, applications of MCDA in oncology are lacking.

AIM

The aim of this paper is to discuss a rationale for the use of MCDA in oncology. In this context, the following research question emerged: How can MCDA be used to develop a clinical decision support tool in oncology?

METHODS

In this paper, a brief background on decision making is presented, followed by an overview of MCDA methods and process. The paper discusses some applications of MCDA, proposes research opportunities in the context of oncology and presents an illustrative example of how MCDA can be applied to oncology.

FINDINGS

Decisions in oncology involve trade-offs between possible benefits and harms. MCDA can help analyse trade-off preferences. A wide range of MCDA methods exist. Each method has its strengths and weaknesses. Choosing the appropriate method varies depending on the source and nature of information used to inform decision making. The literature review identified eight studies. The analytical hierarchy process (AHP) was the most often used method in the identified studies.

CONCLUSION

Overall, MCDA appears to be a promising tool that can be used to assist clinical decision making in oncology. Nonetheless, field testing is desirable before MCDA becomes an established decision-making tool in this field.

How to use multi-criteria decision analysis methods for reimbursement decision-making in healthcare: a step-by-step guide

In recent years, the quest for more comprehensiveness, structure and transparency in reimbursement decision-making in healthcare has prompted the research into alternative decision-making frameworks. In this environment, multi-criteria decision analysis (MCDA) is arising as a valuable tool to support healthcare decision-making. In this paper, we present the main MCDA decision support methods (elementary methods, value-based measurement models, goal programming models and outranking models) using a case study approach. For each family of methods, an example of how an MCDA model would operate in a real decision-making context is presented from a critical perspective, highlighting the parameters setting, the selection of the appropriate evaluation model as well as the role of sensitivity and robustness analyses. This study aims to provide a step-by-step guide on how to use MCDA methods for reimbursement decision-making in healthcare.

Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada

BACKGROUND

Consistent healthcare decision making requires systematic consideration of decision criteria and evidence available to inform them. This can be tackled by combining multicriteria decision analysis (MCDA) and Health Technology Assessment (HTA). The objective of this study was to field-test a decision support framework (EVIDEM), explore its utility to a drug advisory committee and test its reliability over time.

METHODS

Tramadol for chronic non-cancer pain was selected by the health plan as a case study relevant to their context. Based on extensive literature review, a by-criterion HTA report was developed to provide synthesized evidence for each criterion of the framework (14 criteria for the MCDA Core Model and 6 qualitative criteria for the Contextual Tool). During workshop sessions, committee members tested the framework in three steps by assigning: 1) weights to each criterion of the MCDA Core Model representing individual perspective; 2) scores for tramadol for each criterion of the MCDA Core Model using synthesized data; and 3) qualitative impacts of criteria of the Contextual Tool on the appraisal. Utility and reliability of the approach were explored through discussion, survey and test-retest. Agreement between test and retest data was analyzed by calculating intra-rater correlation coefficients (ICCs) for weights, scores and MCDA value estimates.

RESULTS

The framework was found useful by the drug advisory committee in supporting systematic consideration of a broad range of criteria to promote a consistent approach to appraising healthcare interventions. Directly integrated in the framework as a "by-criterion" HTA report, synthesized evidence for each criterion facilitated its consideration, although this was sometimes limited by lack of relevant data. Test-retest analysis showed fair to good consistency of weights, scores and MCDA value estimates at the individual level (ICC ranging from 0.676 to 0.698), thus lending some support for the reliability of the approach. Overall, committee members endorsed the inclusion of most framework criteria and revealed important areas of discussion, clarification and adaptation of the framework to the needs of the committee.

CONCLUSIONS

By promoting systematic consideration of all decision criteria and the underlying evidence, the framework allows a consistent approach to appraising healthcare interventions. Further testing and validation are needed to advance MCDA approaches in healthcare decisionmaking.