Methods to elicit probability distributions from experts: a systematic review of reported practice in health technology assessment

Potential costs and consequences associated with medication error at hospital discharge: an expert judgement study

OBJECTIVES

Assessing the cost-effectiveness of complex pharmaceutical care interventions and medication error outcomes is hindered by lack of available data on actual outcomes consequent to errors that were intercepted for patient safety reasons. Expert judgement is an approach to acquire data regarding unknown parameters in an economic model which are otherwise insufficient or not possible to obtain. The aim of this paper is to describe a method to approach this problem using findings from a single intervention study and to calculate the potential costs and consequences associated with discharge medication error.

METHODS

Using data from a previous intervention study, the hypothetical consequences of medication error(s) at hospital discharge, in terms of diagnosis, healthcare resource utilisation and impact on health-related quality of life, were identified by expert judgement of anonymised cases. Primary healthcare utilisation costs were derived from published tariffs, inpatient costs were derived by simulation in the hospital discharge activity database test environment and the difference between adjudicated baseline and posterror health state was expressed as quality-adjusted life year (QALY) decrement.

RESULTS

Four experts provided judgement on 81 cases. Of these, 75 were judged to have potential clinical consequences. Between 56 and 69 of the 81 cases were variably judged to require remedial healthcare utilisation. The mean calculated cost per case (representing an individual patient), based on all 81 cases, was €1009.58, 95% CI 726.64 to 1585.67. The mean QALY loss was 0.03 (95% CI 0.01 to 0.05).

CONCLUSION

An expert judgement process proved feasible and useful to estimate financial cost and QALY loss associated with hospital discharge medication error. These estimates will be employed in model-based economic evaluation. This method could be transferred to other prospective observational patient safety research which seeks to assess value for money of complex interventions.

Methods for Extrapolating Survival Analyses for the Economic Evaluation of Advanced Therapy Medicinal Products

There are two significant challenges for analysts conducting economic evaluations of advanced therapy medicinal products (ATMPs): (i) estimating long-term treatment effects in the absence of mature clinical data, and (ii) capturing potentially complex hazard functions. This review identifies and critiques a variety of methods that can be used to overcome these challenges. The narrative review is informed by a rapid literature review of methods used for the extrapolation of survival analyses in the economic evaluation of ATMPs. There are several methods that are more suitable than traditional parametric survival modelling approaches for capturing complex hazard functions, including, cure-mixture models and restricted cubic spline models. In the absence of mature clinical data, analysts may augment clinical trial data with data from other sources to aid extrapolation, however, the relative merits of employing methods for including data from different sources is not well understood. Given the high and potentially irrecoverable costs of making incorrect decisions concerning the reimbursement or commissioning of ATMPs, it is important that economic evaluations are correctly specified, and that both parameter and structural uncertainty associated with survival extrapolations are considered. Value of information analyses allow for this uncertainty to be expressed explicitly, and in monetary terms.

Understanding current UK practice for the incidental identification of vertebral fragility fractures from CT scans: an expert elicitation study

Background

There is an emerging interest in using automated approaches to enable the incidental identification of vertebral fragility fractures (VFFs) on existing medical images visualising the spine.

Aim

To quantify values, and the degree of uncertainty associated with them, for the incidental identification of VFFs from computed tomography (CT) scans in current practice.

Methods

An expert elicitation exercise was conducted to generate point estimates and measures of uncertainty for four values representing the probability of: VFF being correctly reported by the radiologist; the absence of VFF being correctly assessed by the radiologist; being referred for management when a VFF is identified; having a dual-energy X-ray absorptiometry (DXA) scan after general practitioner (GP) referral. Data from a sample of seven experts in the diagnosis and management of people with VFFs were pooled using mathematical aggregation.

Results

The estimated mean values for each probability parameter were: VFF being correctly reported by the radiologist = 0.25 (standard deviation (SD): 0.21); absence of VFF being correctly assessed by the radiologist = 0.89 (0.10); being referred for management when a VFF is identified by the radiologist = 0.15 (0.12); having a DXA scan after GP referral = 0.66 (0.28).

Discussion

These estimates could be used to facilitate the subsequent early economic evaluation of potential new approaches to improve the health outcomes of people with VFFs.

Conclusion

In the absence of epidemiological studies, this study produced point estimates and measures of uncertainty for key parameters needed to describe current pathways for the incidental diagnosis of VFFs.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40520-022-02124-w.

Expert agreement in prior elicitation and its effects on Bayesian inference

Bayesian inference requires the specification of prior distributions that quantify the pre-data uncertainty about parameter values. One way to specify prior distributions is through prior elicitation, an interview method guiding field experts through the process of expressing their knowledge in the form of a probability distribution. However, prior distributions elicited from experts can be subject to idiosyncrasies of experts and elicitation procedures, raising the spectre of subjectivity and prejudice. Here, we investigate the effect of interpersonal variation in elicited prior distributions on the Bayes factor hypothesis test. We elicited prior distributions from six academic experts with a background in different fields of psychology and applied the elicited prior distributions as well as commonly used default priors in a re-analysis of 1710 studies in psychology. The degree to which the Bayes factors vary as a function of the different prior distributions is quantified by three measures of concordance of evidence: We assess whether the prior distributions change the Bayes factor direction, whether they cause a switch in the category of evidence strength, and how much influence they have on the value of the Bayes factor. Our results show that although the Bayes factor is sensitive to changes in the prior distribution, these changes do not necessarily affect the qualitative conclusions of a hypothesis test. We hope that these results help researchers gauge the influence of interpersonal variation in elicited prior distributions in future psychological studies. Additionally, our sensitivity analyses can be used as a template for Bayesian robustness analyses that involve prior elicitation from multiple experts.

Reference Case Methods for Expert Elicitation in Health Care Decision Making

BACKGROUND

The evidence used to inform health care decision making (HCDM) is typically uncertain. In these situations, the experience of experts is essential to help decision makers reach a decision. Structured expert elicitation (referred to as elicitation) is a quantitative process to capture experts' beliefs. There is heterogeneity in the existing elicitation methodology used in HCDM, and it is not clear if existing guidelines are appropriate for use in this context. In this article, we seek to establish reference case methods for elicitation to inform HCDM.

METHODS

We collated the methods available for elicitation using reviews and critique. In addition, we conducted controlled experiments to test the accuracy of alternative methods. We determined the suitability of the methods choices for use in HCDM according to a predefined set of principles for elicitation in HCDM, which we have also generated. We determined reference case methods for elicitation in HCDM for health technology assessment (HTA).

RESULTS

In almost all methods choices available for elicitation, we found a lack of empirical evidence supporting recommendations. Despite this, it is possible to define reference case methods for HTA. The reference methods include a focus on gathering experts with substantive knowledge of the quantities being elicited as opposed to those trained in probability and statistics, eliciting quantities that the expert might observe directly, and individual elicitation of beliefs, rather than solely consensus methods. It is likely that there are additional considerations for decision makers in health care outside of HTA.

CONCLUSIONS

The reference case developed here allows the use of different methods, depending on the decision-making setting. Further applied examples of elicitation methods would be useful. Experimental evidence comparing methods should be generated.

Integrating expert knowledge for dementia risk prediction in individuals with mild cognitive impairment (MCI): a study protocol

INTRODUCTION

To date, there is no broadly accepted dementia risk score for use in individuals with mild cognitive impairment (MCI), partly because there are few large datasets available for model development. When evidence is limited, the knowledge and experience of experts becomes more crucial for risk stratification and providing MCI patients with prognosis. Structured expert elicitation (SEE) includes formal methods to quantify experts' beliefs and help experts to express their beliefs in a quantitative form, reducing biases in the process. This study proposes to (1) assess experts' beliefs about important predictors for 3-year dementia risk in persons with MCI through SEE methodology and (2) to integrate expert knowledge and patient data to derive dementia risk scores in persons with MCI using a Bayesian approach.

METHODS AND ANALYSIS

This study will use a combination of SEE methodology, prospectively collected clinical data, and statistical modelling to derive a dementia risk score in persons with MCI . Clinical expert knowledge will be quantified using SEE methodology that involves the selection and training of the experts, administration of questionnaire for eliciting expert knowledge, discussion meetings and results aggregation. Patient data from the Prospective Registry for Persons with Memory Symptoms of the Cognitive Neurosciences Clinic at the University of Calgary; the Alzheimer's Disease Neuroimaging Initiative; and the National Alzheimer's Coordinating Center's Uniform Data Set will be used for model training and validation. Bayesian Cox models will be used to incorporate patient data and elicited data to predict 3-year dementia risk.

DISCUSSION

This study will develop a robust dementia risk score that incorporates clinician expert knowledge with patient data for accurate risk stratification, prognosis and management of dementia.

The Use of Expert Elicitation among Computational Modeling Studies in Health Research: A Systematic Review

BACKGROUND

Expert elicitation (EE) has been used across disciplines to estimate input parameters for computational modeling research when information is sparse or conflictual.

OBJECTIVES

We conducted a systematic review to compare EE methods used to generate model input parameters in health research.

DATA SOURCES

PubMed and Web of Science.

STUDY ELIGIBILITY

Modeling studies that reported the use of EE as the source for model input probabilities were included if they were published in English before June 2021 and reported health outcomes.

DATA ABSTRACTION AND SYNTHESIS

Studies were classified as "formal" EE methods if they explicitly reported details of their elicitation process. Those that stated use of expert opinion but provided limited information were classified as "indeterminate" methods. In both groups, we abstracted citation details, study design, modeling methodology, a description of elicited parameters, and elicitation methods. Comparisons were made between elicitation methods.

STUDY APPRAISAL

Studies that conducted a formal EE were appraised on the reporting quality of the EE. Quality appraisal was not conducted for studies of indeterminate methods.

RESULTS

The search identified 1520 articles, of which 152 were included. Of the included studies, 40 were classified as formal EE and 112 as indeterminate methods. Most studies were cost-effectiveness analyses (77.6%). Forty-seven indeterminate method studies provided no information on methods for generating estimates. Among formal EEs, the average reporting quality score was 9 out of 16.

LIMITATIONS

Elicitations on nonhealth topics and those reported in the gray literature were not included.

CONCLUSIONS

We found poor reporting of EE methods used in modeling studies, making it difficult to discern meaningful differences in approaches. Improved quality standards for EEs would improve the validity and replicability of computational models.

HIGHLIGHTS

We find extensive use of expert elicitation for the development of model input parameters, but most studies do not provide adequate details of their elicitation methods.Lack of reporting hinders greater discussion of the merits and challenges of using expert elicitation for model input parameter development.There is a need to establish expert elicitation best practices and reporting guidelines.

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study

BACKGROUND

Many decisions in health care aim to maximise health, requiring judgements about interventions that may have higher health effects but potentially incur additional costs (cost-effectiveness framework). The evidence used to establish cost-effectiveness is typically uncertain and it is important that this uncertainty is characterised. In situations in which evidence is uncertain, the experience of experts is essential. The process by which the beliefs of experts can be formally collected in a quantitative manner is structured expert elicitation. There is heterogeneity in the existing methodology used in health-care decision-making. A number of guidelines are available for structured expert elicitation; however, it is not clear if any of these are appropriate for health-care decision-making.

OBJECTIVES

The overall aim was to establish a protocol for structured expert elicitation to inform health-care decision-making. The objectives are to (1) provide clarity on methods for collecting and using experts' judgements, (2) consider when alternative methodology may be required in particular contexts, (3) establish preferred approaches for elicitation on a range of parameters, (4) determine which elicitation methods allow experts to express uncertainty and (5) determine the usefulness of the reference protocol developed.

METHODS

A mixed-methods approach was used: systemic review, targeted searches, experimental work and narrative synthesis. A review of the existing guidelines for structured expert elicitation was conducted. This identified the approaches used in existing guidelines (the 'choices') and determined if dominant approaches exist. Targeted review searches were conducted for selection of experts, level of elicitation, fitting and aggregation, assessing accuracy of judgements and heuristics and biases. To sift through the available choices, a set of principles that underpin the use of structured expert elicitation in health-care decision-making was defined using evidence generated from the targeted searches, quantities to elicit experimental evidence and consideration of constraints in health-care decision-making. These principles, including fitness for purpose and reflecting individual expert uncertainty, were applied to the set of choices to establish a reference protocol. An applied evaluation of the developed reference protocol was also undertaken.

RESULTS

For many elements of structured expert elicitation, there was a lack of consistency across the existing guidelines. In almost all choices, there was a lack of empirical evidence supporting recommendations, and in some circumstances the principles are unable to provide sufficient justification for discounting particular choices. It is possible to define reference methods for health technology assessment. These include a focus on gathering experts with substantive skills, eliciting observable quantities and individual elicitation of beliefs. Additional considerations are required for decision-makers outside health technology assessment, for example at a local level, or for early technologies. Access to experts may be limited and in some circumstances group discussion may be needed to generate a distribution.

LIMITATIONS

The major limitation of the work conducted here lies not in the methods employed in the current work but in the evidence available from the wider literature relating to how appropriate particular methodological choices are.

CONCLUSIONS

The reference protocol is flexible in many choices. This may be a useful characteristic, as it is possible to apply this reference protocol across different settings. Further applied studies, which use the choices specified in this reference protocol, are required.

FUNDING

This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 37. See the NIHR Journals Library website for further project information. This work was also funded by the Medical Research Council (reference MR/N028511/1).

How Should the Value Attributes of Novel Antibiotics Be Considered in Reimbursement Decision Making?

Antibiotics have revolutionized the treatment of bacterial infections. However, it is widely held that there is underinvestment in antibiotics research and development relative to the socially optimal level for a number of reasons. In this article, we discuss whether existing health technology assessment procedures recognize the full economic and societal value of new antibiotics to patients and society when making reimbursement decisions. We present three recommendations for modelling the unique attributes of value that are specific to novel antibiotics. We find, based on a review of the literature, that some of the value elements proposed by our framework have previously been discussed qualitatively by health technology assessment bodies when evaluating antibiotics, but are not yet formally captured via modelling. We present a worked example to show how it may be possible to capture these dimensions of value in a more quantitative manner. We conclude by answering the question of the title as follows: the unique attributes of novel antibiotics should be considered in reimbursement decision making, in a way that captures the full range of benefits these important technologies bring to patients, health care systems, and society.

Expert Elicitation to Inform a Cost-Effectiveness Analysis of Screening for Renal Cancer

BACKGROUND

Population screening for renal cell carcinoma (RCC) using ultrasound has the potential to improve survival outcomes; however, a cost-effectiveness analysis (CEA) has yet to be performed. Owing to the lack of existing evidence, we performed structured expert elicitation to derive unknown quantities to inform the CEA.

OBJECTIVE

To elicit the cancer stage distribution (proportion of individuals with each stage of cancer) for different RCC screening scenarios and the annual transition probabilities for undiagnosed disease becoming diagnosed in the National Health Service.

METHODS

The study design and reporting adhered to the Reporting Guidelines for the Use of Expert Judgement in Model-Based Economic Evaluations. The elicitation was conducted face-to-face or via telephone between each individual expert and the facilitator, aided by online material. For multinomial data, Connor-Mosimann and modified Connor-Mosimann distributions were fitted for each expert and for all experts combined using mathematical linear pooling.

RESULTS

A total of 24 clinical experts were invited, and 71% participated (7 urologists, 6 oncologists, 4 radiologists). The modified Connor-Mosimann distribution provided the best fit for most elicited quantities. Greater uncertainty was noted for the elicited transition probabilities compared with the elicited stage distributions.

CONCLUSION

We performed the first expert elicitation of RCC screening parameters, crucial information that will inform the CEA of screening. In addition, the elicited quantities may enable future health economic evaluations assessing the value of diagnostic tools and pathways in RCC.

Estimating Future Health Technology Diffusion Using Expert Beliefs Calibrated to an Established Diffusion Model

OBJECTIVES

Estimates of future health technology diffusion, or future uptake over time, are a requirement for different analyses performed within health technology assessments. Methods for obtaining such estimates include constant uptake estimates based on expert opinion or analogous technologies and on extrapolation from initial data points using parametric curves-but remain divorced from established diffusion theory and modeling. We propose an approach to obtaining diffusion estimates using experts' beliefs calibrated to an established diffusion model to address this methodologic gap.

METHODS

We performed an elicitation of experts' beliefs on future diffusion of a new preterm birth screening illustrative case study technology. The elicited quantities were chosen such that they could be calibrated to yield the parameters of the Bass model of new product growth, which was chosen based on a review of the diffusion literature.

RESULTS

With the elicitation of only three quantities per diffusion curve, our approach enabled us to quantify uncertainty about diffusion of the new technology in different scenarios. Pooled results showed that the attainable number of adoptions was predicted to be relatively low compared with what was thought possible. Further research evidence improved the attainable number of adoptions only slightly but resulted in greater speed of diffusion.

CONCLUSIONS

The proposed approach of eliciting experts' beliefs about diffusion and informing the Bass model has the potential to fill the methodologic gap evident in value of implementation and research, as well as budget impact and some cost-effectiveness analyses.

Experiences of Structured Elicitation for Model-Based Cost-Effectiveness Analyses

BACKGROUND

Empirical evidence supporting the cost-effectiveness estimates of particular health care technologies may be limited, or it may even be missing entirely. In these situations, additional information, often in the form of expert judgments, is needed to reach a decision. There are formal methods to quantify experts' beliefs, termed as structured expert elicitation (SEE), but only limited research is available in support of methodological choices. Perhaps as a consequence, the use of SEE in the context of cost-effectiveness modelling is limited.

OBJECTIVES

This article reviews applications of SEE in cost-effectiveness modelling with the aim of summarizing the basis for methodological choices made in each application and recording the difficulties and challenges reported by the authors in the design, conduct, and analyses.

METHODS

The methods used in each application were extracted along with the criteria used to support methodological and practical choices and any issues or challenges discussed in the text. Issues and challenges were extracted using an open field, and then categorised and grouped for reporting.

RESULTS

The review demonstrates considerable heterogeneity in methods used, and authors acknowledge great methodological uncertainty in justifying their choices. Specificities of the context area emerging as potentially important in determining further methodological research in elicitation are between- expert variation and its interpretation, the fact that substantive experts in the area may not be trained in quantitative subjects, that judgments are often needed on various parameter types, the need for some form of assessment of validity, and the need for more integration with behavioural research to devise relevant debiasing strategies.

CONCLUSIONS

This review of experiences of SEE highlights a number of specificities/constraints that can shape the development of guidance and target future research efforts in this area.

EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment

BACKGROUND

Expert opinion is often sought to complement available information needed to inform model-based economic evaluations in health technology assessments. In this context, we define expert elicitation as the process of encoding expert opinion on a quantity of interest, together with associated uncertainty, as a probability distribution. When availability for face-to-face expert elicitation with a facilitator is limited, elicitation can be conducted remotely, overcoming challenges of finding an appropriate time to meet the expert and allowing access to experts situated too far away for practical face-to-face sessions. However, distance elicitation is associated with reduced response rates and limited assistance for the expert during the elicitation session. The aim of this study was to inform the development of a remote elicitation tool by exploring the influence of mode of elicitation on elicited beliefs.

METHODS

An Excel-based tool (EXPLICIT) was developed to assist the elicitation session, including the preparation of the expert and recording of their responses. General practitioners (GPs) were invited to provide expert opinion about population alcohol consumption behaviours. They were randomised to complete the elicitation by either a face-to-face meeting or email. EXPLICIT was used in the elicitation sessions for both arms.

RESULTS

Fifteen GPs completed the elicitation session. Those conducted by email were longer than the face-to-face sessions (13 min 30 s vs 10 min 26 s, p = 0.1) and the email-elicited estimates contained less uncertainty. However, the resulting aggregated distributions were comparable.

CONCLUSIONS

EXPLICIT was useful in both facilitating the elicitation task and in obtaining expert opinion from experts via email. The findings support the opinion that remote, self-administered elicitation is a viable approach within the constraints of HTA to inform policy making, although poor response rates may be observed and additional time for individual sessions may be required.

Informing Reimbursement Decisions Using Cost-Effectiveness Modelling: A Guide to the Process of Generating Elicited Priors to Capture Model Uncertainties

In informing decisions, utilising health technology assessment (HTA), expert elicitation can provide valuable information, particularly where there is a less-developed evidence-base at the point of market access. In these circumstances, formal methods to elicit expert judgements are preferred to improve the accountability and transparency of the decision-making process, help reduce bias and the use of heuristics, and also provide a structure that allows uncertainty to be expressed. Expert elicitation is the process of transforming the subjective and implicit knowledge of experts into their quantifiable expressions. The use of expert elicitation in HTA is gaining momentum, and there is particular interest in its application to diagnostics, medical devices and complex interventions such as in public health or social care. Compared with the gathering of experimental evidence, elicitation constitutes a reasonably low-cost source of evidence. Given its inherent subject nature, the potential biases in elicited evidence cannot be ignored and, due to its infancy in HTA, there is little guidance to the analyst wishing to conduct a formal elicitation exercise. This article attempts to summarise the stages of designing and conducting an expert elicitation, drawing on key literature and examples, most of which are not in HTA. In addition, we critique their applicability to HTA, given its distinguishing features. There are a number of issues that the analyst should be mindful of, in particular the need to appropriately characterise the uncertainty associated with model inputs and the fact that there are often numerous parameters required, not all of which can be defined using the same quantities. This increases the need for the elicitation task to be as straightforward as possible for the expert to complete.

Reporting Guidelines for the Use of Expert Judgement in Model-Based Economic Evaluations

INTRODUCTION

Expert judgement has a role in model-based economic evaluations (EEs) of healthcare interventions. This study aimed to produce reporting criteria for two types of study design to use expert judgement in model-based EE: (i) an expert elicitation (quantitative) study; and (ii) a Delphi study to collate (qualitative) expert opinion.

METHODS

A two-round online Delphi process identified the degree of consensus for four core definitions (expert; expert parameter values; expert elicitation study; expert opinion) and two sets of reporting criteria in a purposive sample of experts. The initial set of reporting criteria comprised 17 statements for reporting a study to elicit parameter values and/or distributions and 11 statements for reporting a Delphi survey to obtain expert opinion. Fifty experts were invited to become members of the Delphi process panel by e-mail. Data analysis summarised the extent of agreement (using a pre-defined 75 % 'consensus' threshold) on the definitions and suggested reporting criteria. Free-text comments were analysed using thematic analysis.

RESULTS

The final panel comprised 12 experts. Consensus was achieved for the definitions of expert (88 %); expert parameter values (83 %); and expert elicitation study (83 %). The panel recommended criteria to use when reporting an expert elicitation study (16 criteria) and a Delphi study to collate expert opinion (11 criteria).

CONCLUSION

This study has produced guidelines for reporting two types of study design to use expert judgement in model-based EE: (i) an expert elicitation study requiring 16 reporting criteria; and (ii) a Delphi study to collate expert opinion requiring 11 reporting criteria.

USING EXPERT OPINION IN HEALTH TECHNOLOGY ASSESSMENT: A GUIDELINE REVIEW

OBJECTIVES

External experts can be consulted at different stages of an HTA. When using vague information sources, it is particularly important to plan, analyze, and report the information processing in a standardized and transparent way. Our objective was to search and analyze recommendations regarding where and how to include expert data in HTA.

METHODS

We performed a systematic database search and screened the Internet pages of seventy-seven HTA organizations for guidelines, recommendations, and methods papers that address the inclusion of experts in HTA. Relevant documents were downloaded, and information was extracted in a standard form. Results were merged in tables and narrative evidence synthesis.

RESULTS

From twenty-two HTA organizations, we included forty-two documents that consider the use of expert opinion in HTA. Nearly all documents mention experts in the step of preparation of the evidence report. Six documents address their role for priority setting of topics, fifteen for scoping, twelve for the appraisal of evidence and results, another twelve documents mention experts when considering the dissemination of HTA results. During the assessment step, experts are most often asked to amend the literature search or to provide expertise for special data analyses. Another issue for external experts is to appraise the HTA results and refer them back to a clinical and social context. Little is reported on methods of expert elicitation when their input substitutes study data.

CONCLUSIONS

Despite existing recommendations on the use of expert opinion in HTA, common standards for elicitation are scarce in HTA guidelines.

A comparison of two methods for expert elicitation in health technology assessments

BACKGROUND

When data needed to inform parameters in decision models are lacking, formal elicitation of expert judgement can be used to characterise parameter uncertainty. Although numerous methods for eliciting expert opinion as probability distributions exist, there is little research to suggest whether one method is more useful than any other method. This study had three objectives: (i) to obtain subjective probability distributions characterising parameter uncertainty in the context of a health technology assessment; (ii) to compare two elicitation methods by eliciting the same parameters in different ways; (iii) to collect subjective preferences of the experts for the different elicitation methods used.

METHODS

Twenty-seven clinical experts were invited to participate in an elicitation exercise to inform a published model-based cost-effectiveness analysis of alternative treatments for prostate cancer. Participants were individually asked to express their judgements as probability distributions using two different methods - the histogram and hybrid elicitation methods - presented in a random order. Individual distributions were mathematically aggregated across experts with and without weighting. The resulting combined distributions were used in the probabilistic analysis of the decision model and mean incremental cost-effectiveness ratios and the expected values of perfect information (EVPI) were calculated for each method, and compared with the original cost-effectiveness analysis. Scores on the ease of use of the two methods and the extent to which the probability distributions obtained from each method accurately reflected the expert's opinion were also recorded.

RESULTS

Six experts completed the task. Mean ICERs from the probabilistic analysis ranged between £162,600-£175,500 per quality-adjusted life year (QALY) depending on the elicitation and weighting methods used. Compared to having no information, use of expert opinion decreased decision uncertainty: the EVPI value at the £30,000 per QALY threshold decreased by 74-86 % from the original cost-effectiveness analysis. Experts indicated that the histogram method was easier to use, but attributed a perception of more accuracy to the hybrid method.

CONCLUSIONS

Inclusion of expert elicitation can decrease decision uncertainty. Here, choice of method did not affect the overall cost-effectiveness conclusions, but researchers intending to use expert elicitation need to be aware of the impact different methods could have.

Electrocardiographic Screening for Prolonged QT Interval to Reduce Sudden Cardiac Death in Psychiatric Patients: A Cost-Effectiveness Analysis

IMPORTANCE

Sudden cardiac death is a leading cause of mortality in psychiatric patients. Long QT (LQT) is common in this population and predisposes to Torsades-de-Pointes (TdP) and subsequent mortality.

OBJECTIVE

To estimate the cost-effectiveness of electrocardiographic screening to detect LQT in psychiatric inpatients.

DESIGN, SETTING, AND PARTICIPANTS

We built a decision analytic model based on a decision tree to evaluate the cost-effectiveness and utility of LQT screening from a health care perspective. LQT proportion parameters were derived from an in-hospital cross-sectional study. We performed experts' elicitation to estimate the risk of TdP, given extent of QT prolongation. A TdP reduction of 65% after LQT detection was based on positive drug dechallenge rate and through adequate treatment and electrolyte adjustments. The base-case model uncertainty was assessed with one-way and probabilistic sensitivity analyses. Finally, the TdP related mortality and TdP avoidance parameters were varied in a two-way sensitivity analysis to assess their effect on the Incremental Cost-Effectiveness Ratio (ICER).

MAIN OUTCOMES AND MEASURES

Costs, Quality Ajusted Life Year (QALY), ICER, and probability of cost effectiveness thresholds ($ 10,000, $25,000, and $50,000 per QALY).

RESULTS

In the base-case scenario, the numbers of patients needed to screen were 1128 and 2817 to avoid one TdP and one death, respectively. The ICER of systematic ECG screening was $8644 (95%CI, 3144-82 498) per QALY. The probability of cost-effectiveness was 96% at a willingness-to-pay of $50,000 for one QALY. In sensitivity analyses, results were sensitive to the case-fatality of TdP episodes and to the TdP reduction following the diagnosis of LQT.

CONCLUSION AND RELEVANCE

In psychiatric hospitals, performing systematic ECG screening at admission help reduce the number of sudden cardiac deaths in a cost-effective fashion.

Concepts of 'personalization' in personalized medicine: implications for economic evaluation

CONTEXT

This study assesses if, and how, existing methods for economic evaluation are applicable to the evaluation of personalized medicine (PM) and, if not, where extension to methods may be required.

METHODS

A structured workshop was held with a predefined group of experts (n = 47), and was run using a modified nominal group technique. Workshop findings were recorded using extensive note taking, and summarized using thematic data analysis. The workshop was complemented by structured literature searches.

RESULTS

The key finding emerging from the workshop, using an economic perspective, was that two distinct, but linked, interpretations of the concept of PM exist (personalization by 'physiology' or 'preferences'). These interpretations involve specific challenges for the design and conduct of economic evaluations. Existing evaluative (extra-welfarist) frameworks were generally considered appropriate for evaluating PM. When 'personalization' is viewed as using physiological biomarkers, challenges include representing complex care pathways; representing spillover effects; meeting data requirements such as evidence on heterogeneity; and choosing appropriate time horizons for the value of further research in uncertainty analysis. When viewed as tailoring medicine to patient preferences, further work is needed regarding revealed preferences, e.g. treatment (non)adherence; stated preferences, e.g. risk interpretation and attitude; consideration of heterogeneity in preferences; and the appropriate framework (welfarism vs. extra-welfarism) to incorporate non-health benefits.

CONCLUSIONS

Ideally, economic evaluations should take account of both interpretations of PM and consider physiology and preferences. It is important for decision makers to be cognizant of the issues involved with the economic evaluation of PM to appropriately interpret the evidence and target future research funding.