Understanding the stakeholders' intention to use economic decision-support tools: A cross-sectional study with the tobacco return on investment tool

Knowledge, socio-cognitive perceptions and the practice of hand hygiene and social distancing during the COVID-19 pandemic: a cross-sectional study of UK university students

BACKGROUND

During the first wave of the COVID-19 pandemic, social distancing and hand hygiene have been the primary means of reducing transmission in the absence of effective treatments or vaccines, but understanding of their determinants is limited. This study aimed to investigate knowledge and socio-cognitive perceptions, and their associations with such protective behaviours, in UK university students.

METHODS

A cross-sectional online survey of 293 students was undertaken on 13 May 2020. Survey questions addressed demographics, knowledge of the disease and effectiveness of the protective measures, risk perception, socio-cognitive perceptions (e.g. attitude, social support, and self-efficacy), habit, time factors and trust, as well as the hand hygiene and social distancing behaviours. Multiple linear regression was used to identify the strongest associations of potential determinants with behaviour.

RESULTS

Participants reported high levels of social distancing with 88.9% answering "Mostly" or "Always" for every activity, but only 42.0% reporting the same for all hand hygiene activities. Knowledge of the effectiveness of each activity in preventing transmission was high, with 90.7% and 93.5% respectively identifying at least 7 of 8 hand hygiene or 9 of 10 social distancing activities correctly. Habit (β = 0.39, p = 0.001) and time factors (β = 0.28, p = 0.001) were the greatest contributors to unique variance in hand hygiene behaviour, followed by ethnicity (β = - 0.13, p = 0.014) and risk perception (β = 0.13, p = 0.016). For social distancing behaviour, the determinants were self-efficacy (β = 0.25, p < 0.001), perceived advantages (β = 0.15, p = 0.022), trust in policy (β = 0.14, p = 0.026) and gender (β = - 0.14, p = 0.016). Regression models explained 40% hand hygiene and 25% social distancing variance.

CONCLUSIONS

This study indicated that communications about effectiveness of hand hygiene and social distancing behaviours had been effective in terms of knowledge acquisition. However, in the light of likely second waves of COVID-19, attention to maintaining social distancing behaviour and improving hand hygiene behaviour may need to address more difficult areas of changing habits, overcoming time factors and building trust, as well as interventions to increase self-efficacy and address risk perception concerns.

Applying an Implementation Framework to the Use of Evidence from Economic Evaluations in Making Healthcare Decisions

BACKGROUND AND OBJECTIVE

There is a need for the application of theory in understanding the use of evidence from economic evaluations in healthcare decision making. The purpose of this study is to review the published literature on the use of evidence from economic evaluations for healthcare decision making and to map the findings to the Consolidated Framework for Implementation Research (CFIR).

METHODS

A systematic search strategy was used to identify studies investigating the factors that determine the use of evidence from economic evaluation in healthcare decision making. Barriers and facilitators identified in the included studies were mapped across the five CFIR domains, with the "intervention" referring to the use of economic evaluations in decision making. Gaps, inconsistencies and emergent relations were identified through the mapping process.

RESULTS

Fifty-three studies met eligibility criteria and were included in the review. The CFIR constructs associated with the Intervention Characteristics and those associated with the knowledge and beliefs of users of economic evaluations were widely cited in the identified barriers and facilitators. Other constructs from the CFIR had not been reported in the literature, such as 'organisational networks' and 'individual stage of change'. Most of the stages in the implementation process as described by the CFIR were reflected in the identified barriers and facilitators.

DISCUSSION

By categorising barriers and facilitators into domains, the CFIR provides a systematic approach to assess how these factors interact. Literature gaps in the literature regarding the use of economic evaluation in healthcare decision making were identified, specifically issues regarding organisational networks and the role of feedback.

CONCLUSIONS

Through mapping findings from studies of the use of evidence from economic evaluations in healthcare decision making, we present an implementation framework based on the CFIR for understanding the use of economic evaluations into practice.

How to engage stakeholders in research: design principles to support improvement

Background

Closing the gap between research production and research use is a key challenge for the health research system. Stakeholder engagement is being increasingly promoted across the board by health research funding organisations, and indeed by many researchers themselves, as an important pathway to achieving impact. This opinion piece draws on a study of stakeholder engagement in research and a systematic literature search conducted as part of the study.

Main body

This paper provides a short conceptualisation of stakeholder engagement, followed by ‘design principles’ that we put forward based on a combination of existing literature and new empirical insights from our recently completed longitudinal study of stakeholder engagement. The design principles for stakeholder engagement are organised into three groups, namely organisational, values and practices. The organisational principles are to clarify the objectives of stakeholder engagement; embed stakeholder engagement in a framework or model of research use; identify the necessary resources for stakeholder engagement; put in place plans for organisational learning and rewarding of effective stakeholder engagement; and to recognise that some stakeholders have the potential to play a key role. The principles relating to values are to foster shared commitment to the values and objectives of stakeholder engagement in the project team; share understanding that stakeholder engagement is often about more than individuals; encourage individual stakeholders and their organisations to value engagement; recognise potential tension between productivity and inclusion; and to generate a shared commitment to sustained and continuous stakeholder engagement. Finally, in terms of practices, the principles suggest that it is important to plan stakeholder engagement activity as part of the research programme of work; build flexibility within the research process to accommodate engagement and the outcomes of engagement; consider how input from stakeholders can be gathered systematically to meet objectives; consider how input from stakeholders can be collated, analysed and used; and to recognise that identification and involvement of stakeholders is an iterative and ongoing process.

Conclusion

It is anticipated that the principles will be useful in planning stakeholder engagement activity within research programmes and in monitoring and evaluating stakeholder engagement. A next step will be to address the remaining gap in the stakeholder engagement literature concerned with how we assess the impact of stakeholder engagement on research use.

Electronic supplementary material

The online version of this article (10.1186/s12961-018-0337-6) contains supplementary material, which is available to authorized users.

A utility of model input uncertainty analysis in transferring tobacco control-related economic evidence to countries with scarce resources: results from the EQUIPT study

AIMS

To inform the transferability of tobacco control-related economic evidence to resource-poor countries.

METHODS

We ran a univariate sensitivity analysis on a return on investment (ROI) model, the European study on Quantifying Utility of Investment in Protection from Tobacco model (EQUIPTMOD), to identify key input values to which the ROI estimates were sensitive. The EQUIPTMOD used a Markov-based state transition model to estimate the ROI of several tobacco control interventions in five European countries (England, Germany, Spain, Hungary and the Netherlands). Base case ROI estimates were obtained through average values of model inputs (throughout the five countries), which were then replaced one at a time with country-specific values. Tornado diagrams were used to evaluate the significance of sensitivity, defined as a ≥ 10% difference in ROI estimates from the base case estimates.

RESULTS

The ROI estimates were sensitive to 18 (of 46) input values. Examples of model inputs to which ROI estimates were sensitive included: smoking rate, costs of smoking-related diseases (e.g. lung cancer) and general population attributes.

CONCLUSION

Countries that have limited research time and other resources can adapt EQUIPTMOD to their own settings by choosing to collect data on a small number of model inputs. EQUIPTMOD can therefore facilitate transfer of tobacco control related economic evidence to new jurisdictions.

Development and application of an economic model (EQUIPTMOD) to assess the impact of smoking cessation

BACKGROUND AND AIMS

Although clear benefits are associated with reducing smoking, there is increasing pressure on public health providers to justify investment in tobacco control measures. Decision-makers need tools to assess the Return on Investment (ROI)/cost-effectiveness of programmes. The EQUIPT project adapted an ROI tool for England to four European countries (Germany, the Netherlands, Spain and Hungary). EQUIPTMOD, the economic model at the core of the ROI tool, is designed to assess the efficiency of packages of smoking cessation interventions. The objective of this paper is to describe the methods for EQUIPTMOD and identify key outcomes associated with continued and cessation of smoking.

METHODS

EQUIPTMOD uses a Markov model to estimate life-time costs, quality-adjusted life years (QALYs) and life years associated with a current and former smoker. It uses population data on smoking prevalence, disease prevalence, mortality and the impact of smoking combined with associated costs and utility effects of disease. To illustrate the tool's potential, costs, QALYs and life expectancy were estimated for the average current smoker for five countries based on the assumptions that they continue and that they cease smoking over the next 12 months. Costs and effects were discounted at country-specific rates.

RESULTS

For illustration, over a life-time horizon, not quitting smoking within the next 12 months in England will reduce life expectancy by 0.66, reduce QALYs by 1.09 and result in £4961 higher disease-related health care costs than if the smoker ceased smoking in the next 12 months. For all age-sex categories, costs were lower and QALYs higher for those who quit smoking in the 12 months than those who continued.

CONCLUSIONS

EQUIPTMOD facilitates assessment of the cost effectiveness of smoking cessation strategies. The demonstrated results indicate large potential benefits from smoking cessation at both an individual and population level.

Cost-effectiveness of increasing the reach of smoking cessation interventions in Germany: results from the EQUIPTMOD

AIMS

To evaluate costs, effects and cost-effectiveness of increased reach of specific smoking cessation interventions in Germany.

DESIGN

A Markov-based state transition return on investment model (EQUIPTMOD) was used to evaluate current smoking cessation interventions as well as two prospective investment scenarios. A health-care perspective (extended to include out-of-pocket payments) with life-time horizon was considered. A probabilistic analysis was used to assess uncertainty concerning predicted estimates.

SETTING

Germany.

PARTICIPANTS

Cohort of current smoking population (18+ years) in Germany.

INTERVENTIONS

Interventions included group-based behavioural support, financial incentive programmes and varenicline. For prospective scenario 1 the reach of group-based behavioral support, financial incentive programme and varenicline was increased by 1% of yearly quit attempts (= 57 915 quit attempts), while prospective scenario 2 represented a higher reach, mirroring the levels observed in England.

MEASUREMENTS

EQUIPTMOD considered reach, intervention cost, number of quitters, quality-of-life years (QALYs) gained, cost-effectiveness and return on investment.

FINDINGS

The highest returns through reduction in smoking-related health-care costs were seen for the financial incentive programme (€2.71 per €1 invested), followed by that of group-based behavioural support (€1.63 per €1 invested), compared with no interventions. Varenicline had lower returns (€1.02 per €1 invested) than the other two interventions. At the population level, prospective scenario 1 led to 15 034 QALYs gained and €27 million cost-savings, compared with current investment. Intervention effects and reach contributed most to the uncertainty around the return-on-investment estimates. At a hypothetical willingness-to-pay threshold of only €5000, the probability of being cost-effective is approximately 75% for prospective scenario 1.

CONCLUSIONS

Increasing the reach of group-based behavioural support, financial incentives and varenicline for smoking cessation by just 1% of current annual quit attempts provides a strategy to German policymakers that improves the population's health outcomes and that may be considered cost-effective.

OPTIMIZING USABILITY OF AN ECONOMIC DECISION SUPPORT TOOL: PROTOTYPE OF THE EQUIPT TOOL

OBJECTIVES

Economic decision-support tools can provide valuable information for tobacco control stakeholders, but their usability may impact the adoption of such tools. This study aims to illustrate a mixed-method usability evaluation of an economic decision-support tool for tobacco control, using the EQUIPT ROI tool prototype as a case study.

METHODS

A cross-sectional mixed methods design was used, including a heuristic evaluation, a thinking aloud approach, and a questionnaire testing and exploring the usability of the Return of Investment tool.

RESULTS

A total of sixty-six users evaluated the tool (thinking aloud) and completed the questionnaire. For the heuristic evaluation, four experts evaluated the interface. In total twenty-one percent of the respondents perceived good usability. A total of 118 usability problems were identified, from which twenty-six problems were categorized as most severe, indicating high priority to fix them before implementation.

CONCLUSIONS

Combining user-based and expert-based evaluation methods is recommended as these were shown to identify unique usability problems. The evaluation provides input to optimize usability of a decision-support tool, and may serve as a vantage point for other developers to conduct usability evaluations to refine similar tools before wide-scale implementation. Such studies could reduce implementation gaps by optimizing usability, enhancing in turn the research impact of such interventions.

Understanding perceived availability and importance of tobacco control interventions to inform European adoption of a UK economic model: a cross-sectional study

BACKGROUND

The evidence on the extent to which stakeholders in different European countries agree with availability and importance of tobacco-control interventions is limited. This study assessed and compared stakeholders' views from five European countries and compared the perceived ranking of interventions with evidence-based ranking using cost-effectiveness data.

METHODS

An interview survey (face-to-face, by phone or Skype) was conducted between April and July 2014 with five categories of stakeholders - decision makers, service purchasers, service providers, evidence generators and health promotion advocates - from Germany, Hungary, the Netherlands, Spain, and the United Kingdom. A list of potential stakeholders drawn from the research team's contacts and snowballing served as the sampling frame. An email invitation was sent to all stakeholders in this list and recruitment was based on positive replies. Respondents were asked to rate availability and importance of 30 tobacco control interventions. Kappa coefficients assessed agreement of stakeholders' views. A mean importance score for each intervention was used to rank the interventions. This ranking was compared with the ranking based on cost-effectiveness data from a published review.

RESULTS

Ninety-three stakeholders (55.7% response rate) completed the survey: 18.3% were from Germany, 17.2% from Hungary, 30.1% from the Netherlands, 19.4% from Spain, and 15.1% from the UK. Of those, 31.2% were decision makers, 26.9% evidence generators, 19.4% service providers, 15.1% health-promotion advocates, and 7.5% purchasers of services/pharmaceutical products. Smoking restrictions in public areas were rated as the most important intervention (mean score = 1.89). The agreement on availability of interventions between the stakeholders was very low (kappa = 0.098; 95% CI = [0.085, 0.111] but the agreement on the importance of the interventions was fair (kappa = 0.239; 95% CI = [0.208, 0.253]). A correlation was found between availability and importance rankings for stage-based interventions. The importance ranking was not statistically concordant with the ranking based on published cost-effectiveness data (Kendall rank correlation coefficient = 0.40; p-value = 0.11; 95% CI = [- 0.09, 0.89]).

CONCLUSIONS

The intrinsic differences in stakeholder views must be addressed while transferring economic evidence Europe-wide. Strong engagement with stakeholders, focussing on better communication, has a potential to mitigate this challenge.

Serious gaming as a method for changing stakeholders’ perspectives on integrated care

Purpose

Serious gaming provides opportunities to harmonize the views of stakeholders regarding integrated care. In order to provide first insights on the effects and stakeholders’ satisfaction of serious gaming, the purpose of this paper is to explore what effects serious gaming has on the perceptions of different stakeholders regarding integrated care, and to evaluate a trial case of serious gaming on integrated care.

Design/methodology/approach

A pre- and post-test design was used, with two questionnaires. The first questionnaire focused on integrated care, based on the integrated change model, and was given to participants twice, once before and once after the serious game “Long Life Lab” was completed, to assess changes in perception. The second questionnaire focused on the evaluation of serious gaming, and was given to the participants only after the serious game.

Findings

With nine participants, the results yielded no statistical effects with the exception of three salient beliefs. Despite the small sample, differences in specific beliefs were found for knowledge, attitude and self-efficacy. Furthermore, the game was positively evaluated, but participants indicated that there is room for improvement.

Originality/value

Participants have positive beliefs toward the use of serious gaming as a tool for changing stakeholders’ perspectives on integrated care. Further studies in greater sample size are needed to confirm the potential value of serious gaming to improve integrated care.

MOST IMPORTANT BARRIERS AND FACILITATORS REGARDING THE USE OF HEALTH TECHNOLOGY ASSESSMENT

Objectives: Several studies have reported multiple barriers to and facilitators for the uptake of health technology assessment (HTA) information by policy makers. This study elicited, using best-worst scaling (BWS), the most important barriers and facilitators and their relative weight in the use of HTA by policy makers.

Methods: Two BWS object case surveys (one for barriers, one for facilitators) were conducted among sixteen policy makers and thirty-three HTA experts in the Netherlands. A list of twenty-two barriers and nineteen facilitators was included. In each choice task, participants were asked to choose the most important and the least important barrier/facilitator from a set of five. We used Hierarchical Bayes modeling to generate the mean relative importance score (RIS) for each factor and a subgroup analysis was conducted to assess differences between policy makers and HTA experts.

Results: The five most important barriers (RIS > 6.00) were “no explicit framework for decision-making process,” “insufficient support by stakeholders,” “lack of support,” “limited generalizability,” and “absence of appropriate incentives.” The six most important facilitators were: “availability of explicit framework for decision making,” “sufficient support by stakeholders,” “appropriate incentives,” “sufficient quality,” “sufficient awareness,” and “sufficient support within the organization.” Overall, perceptions did not differ markedly between policy makers and HTA experts.

Conclusions: Our study suggests that barriers and facilitators related to “policy characteristics” and “organization and resources” were particularly important. It is important to stimulate a pulse at the national level to create an explicit framework for including HTA in the decision-making context.

Model-based economic evaluations in smoking cessation and their transferability to new contexts: a systematic review

AIMS

To identify different types of models used in economic evaluations of smoking cessation, analyse the quality of the included models examining their attributes and ascertain their transferability to a new context.

METHODS

A systematic review of the literature on the economic evaluation of smoking cessation interventions published between 1996 and April 2015, identified via Medline, EMBASE, National Health Service (NHS) Economic Evaluation Database (NHS EED), Health Technology Assessment (HTA). The checklist-based quality of the included studies and transferability scores was based on the European Network of Health Economic Evaluation Databases (EURONHEED) criteria. Studies that were not in smoking cessation, not original research, not a model-based economic evaluation, that did not consider adult population and not from a high-income country were excluded.

FINDINGS

Among the 64 economic evaluations included in the review, the state-transition Markov model was the most frequently used method (n = 30/64), with quality adjusted life years (QALY) being the most frequently used outcome measure in a life-time horizon. A small number of the included studies (13 of 64) were eligible for EURONHEED transferability checklist. The overall transferability scores ranged from 0.50 to 0.97, with an average score of 0.75. The average score per section was 0.69 (range = 0.35-0.92). The relative transferability of the studies could not be established due to a limitation present in the EURONHEED method.

CONCLUSION

All existing economic evaluations in smoking cessation lack in one or more key study attributes necessary to be fully transferable to a new context.

Similarities and differences between stakeholders' opinions on using Health Technology Assessment (HTA) information across five European countries: results from the EQUIPT survey

Background

The European-study on Quantifying Utility of Investment in Protection from Tobacco (EQUIPT) project aimed to study transferability of economic evidence by co-creating the Tobacco Return On Investment (ROI) tool, previously developed in the United Kingdom, for four sample countries (Germany, Hungary, Spain and the Netherlands). The EQUIPT tool provides policymakers and stakeholders with customized information about the economic and wider returns on the investment in evidence-based tobacco control, including smoking cessation interventions. A Stakeholder Interview Survey was developed to engage with the stakeholders in early phases of the development and country adaptation of the ROI tool. The survey assessed stakeholders’ information needs, awareness about underlying principles used in economic analyses, opinion about the importance, effectiveness and cost-effectiveness of tobacco control interventions, and willingness to use a Health Technology Assessment (HTA) tool such as the ROI tool.

Methods

A cross sectional study using a mixed method approach was conducted among participating stakeholders in the sample countries and the United Kingdom. The individual questionnaire contained open-ended questions as well as single choice and 7- or 3-point Likert-scale questions. The results corresponding to the priority and needs assessment and to the awareness of stakeholders about underlying principles used in economic analysis are analysed by country and stakeholder categories.

Results

Stakeholders considered it important that the decisions on the investments in tobacco control interventions should be supported by scientific evidence, including prevalence of smoking, cost of smoking, quality of life, mortality due to smoking, and effectiveness, cost-effectiveness and budget impact of smoking cessation interventions. The proposed ROI tool was required to provide this granularity of information. The majority of the stakeholders were aware of the general principles of economic analyses used in decision making contexts but they did not appear to have in-depth knowledge about specific technical details. Generally, stakeholders’ answers showed larger variability by country than by stakeholder category.

Conclusions

Stakeholders across different European countries viewed the use of HTA evidence to be an important factor in their decision-making process. Further, they considered themselves to be capable of interpreting the results from a ROI tool and were highly motivated to use it.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-016-0110-7) contains supplementary material, which is available to authorized users.