Methods for public health economic evaluation: A Delphi survey of decision makers in English and Welsh local government

A framework to integrate equity in public health emergency response dashboards: Dashboard instrument to review equity (DIRE)

OBJECTIVES

COVID-19 created an urgent element of clinical and financial strain to the public health system, forcing it into rapid response mode. Key public health decisions were quickly made, with limited data and guidance to address decision trade-offs and community inequities. Gaps identified in the pandemic confirmed the need for a new tool, like this study's Dashboard Instrument to Review Equity (DIRE) Framework, to ensure decision-makers are given quick and equitable decision-making guidance.

STUDY DESIGN

Scoping review and tool development.

METHODS

The scoping review was conducted through PRISMA-ScR, and by utilizing tools like PubMed, Scopus, and Paper Piles to compile and cite. Three levels of thematic analysis were completed. Tool development consisted of building a conceptual model on the DIKW Pyramid and Informatics Stack. Then the review's five themes were integrated into DIRE.

RESULTS

The review closed at a final count of 102 articles, with five themes emerging: COVID-19 impact, Health Equity, Decisions During Emergencies, Dashboards and Decision Support, and Frameworks. COVID-19 dashboards were also reviewed. DIRE was designed into three layers (context, data flow, and dashboard users) and three data flow buckets (data sources, DIK pillars, and interventions).

CONCLUSIONS

This study aimed to (1) Establish a research foundation of health equity, COVID-19 lessons learned, and decision support, and (2) Develop an evidence-based framework. Though further research is still recommended, DIRE is now the first 3-point framework aimed at preparing decision-makers to respond quickly and equitably to future emergencies.

Expert Opinions on an Optimal Infant Feeding Quantitative Data Framework: A Mixed Methods Delphi-Style Study in the UK

BACKGROUND

This study aimed to explore how expert stakeholders involved in research, policy and practice would define an ideal dataset for collecting infant feeding data, to better align efforts to monitor and evaluate breastfeeding across the UK four nations.

METHODOLOGY

Using the Delphi method, two phases of consultation were completed with a total of 42 stakeholders. First, qualitative (Round 1) and quantitative (Round 2) surveys were distributed to an interdisciplinary panel of experts, to identify individual-level agreement for key terms and timepoints for data collection relevant to infant age. Second, policy-led stakeholders discussed outcomes from Phase 1, before contributing to a written consultation response for their nation to indicate group-level agreement.

RESULTS

Across Phase 1 surveys, 13 of 15 indicators reached consensus for definitions, and 11 of 13 reached consensus for timepoints. During Phase 2, 5 of 7 indicators reached a level of final agreement. Data collection was suggested to focus on the intention to breastfeed (around birth), and the early initiation of feeding (from birth to 10 days). Monitoring of 'exclusive' breastfeeding from 0 to 6 months, 'any' breastfeeding from 0 to 24 months, and complementary feeding at 6 and 12 months, were identified as key touchpoints.

PRINCIPAL CONCLUSIONS

To support the feasibility of data reforms across UK nations, these results identify consensus for a shortlist of shared data indicators (see Supplementary Figure 1), highlighting opportunities for data collection that close the gap with international standards, and align with existing monitoring frameworks and healthcare practice.

Is Economic Evaluation and Care Commissioning Focused on Achieving the Same Outcomes? Resource-Allocation Considerations and Challenges Using England as a Case Study

Commissioning describes the process of contracting appropriate care services to address pre-identified needs through pre-agreed payment structures. Outcomes-based commissioning (i.e., paying services for pre-agreed outcomes) shares a common goal with economic evaluation: achieving value for money for relevant outcomes (e.g., health) achieved from a finite budget. We describe considerations and challenges as to the practical role of relevant outcomes for evaluation and commissioning, seeking to bridge a gap between economic evaluation evidence and care commissioning. We describe conceptual (e.g., what are 'relevant' outcomes) alongside practical considerations (e.g., quantifying and using relevant endpoint or surrogate outcomes) and pertinent issues when linking outcomes to commissioning-based payment mechanisms, using England as a case study. Economic evaluation often focuses on a single endpoint health-focused maximand, e.g., quality-adjusted life-years (QALYs), whereas commissioning often focuses on activity-based surrogate outcomes (e.g., health monitoring), as easier-to-measure key performance indicators that are more acceptable (e.g., by clinicians) and amenable to being linked with payment structures. However, payments linked to endpoint and/or surrogate outcomes can lead to market inefficiencies; for example, when surrogates do not have the intended causal effect on endpoint outcomes or when service activity focuses on only people who can achieve prespecified payment-linked outcomes. Accounting for and explaining direct links from commissioners' payment structures to surrogate and then endpoint economic outcomes is a vital step to bridging a gap between economic evaluation approaches and commissioning. Decision-analytic models could aid this but they must be designed to account for relevant surrogate and endpoint outcomes, the payments assigned to such outcomes, and their interaction with the system commissioners purport to influence.

Family-Centered Early Intervention Deaf/Hard of Hearing (FCEI-DHH): Methods

This is the fourth article in a series of eight that comprise a special issue on family-centered early intervention (FCEI) for children who are deaf or hard of hearing (DHH) and their families, FCEI-DHH. This article describes the co-production team and the consensus review method used to direct the creation of the 10 Principles described in this special issue. Co-production is increasingly being used to produce evidence that is useful, usable, and used. A draft set of 10 Principles for FCEI-DHH and associated Tables of recommended behaviors were developed using the knowledge creation process. Principles were refined through two rounds of eDelphi review. Results for each round were analyzed using measures of overall group agreement and measures that indicated the extent to which the group members agreed with each other. After Round 2, with strong agreement and low to moderate variation in extent of agreement, consensus was obtained for the 10 Principles for FCEI-DHH presented in this special issue. This work can be used to enhance evolution of FCEI-DHH program/services and systems world-wide and adds to knowledge in improvement science.

Economic Evaluation Evidence for Resource-Allocation Decision Making: Bridging the Gap for Local Decision Makers Using English Case Studies

Best-practice economic evaluation methods for health-related decision making at a national level in England are well established, and as a first principle generally involve attempting to maximise the amount of health generated from the health system's budget. Such methods are applied in ways that are broadly transparent and accountable, often at arm's length from explicit political pressures. At local levels of decision making, however, decision making is arguably less likely to be applied according to established overarching principles, is less transparent and is more subject to political pressures. This may be owing to a multiplicity of reasons, for example, undesirability/inappropriateness of such methods, or a failure to make the methods clear to local decision makers. We outline principles for economic evaluations and break down these methods into their component parts, considering their relevance in the English local context. These include taxonomies of decision-making frameworks, budgets, costs, outcome, and characterisations of cost effectiveness. We also explore the role of broader factors, including the relevance of assuming a single fixed budget, pressures resulting from political and budgetary cycles and affordability. We consider the data requirements to inform such deliberations. By setting out principles for economic evaluation methods in a clear language aimed at local decision making, a potential role for such methods can be established, which to date has failed to emerge. While the extent to which these methods can and should be applied are a matter for continued debate, the establishment of such a mutual understanding may assist in the improvement of methods for such decision making and the outcomes resulting from their application.

Methods for the economic evaluation of obesity prevention dietary interventions in children: A systematic review and critical appraisal of the evidence

OBJECTIVES

We aim to describe and provide a discussion of methods used to conduct economic evaluations of dietary interventions in children and adolescents, including long-term modelling, and to make recommendations to assist health economists in the design and reporting of such evaluations.

METHODS

A systematic review was conducted in 11 bibliographic databases and the grey literature with searches undertaken between January 2000 and December 2021. A study was included if it (1) was an economic evaluation or modelling study of an obesity-prevention dietary intervention and (2) targeted 2- to 18-year-olds.

RESULTS

Twenty-six studies met the inclusion criteria. Twelve studies conducted an economic evaluation alongside a clinical trial, and 14 studies modelled long-term health and cost outcomes. Four overarching methodological challenges were identified: modelling long-term impact of interventions, measuring and valuing health outcomes, cost inclusions and equity considerations.

CONCLUSIONS

Variability in methods used to predict, measure and value long-term benefits in adulthood from short-term clinical outcomes in childhood was evident across studies. Key recommendations to improve the design and analysis of future economic evaluations include the consideration of weight regain and diminishing intervention effects within future projections; exploration of wider intervention benefits not restricted to quality-of-life outcomes; and inclusion of parental or caregiver opportunity costs.

Program Evaluation of Population- and System-Level Policies: Evidence for Decision Making

BACKGROUND

Policy evaluations often focus on ex post estimation of causal effects on short-term surrogate outcomes. The value of such information is limited for decision making, as the failure to reflect policy-relevant outcomes and disregard for opportunity costs prohibits the assessment of value for money. Further, these evaluations do not always consider all relevant evidence, other courses of action, or decision uncertainty.

METHODS

In this article, we explore how policy evaluation could better meet the needs of decision making. We begin by defining the evidence required to inform decision making. We then conduct a literature review of challenges in evaluating policies. Finally, we highlight potential methods available to help address these challenges.

RESULTS

The evidence required to inform decision making includes the impacts on the policy-relevant outcomes, the costs and associated opportunity costs, and the consequences of uncertainty. Challenges in evaluating health policies are described using 8 categories: 1) valuation space; 2) comparators; 3) time of evaluation; 4) mechanisms of action; 5) effects; 6) resources, constraints, and opportunity costs; 7) fidelity, adaptation, and level of implementation; and 8) generalizability and external validity. Methods from a broad set of disciplines are available to improve policy evaluation, relating to causal inference, decision-analytic modeling, theory of change, realist evaluation, and structured expert elicitation.

LIMITATIONS

The targeted review may not identify all possible challenges, and the methods covered are not exhaustive.

CONCLUSIONS

Evaluations should provide appropriate evidence to inform decision making. There are challenges in evaluating policies, but methods from multiple disciplines are available to address these challenges.

IMPLICATIONS

Evaluators need to carefully consider the decision being informed, the necessary evidence to inform it, and the appropriate methods.[Box: see text].

Modelling tool to support decision-making in the NHS Health Check programme: workshops, systematic review and co-production with users

BACKGROUND

Local authorities in England commission the NHS Health Check programme to invite everyone aged 40-74 years without pre-existing conditions for risk assessment and eventual intervention, if needed. However, the programme's effectiveness, cost-effectiveness and equity impact remain uncertain.

AIM

To develop a validated open-access flexible web-based model that enables local commissioners to quantify the cost-effectiveness and potential for equitable population health gain of the NHS Health Check programme.

OBJECTIVES

The objectives were as follows: (1) co-produce with stakeholders the desirable features of the user-friendly model; (2) update the evidence base to support model and scenario development; (3) further develop our computational model to allow for developments and changes to the NHS Health Check programme and the diseases it addresses; (4) assess the effectiveness, cost-effectiveness and equity of alternative strategies for implementation to illustrate the use of the tool; and (5) propose a sustainability and implementation plan to deploy our user-friendly computational model at the local level.

DESIGN

Co-production workshops surveying the best-performing local authorities and a systematic literature review of strategies to increase uptake of screening programmes informed model use and development. We then co-produced the workHORSE (working Health Outcomes Research Simulation Environment) model to estimate the health, economic and equity impact of different NHS Health Check programme implementations, using illustrative-use cases.

SETTING

Local authorities in England.

PARTICIPANTS

Stakeholders from local authorities, Public Health England, the NHS, the British Heart Foundation, academia and other organisations participated in the workshops. For the local authorities survey, we invited 16 of the best-performing local authorities in England.

INTERVENTIONS

The user interface allows users to vary key parameters that represent programme activities (i.e. invitation, uptake, prescriptions and referrals). Scenarios can be compared with each other.

MAIN OUTCOME MEASURES

Disease cases and case-years prevented or postponed, incremental cost-effectiveness ratios, net monetary benefit and change in slope index of inequality.

RESULTS

The survey of best-performing local authorities revealed a diversity of effective approaches to maximise the coverage and uptake of NHS Health Check programme, with no distinct 'best buy'. The umbrella literature review identified a range of effective single interventions. However, these generally need to be combined to maximally improve uptake and health gains. A validated dynamic, stochastic microsimulation model, built on robust epidemiology, enabled service options analysis. Analyses of three contrasting illustrative cases estimated the health, economic and equity impact of optimising the Health Checks, and the added value of obtaining detailed local data. Optimising the programme in Liverpool can become cost-effective and equitable, but simply changing the invitation method will require other programme changes to improve its performance. Detailed data inputs can benefit local analysis.

LIMITATIONS

Although the approach is extremely flexible, it is complex and requires substantial amounts of data, alongside expertise to both maintain and run.

CONCLUSIONS

Our project showed that the workHORSE model could be used to estimate the health, economic and equity impact comprehensively at local authority level. It has the potential for further development as a commissioning tool and to stimulate broader discussions on the role of these tools in real-world decision-making.

FUTURE WORK

Future work should focus on improving user interactions with the model, modelling simulation standards, and adapting workHORSE for evaluation, design and implementation support.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42019132087.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 35. See the NIHR Journals Library website for further project information.

Cost-effectiveness evidence of mental health prevention and promotion interventions: A systematic review of economic evaluations

BACKGROUND

The prevention of mental disorders and promotion of mental health and well-being are growing fields. Whether mental health promotion and prevention interventions provide value for money in children, adolescents, adults, and older adults is unclear. The aim of the current study is to update 2 existing reviews of cost-effectiveness studies in this field in order to determine whether such interventions are cost-effective.

METHODS AND FINDINGS

Electronic databases (including MEDLINE, PsycINFO, CINAHL, and EconLit through EBSCO and Embase) were searched for published cost-effectiveness studies of prevention of mental disorders and promotion of mental health and well-being from 2008 to 2020. The quality of studies was assessed using the Quality of Health Economic Studies Instrument (QHES). The protocol was registered with PROSPERO (# CRD42019127778). The primary outcomes were incremental cost-effectiveness ratio (ICER) or return on investment (ROI) ratio across all studies. A total of 65 studies met the inclusion criteria of a full economic evaluation, of which, 23 targeted children and adolescents, 35 targeted adults, while the remaining targeted older adults. A large number of studies focused on prevention of depression and/or anxiety disorders, followed by promotion of mental health and well-being and other mental disorders. Although there was high heterogeneity in terms of the design among included economic evaluations, most studies consistently found that interventions for mental health prevention and promotion were cost-effective or cost saving. The review found that targeted prevention was likely to be cost-effective compared to universal prevention. Screening plus psychological interventions (e.g., cognitive behavioural therapy [CBT]) at school were the most cost-effective interventions for prevention of mental disorders in children and adolescents, while parenting interventions and workplace interventions had good evidence in mental health promotion. There is inconclusive evidence for preventive interventions for mental disorders or mental health promotion in older adults. While studies were of general high quality, there was limited evidence available from low- and middle-income countries. The review was limited to studies where mental health was the primary outcome and may have missed general health promoting strategies that could also prevent mental disorder or promote mental health. Some ROI studies might not be included given that these studies are commonly published in grey literature rather than in the academic literature.

CONCLUSIONS

Our review found a significant growth of economic evaluations in prevention of mental disorders or promotion of mental health and well-being over the last 10 years. Although several interventions for mental health prevention and promotion provide good value for money, the varied quality as well as methodologies used in economic evaluations limit the generalisability of conclusions about cost-effectiveness. However, the finding that the majority of studies especially in children, adolescents, and adults demonstrated good value for money is promising. Research on cost-effectiveness in low-middle income settings is required.

TRIAL REGISTRATION

PROSPERO registration number: CRD42019127778.

Cost comparison of five Australasian obesity prevention interventions for children aged from birth to two years

BACKGROUND

In the absence of rigorous evidence of cost-effectiveness for early childhood obesity prevention interventions, the next-best option may be for decision-makers to consider the relevant costs of interventions when allocating resources.

OBJECTIVES

This study aimed to estimate systematically the cost of five obesity prevention interventions in children aged 0-2 years, undertaken in research settings in Australia and New Zealand.

METHODS

A standardised costing protocol informed the costing methodology, ensuring comparability of results across interventions. Micro-costing was undertaken, with intervention costs defined from the funder perspective and valued in 2018 Australian dollars using unit costs from the trials or market rates.

RESULTS

Interventions varied widely in their resource use. The total cost per participant ranged from $80 for the CHAT SMS intervention arm (95% UI $77-$82) to $1135 for the Healthy Beginnings intervention (95% UI $1059-$1189). Time costs of personnel delivering interventions contributed >50% of total intervention costs for all included studies.

CONCLUSIONS

An understanding of the costs associated with intervention delivery modes is important, alongside effectiveness. Telephone delivery may include unexpected costs associated with connection to intervention participants at convenient times. A SMS-based intervention had the lowest delivery cost in this study.