Implementation and evaluation of local-level priority setting for stroke

Evaluating health service outcomes of public involvement in health service design in high-income countries: a systematic review

BACKGROUND

Internationally, it is expected that health services will involve the public in health service design. Evaluation of public involvement has typically focused on the process and experiences for participants. Less is known about outcomes for health services. The aim of this systematic review was to a) identify and synthesise what is known about health service outcomes of public involvement and b) document how outcomes were evaluated.

METHODS

Searches were undertaken in MEDLINE, EMBASE, The Cochrane Library, PsycINFO, Web of Science, and CINAHL for studies that reported health service outcomes from public involvement in health service design. The review was limited to high-income countries and studies in English. Study quality was assessed using the Mixed Methods Appraisal Tool and critical appraisal guidelines for assessing the quality and impact of user involvement in health research. Content analysis was used to determine the outcomes of public involvement in health service design and how outcomes were evaluated.

RESULTS

A total of 93 articles were included. The majority were published in the last 5 years, were qualitative, and were located in the United Kingdom. A range of health service outcomes (discrete products, improvements to health services and system/policy level changes) were reported at various levels (service level, across services, and across organisations). However, evaluations of outcomes were reported in less than half of studies. In studies where outcomes were evaluated, a range of methods were used; most frequent were mixed methods. The quality of study design and reporting was inconsistent.

CONCLUSION

When reporting public involvement in health service design authors outline a range of outcomes for health services, but it is challenging to determine the extent of outcomes due to inadequate descriptions of study design and poor reporting. There is an urgent need for evaluations, including longitudinal study designs and cost-benefit analyses, to fully understand outcomes from public involvement in health service design.

Quantifying the impact and relevance of scientific research

Qualitative and quantitative methods are being developed to measure the impacts of research on society, but they suffer from serious drawbacks associated with linking a piece of research to its subsequent impacts. We have developed a method to derive impact scores for individual research publications according to their contribution to answering questions of quantified importance to end users of research. To demonstrate the approach, here we evaluate the impacts of research into means of conserving wild bee populations in the UK. For published papers, there is a weak positive correlation between our impact score and the impact factor of the journal. The process identifies publications that provide high quality evidence relating to issues of strong concern. It can also be used to set future research agendas.

Priority setting: what constitutes success? A conceptual framework for successful priority setting

Background

The sustainability of healthcare systems worldwide is threatened by a growing demand for services and expensive innovative technologies. Decision makers struggle in this environment to set priorities appropriately, particularly because they lack consensus about which values should guide their decisions. One way to approach this problem is to determine what all relevant stakeholders understand successful priority setting to mean. The goal of this research was to develop a conceptual framework for successful priority setting.

Methods

Three separate empirical studies were completed using qualitative data collection methods (one-on-one interviews with healthcare decision makers from across Canada; focus groups with representation of patients, caregivers and policy makers; and Delphi study including scholars and decision makers from five countries).

Results

This paper synthesizes the findings from three studies into a framework of ten separate but interconnected elements germane to successful priority setting: stakeholder understanding, shifted priorities/reallocation of resources, decision making quality, stakeholder acceptance and satisfaction, positive externalities, stakeholder engagement, use of explicit process, information management, consideration of values and context, and revision or appeals mechanism.

Conclusion

The ten elements specify both quantitative and qualitative dimensions of priority setting and relate to both process and outcome components. To our knowledge, this is the first framework that describes successful priority setting. The ten elements identified in this research provide guidance for decision makers and a common language to discuss priority setting success and work toward improving priority setting efforts.

Developing a prioritisation framework in an English Primary Care Trust

Background

In the English NHS, Primary Care Trusts (PCTs) are required to commission health services, to maximise the well-being of the population, subject to the available budget. There are numerous techniques employed to make decisions, some more rational and transparent than others. A weighted benefit score can be used to rank options but this does not take into account value for money from investments.

Methods

We developed a weighted benefit score framework for use in an English PCT which ranked options in order of 'cost-value' or 'cost per point of benefit'. Our method differs from existing techniques by explicitly combining cost and a composite weighted benefit score into the cost-value ratio.

Results

The technique proved readily workable, and was able to accommodate a wide variety of data and competing criteria. Participants felt able to assign scores to proposed services, and generate a ranked list, which provides a solid starting point for the PCT Board to discuss and make funding decisions. Limitations included potential for criteria to be neither exhaustive nor mutually exclusive and the lack of an interval property in the benefit score limiting the usefulness of a cost-value ratio.

Conclusion

A technical approach to decision making is insufficient for making prioritisation decisions, however our technique provides a very valuable, structured and informed starting point for PCT decision making.