Generating demand for and use of evaluation evidence in government health ministries: lessons from a pilot programme in Uganda and Zambia

Policy implementation and outcome evaluation: establishing a framework and expanding capacity for advocacy organizations to assess the impact of their work in public policy

Advocacy organizations can play a crucial role in evaluating whether legislation or regulation has had its intended effect by supporting robust public policy implementation and outcome evaluation. The American Heart Association, working with expert advisors, has developed a framework for effective evaluation that can be used by advocacy organizations, in partnership with researchers, public health agencies, funders, and policy makers to assess the health and equity impact of legislation and regulation over time. Advocacy organizations can use parts of this framework to evaluate the impact of policies relevant to their own advocacy and public policy efforts and inform policy development and guide their organizational resource allocation. Ultimately, working in partnership, advocacy organizations can help bring capacity, commitment and funding to this important implementation and outcome evaluation work that informs impactful public policy for equitable population health and well-being.

Closing the Virtual Gap in Health Care: A Series of Case Studies Illustrating the Impact of Embedding Evaluation Alongside System Initiatives

Early decisions relating to the implementation of virtual care relied on necessity and clinical judgement, but there is a growing need for the generation of evidence to inform policy and practice designs. The need for stronger partnerships between researchers and decision-makers is well recognized, but how these partnerships can be structured and how research can be embedded alongside existing virtual care initiatives remain unclear. We present a series of case studies that illustrate how embedded research can inform policy decisions related to the implementation of virtual care, where decisions are either to (1) discontinue (red light), (2) redesign (yellow light), or (3) scale up existing initiatives (green light). Data were collected through document review and informal interviews with key study personnel. Case 1 involved an evaluation of a mobile diabetes platform that demonstrated a mismatch between the setting and the technology (decision outcome: discontinue). Case 2 involved an evaluation of a mental health support platform that suggested evidence-based modifications to the delivery model (decision outcome: redesign). Case 3 involved an evaluation of video visits that generated evidence to inform the ideal model of implementation at scale (decision outcome: scale up). In this paper, we highlight the characteristics of the partnership and the process that enabled success and use the cases to illustrate how these characteristics were operationalized. Structured communication included monthly check-ins and iterative report development. We also outline key characteristics of the partnership (ie, trust and shared purpose) and the process (ie, timeliness, tailored reporting, and adaptability) that drove the uptake of evidence in decision-making. Across each case, the evaluation was designed to address policy questions articulated by our partners. Furthermore, structured communication provided opportunities for knowledge mobilization. Structured communication was operationalized through monthly meetings as well as the delivery of interim and final reports. These case studies demonstrate the importance of partnering with health system decision-makers to generate and mobilize scientific evidence. Embedded research partnerships founded on a shared purpose of system service provided an effective strategy to bridge the oft-cited gap between science and policy. Structured communication provided a mechanism for collaborative problem-solving and real-time feedback, and it helped contextualize emerging insights.

What, why and how do health systems learn from one another? Insights from eight low- and middle-income country case studies

BACKGROUND

All health systems struggle to meet health needs within constrained resources. This is especially true for low-income countries. It is critical that they can learn from wider contexts in order to improve their performance. This article examines policy transfer and evidence use linked to it in low- and middle-income settings. The objective was to inform international investments in improved learning across health systems.

METHODS

The article uses a comparative case study design, drawing on case studies conducted in Bangladesh, Burkina Faso, Cambodia, Ethiopia, Georgia, Nepal, Rwanda and Solomon Islands. One or two recent health system reforms were selected in each case and 148 key informants were interviewed in total, using a semi-structured tool focused on different stages of the policy cycle. Interviewees were selected for their engagement in the policy process and represented political, technical, development partner, non-governmental, academic and civil society constituencies. Data analysis used a framework approach, allowing for new themes to be developed inductively, focusing initially on each case and then on patterns across cases.

RESULTS

The selected policies demonstrated a range of influences of externally imposed, co-produced and home-grown solutions on the development of initial policy ideas. Eventual uptake of policy was strongly driven in most settings by local political economic considerations. Policy development post-adoption demonstrated some strong internal review, monitoring and sharing processes but there is a more contested view of the role of evaluation. In many cases, learning was facilitated by direct personal relationships with local development partner staff. While barriers and facilitators to evidence use included supply and demand factors, the most influential facilitators were incentives and capacity to use evidence.

CONCLUSIONS

These findings emphasise the agency of local actors and the importance of developing national and sub-national institutions for gathering, filtering and sharing evidence. Developing demand for and capacity to use evidence appears more important than augmenting supply of evidence, although specific gaps in supply were identified. The findings also highlight the importance of the local political economy in setting parameters within which evidence is considered and the need for a conceptual framework for health system learning.