How Do We Evaluate Health in All Policies? Comment on "Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia"

Evaluating ‘Health in All Policies’ in Norwegian Municipalities

The Health in All Policies (HiAP) approach has emerged as a central strategy for promoting health at local, regional, and state levels in different countries. Representing a complex and complicated strategy, evaluations of HIAP at the local level are scarce, and scholars call for more knowledge and critical discussions regarding how to evaluate at this level. In this conceptual paper, I discuss how summative and formative evaluation approaches might supplement each other when evaluating HiAP at the local level. First, I discuss the possibilities of using summative and formative evaluation of HiAP. Further, I discuss how formative-dialogue research might represent possibilities for evaluation by combining the two approaches. Finally, I ask if there has been a shift in the Norwegian evaluation discourse, from the promotion of summative evaluation alone to a combination of both summative and formative methods.

The future of public health policymaking after COVID-19: a qualitative systematic review of lessons from Health in All Policies

Background: 'Health in All Policies' (HiAP) describes the pursuit of health equity. It has five main elements: treat health as a human right; identify evidence of the 'social determinants' of health inequalities, recognise that most powers to affect health are not held by health departments, promote intersectoral policymaking and collaboration inside and outside of government, and generate political will. Studies describe its potential but bemoan a major implementation gap. Some HiAP scholars learn from policymaking research how to understand this gap, but the use of policy theories is patchy. In that context, our guiding research question is: How does HiAP research use policy theory to understand policymaking? It allows us to zoom-out to survey the field and zoom-in to identify: the assumed and actual causes of policy change, and transferable lessons to HiAP scholars and advocates. Methods: Our qualitative systematic review (two phases, 2018 and 2020) identified 4972 HiAP articles. Of these, 113 journal articles (research and commentary) provide a non-trivial reference to policymaking (at least one reference to a policymaking concept). We use the 113 articles to produce a general HiAP narrative and explore how the relatively theory-informed articles enhance it. Results: Most articles focus on policy analysis (identifying policy problems and solutions) rather than policy theory (explaining policymaking dynamics). They report a disappointing gap between HiAP expectations and policy outcomes. Theory-informed articles contribute to a HiAP playbook to close that gap or a programme theory to design and evaluate HiAP in new ways.   Conclusions: Few HiAP articles use policy theories for their intended purpose. Policy theories provide lessons to aid critical reflection on power, political dilemmas, and policymaking context. HiAP scholars seek more instrumental lessons, potentially at the cost of effective advocacy and research.

The future of public health policymaking after COVID-19: a qualitative systematic review of lessons from Health in All Policies

Background: 'Health in All Policies' (HiAP) describes the pursuit of health equity. It has five main elements: treat health as a human right; identify evidence of the 'social determinants' of health inequalities, recognise that most powers to affect health are not held by health departments, promote intersectoral policymaking and collaboration inside and outside of government, and generate political will. Studies describe its potential but bemoan a major implementation gap. Some HiAP scholars learn from policymaking research how to understand this gap, but the use of policy theories is patchy. In that context, our guiding research question is: How does HiAP research use policy theory to understand policymaking? It allows us to zoom-out to survey the field and zoom-in to identify: the assumed and actual causes of policy change, and transferable lessons to HiAP scholars and advocates. Methods: Our qualitative systematic review (two phases, 2018 and 2020) identified 4972 HiAP articles. Of these, 113 journal articles (research and commentary) provide a non-trivial reference to policymaking (at least one reference to a policymaking concept). We use the 113 articles to produce a general HiAP narrative and explore how the relatively theory-informed articles enhance it. Results: Most articles focus on policy analysis (identifying policy problems and solutions) rather than policy theory (explaining policymaking dynamics). They report a disappointing gap between HiAP expectations and policy outcomes. Theory-informed articles contribute to a HiAP playbook to close that gap or a programme theory to design and evaluate HiAP in new ways.   Conclusions: Few HiAP articles use policy theories for their intended purpose. Policy theories provide lessons to aid critical reflection on power, political dilemmas, and policymaking context. HiAP scholars seek more instrumental lessons, potentially at the cost of effective advocacy and research.

Using the Project INTEGRATE Framework in Practice in Central Coast, Australia

Introduction:

Integrated care implies sustained change in complex systems and progress is not always linear or easy to assess. The Central Coast integrated Care Program (CCICP) was planned as a ten-year place-based system change. This paper reports the first formative evaluation to provide a detailed description of the implementation of the CCICP, after two years of activity, and the current progress towards integrated care.

Theory and Methods:

Progress towards integrated care achieved by the CCICP was evaluated using the Project INTEGRATE Framework data in a mixed methods approach included semi-structured interviews (n = 23) and Project INTEGRATE Framework based surveys (n = 27). All data collected involved key stakeholders, with close involvement in the program, self-reporting.

Results:

Progress has been mixed. Gains had most clearly been made in the areas of clinical and professional integration; specifically, relationship building and improved collaboration and cooperation between service providers. The areas of systemic and functional integration were least improved with funding uncertainty being an ongoing significant problem. The evaluation also showed that the Project INTEGRATE framework provided a consistent language for CCICP partners and for evaluators and consistent indicators of progress. The framework also helped to identify key facilitators and barriers.

Discussion and Conclusion:

The findings highlight the willingness and commitment of key staff but also the importance of leadership, good communication, relationship building, and cultural transformation.