From Mid-Level Policy Analysis to Macro-Level Political Economy Comment on "Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia"

The future of public health policymaking after COVID-19: a qualitative systematic review of lessons from Health in All Policies

Background: 'Health in All Policies' (HiAP) describes the pursuit of health equity. It has five main elements: treat health as a human right; identify evidence of the 'social determinants' of health inequalities, recognise that most powers to affect health are not held by health departments, promote intersectoral policymaking and collaboration inside and outside of government, and generate political will. Studies describe its potential but bemoan a major implementation gap. Some HiAP scholars learn from policymaking research how to understand this gap, but the use of policy theories is patchy. In that context, our guiding research question is: How does HiAP research use policy theory to understand policymaking? It allows us to zoom-out to survey the field and zoom-in to identify: the assumed and actual causes of policy change, and transferable lessons to HiAP scholars and advocates. Methods: Our qualitative systematic review (two phases, 2018 and 2020) identified 4972 HiAP articles. Of these, 113 journal articles (research and commentary) provide a non-trivial reference to policymaking (at least one reference to a policymaking concept). We use the 113 articles to produce a general HiAP narrative and explore how the relatively theory-informed articles enhance it. Results: Most articles focus on policy analysis (identifying policy problems and solutions) rather than policy theory (explaining policymaking dynamics). They report a disappointing gap between HiAP expectations and policy outcomes. Theory-informed articles contribute to a HiAP playbook to close that gap or a programme theory to design and evaluate HiAP in new ways.   Conclusions: Few HiAP articles use policy theories for their intended purpose. Policy theories provide lessons to aid critical reflection on power, political dilemmas, and policymaking context. HiAP scholars seek more instrumental lessons, potentially at the cost of effective advocacy and research.

The future of public health policymaking after COVID-19: a qualitative systematic review of lessons from Health in All Policies

Background: 'Health in All Policies' (HiAP) describes the pursuit of health equity. It has five main elements: treat health as a human right; identify evidence of the 'social determinants' of health inequalities, recognise that most powers to affect health are not held by health departments, promote intersectoral policymaking and collaboration inside and outside of government, and generate political will. Studies describe its potential but bemoan a major implementation gap. Some HiAP scholars learn from policymaking research how to understand this gap, but the use of policy theories is patchy. In that context, our guiding research question is: How does HiAP research use policy theory to understand policymaking? It allows us to zoom-out to survey the field and zoom-in to identify: the assumed and actual causes of policy change, and transferable lessons to HiAP scholars and advocates. Methods: Our qualitative systematic review (two phases, 2018 and 2020) identified 4972 HiAP articles. Of these, 113 journal articles (research and commentary) provide a non-trivial reference to policymaking (at least one reference to a policymaking concept). We use the 113 articles to produce a general HiAP narrative and explore how the relatively theory-informed articles enhance it. Results: Most articles focus on policy analysis (identifying policy problems and solutions) rather than policy theory (explaining policymaking dynamics). They report a disappointing gap between HiAP expectations and policy outcomes. Theory-informed articles contribute to a HiAP playbook to close that gap or a programme theory to design and evaluate HiAP in new ways.   Conclusions: Few HiAP articles use policy theories for their intended purpose. Policy theories provide lessons to aid critical reflection on power, political dilemmas, and policymaking context. HiAP scholars seek more instrumental lessons, potentially at the cost of effective advocacy and research.

To what extent can the activities of the South Australian Health in All Policies initiative be linked to population health outcomes using a program theory-based evaluation?

Background

This paper reports on a five-year study using a theory-based program logic evaluation, and supporting survey and interview data to examine the extent to which the activites of the South Australian Health in All Policies initiative can be linked to population health outcomes.

Methods

Mixed-methods data were collected between 2012 and 2016 in South Australia (144 semi-structured key informant interviews; two electronic surveys of public servants in 2013 (n = 435) and 2015 (n = 483); analysis of state government policy documents; and construction of a program logic model to shape assessment of the feasibility of attribution to population health outcomes).

Results

Multiple actions on social determinants of health in a range of state government sectors were reported and most could be linked through a program logic model to making some contribution to future population health outcomes. Context strongly influences implementation; not all initiatives will be successful and experimentation is vital. Successful initiatives included HiAP influencing the urban planning department to be more concerned with the health impacts of planning decisions, and encouraging the environment department to be concerned with the health impacts of its work.

Conclusions

The theory-based program logic suggests that SA HiAP facilitated improved population health through working with multiple government departments. Public servants came to appreciate how their sectors impact on health. Program logic is a mechanism to evaluate complex public health interventions in a way that takes account of political and economic contexts. SA HiAP was mainly successful in avoiding lifestyle drift in strategy. The initiative encouraged a range of state government departments to tackle conditions of daily living. The broader underpinning factors dictating the distribution of power, money and resources were not addressed by HiAP. This reflects HiAP’s use of a consensus model which was driven by (rather than drove) state priorities and sought ‘win-win’ strategies.

Researching Healthy Public Policy: Navigating the 'Black Box' Means Thinking More About Power Comment on "Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia"

Lawless et al provide a valuable narrative of using program logic to develop an evaluation of Health in All Policies (HiAP) in South Australia. In this commentary I argue that the paper and analysis is an extremely useful example of navigating the supposed black box of policy-making. However the original makes the reader work too hard and is distracting from the main narrative of explaining the logic behind the HiAP approach in South Australia. My response covers avoiding epistemological traps and weighing up the pragmatics of collaborative policy research with more complex institutional policy issues like power.

Policy, Theory, and Evaluation: Stop Mixing the Fruit Salad Comment on "Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia"

The study of Health in All Policies (HiAP) is gaining momentum. Authors are increasingly turning to wide swathes of political and social theory to frame (Program) Theory Based (or Informed) Evaluation (TBE) approaches. TBE for HiAP is not only prudent, it adds a level of elegance and insight to the research toolbox. However, it is still necessary to organize theoretical thinking appropriately. A commentary on a recent Int J Health Policy Manag paper argued that the framing of context and causality were hard to establish. This paper argues that this is not the most pressing issue. Rather, it claims we need to go back to basics to establish an appropriate HiAP evaluation paradigm. Such a basic paradigm would hinge on an understanding of power.