Applying complexity theory: a review to inform evaluation design

Understanding the system dynamics of obesity-related behaviours in 10- to 14-year-old adolescents in Amsterdam from a multi-actor perspective

Introduction and Methods

To develop an understanding of the dynamics driving obesity-related behaviours in adolescents, we conducted systems-based analysis on a causal loop diagram (CLD) created from a multi-actor perspective, including academic researchers, adolescents and local stakeholders.

Results

The CLD contained 121 factors and 31 feedback loops. We identified six subsystems with their goals: (1) interaction between adolescents and the food environment, with profit maximisation as goal, (2) interaction between adolescents and the physical activity environment, with utility maximisation of outdoor spaces as goal, (3) interaction between adolescents and the online environment, with profit maximisation from technology use as goal, (4) interaction between adolescents, parenting and the wider socioeconomic environment, with a goal focused on individual parental responsibility, (5) interaction between healthcare professionals and families, with the goal resulting in treating obesity as an isolated problem, and (6) transition from childhood to adolescence, with the goal centring around adolescents' susceptibility to an environment that stimulates obesity-related behaviours.

Discussion

Analysis showed that inclusion of the researchers' and stakeholders' perspectives contributed to an understanding of how the system structure of an environment works. Integration of the adolescents' perspective enriched insights on how adolescents interact with that environment. The analysis further showed that the dynamics driving obesity-related behaviours are geared towards further reinforcing such behaviours.

Theory of Change in Complex Research for Development Programmes: Challenges and Solutions from the Global Challenges Research Fund

The United Kingdom Research and Innovation (UKRI) Global Challenges Research Fund (GCRF) aimed to address global challenges to achieve the United Nations (UN) Sustainable Development Goals through 12 interdisciplinary research hubs. This research documents key lessons learned around working with Theory of Change (ToC) to guide Monitoring, Evaluation and Learning (MEL) within these complex research for development hubs. Interviews and document reviews were conducted in ten of the research hubs. The results revealed that only one hub invested in an explicit visual system mapping approach, and that funder timelines, budget constraints and issues with capacity and expertise limited the application of these approaches across all hubs. In contrast, many hubs attempted to deal with visual complexity by means of ether constructing multiple, nested ToCs, or a conscious simplification of complexity through reducing their ToC towards a straightforward and uncomplicated chain model or spherical model. While the former approach had some value, most hubs struggled to find capacity to support the full articulation of nested ToCs. In contrast, the latter approach resulted in ToCs which lacked detail or mechanism articulation, but which nevertheless were often 'fit for purpose' in ensuring effective communication and coherence across diverse stakeholders and sub-projects. We conclude that in instances where the reporting, funding and management cycles of complex research for development programmes cannot be adapted to properly support learning-based approaches to ToC development, imposing simplicity in the ToC might be fit for purpose. This might also be preferable to more complex visual approaches that are only partially realised.

Addressing alcohol-related harms in the local night-time economy: a qualitative process evaluation from a complex systems perspective

OBJECTIVES

English local authorities (LAs) are interested in reducing alcohol-related harms and may use discretionary powers such as the Late Night Levy (LNL) to do so. This study aims to describe how system stakeholders hypothesise the levy may generate changes and to explore how the system, its actors and the intervention adapt and co-evolve over time.

DESIGN

A process evaluation from a complex systems perspective, using qualitative methods.

SETTING

A London LA with high densities of residential and commercial properties, which implemented the LNL in 2014.

PARTICIPANTS

Data were generated through interviews with LNL implementers and alcohol consumers, observations in bars and during LNL patrols and documentary review.

INTERVENTION

The LNL allows LAs to charge late-night alcohol retailers an annual fee (£299-£4440) to manage and police the night-time economy (NTE).

RESULTS

When the LNL was being considered, stakeholders from different interest groups advanced diverse opinions about its likely impacts while rarely referencing supporting research evidence. Proponents of the levy argued it could reduce crime and anti-social behaviour by providing additional funds to police and manage the NTE. Critics of the levy hypothesised adverse consequences linked to claims that the intervention would force venues to vary their hours or close, cluster closing times, reduce NTE diversity and undermine public-private partnerships. In the first 2 years, levy-funded patrols developed relationships with the licensed trade and the public. The LNL did not undermine public-private partnerships and while some premises varied their hours, these changes did not undermine the intervention's viability, nor significantly cluster venue closing times, nor obviously damage the area's reputation for having a diverse NTE.

CONCLUSIONS

This study applies a framework for process evaluation from a complex systems perspective. The evaluation could be extended to measure alcohol-related outcomes and to consider the interplay between the national and local systems.

Navigating the river(s) of systems change: a multi-methods, qualitative evaluation exploring the implementation of a systems approach to physical activity in Gloucestershire, England

OBJECTIVES

Systems approaches aim to change the environments in which people live, through cross-sectoral working, by harnessing the complexity of the problem. This paper sought to identify: (1) the strategies which support the implementation of We Can Move (WCM), (2) the barriers to implementation, (3) key contextual factors that influence implementation and (4) impacts associated with WCM.

DESIGN

A multi-methods evaluation of WCM was completed between April 2019 and April 2021. Ripple Effects Mapping (REM) and semi-structured interviewers were used. Framework and content analysis were systematically applied to the dataset.

SETTING

WCM-a physical activity orientated systems approach being implemented in Gloucestershire, England.

PARTICIPANTS

31 stakeholder interviews and 25 stakeholders involved in 15 REM workshops.

RESULTS

A white-water rafting analogy was developed to present the main findings. The successful implementation of WCM required a facilitative, well-connected and knowledgeable guide (ie, the lead organisation), a crew (ie, wider stakeholders) who's vision and agenda aligned with WCM's purpose, and a flexible delivery approach that could respond to ever-changing nature of the river (ie, local and national circumstances). The context surrounding WCM further strengthened and hampered its implementation. Barriers included evaluative difficulties, a difference in stakeholder and organisational perspectives, misaligned expectations and understandings of WCM, and COVID-19 implications (COVID-19 also presented as a facilitative factor). WCM was said to strengthen cohesion and collaboration between partners, benefit other agendas and policies (eg, mental health, town planning, inequality), and improve physical activity opportunities and environments.

CONCLUSIONS

This paper is one of the first to evaluate a systems approach to increasing physical activity. We highlight key strategies and contextual factors that influenced the implementation of WCM and demonstrate some of the wider benefits from such approaches. Further research and methodologies are required to build the evidence base surrounding systems approaches in Public Health.

Presenting a comprehensive multi-scale evaluation framework for participatory modelling programs: A scoping review

INTRODUCTION

Systems modelling and simulation can improve understanding of complex systems to support decision making, better managing system challenges. Advances in technology have facilitated accessibility of modelling by diverse stakeholders, allowing them to engage with and contribute to the development of systems models (participatory modelling). However, despite its increasing applications across a range of disciplines, there is a growing need to improve evaluation efforts to effectively report on the quality, importance, and value of participatory modelling. This paper aims to identify and assess evaluation frameworks, criteria, and/or processes, as well as to synthesize the findings into a comprehensive multi-scale framework for participatory modelling programs.

MATERIALS AND METHODS

A scoping review approach was utilized, which involved a systematic literature search via Scopus in consultation with experts to identify and appraise records that described an evaluation framework, criteria, and/or process in the context of participatory modelling. This scoping review is registered with the Open Science Framework.

RESULTS

The review identified 11 studies, which varied in evaluation purposes, terminologies, levels of examination, and time points. The review of studies highlighted areas of overlap and opportunities for further development, which prompted the development of a comprehensive multi-scale evaluation framework to assess participatory modelling programs across disciplines and systems modelling methods. The framework consists of four categories (Feasibility, Value, Change/Action, Sustainability) with 30 evaluation criteria, broken down across project-, individual-, group- and system-level impacts.

DISCUSSION & CONCLUSION

The presented novel framework brings together a significant knowledge base into a flexible, cross-sectoral evaluation effort that considers the whole participatory modelling process. Developed through the rigorous synthesis of multidisciplinary expertise from existing studies, the application of the framework can provide the opportunity to understand practical future implications such as which aspects are particularly important for policy decisions, community learning, and the ongoing improvement of participatory modelling methods.

The ENCOMPASS framework: a practical guide for the evaluation of public health programmes in complex adaptive systems

Background

Systems thinking embraces the complexity of public health problems, including childhood overweight and obesity. It aids in understanding how factors are interrelated, and it can be targeted to produce favourable changes in a system. There is a growing call for systems approaches in public health research, yet limited practical guidance is available on how to evaluate public health programmes within complex adaptive systems. The aim of this paper is to present an evaluation framework that supports researchers in designing systems evaluations in a comprehensive and practical way.

Methods

We searched the literature for existing public health systems evaluation studies. Key characteristics on how to conduct a systems evaluation were extracted and compared across studies. Next, we overlaid the identified characteristics to the context of the Lifestyle Innovations Based on Youth Knowledge and Experience (LIKE) programme evaluation and analyzed which characteristics were essential to carry out the LIKE evaluation. This resulted in the Evaluation of Programmes in Complex Adaptive Systems (ENCOMPASS) framework.

Results

The ENCOMPASS framework includes five iterative stages: (1) adopting a system dynamics perspective on the overall evaluation design; (2) defining the system boundaries; (3) understanding the pre-existing system to inform system changes; (4) monitoring dynamic programme output at different system levels; and (5) measuring programme outcome and impact in terms of system changes.

Conclusions

The value of ENCOMPASS lies in the integration of key characteristics from existing systems evaluation studies, as well as in its practical, applied focus. It can be employed in evaluating public health programmes in complex adaptive systems. Furthermore, ENCOMPASS provides guidance for the entire evaluation process, all the way from understanding the system to developing actions to change it and to measuring system changes. By the nature of systems thinking, the ENCOMPASS framework will likely evolve further over time, as the field expands with more completed studies.

Evaluating the integration of strategic priorities within a complex research-for-development funding program

This paper examines the application of Complexity Theory constructs to a research-for-development program evaluation and presents an overview of the implications and promising approaches for evaluating complex programs. We discuss lessons learned from an evaluation completed for the International Development Research Centre's Food, Environment and Health (FEH) program, which investigated the integration and outcomes of five strategic program priorities: partnerships, southern leadership, gender and equity, scale, and environmental sustainability. We present interpretations from a secondary, thematic content analysis that categorized evaluation findings across four complexity constructs: emergence, unpredictability, contradiction and self-organization. Viewing the evaluation through these constructs surfaced some important features of the FEH program to date, specifically its evolving approach, adaptiveness to emergent issues, non-linear outcomes, and self-organizing agents, which had several implications for the evaluative process. We conclude that the most appropriate evaluation designs for complex funding programs are participatory (to explore all stakeholders' influence), adaptive (to capture the unexpected) and assess external contexts. The application of complexity constructs may be useful for evaluators to gain a deeper understanding of how program contexts change in the face of complexity and why some evaluation methods work more effectively than others.

Implementation processes in a cognitive rehabilitation intervention for people with dementia: a complexity-informed qualitative analysis

OBJECTIVES

Healthcare is often delivered through complex interventions. Understanding how to implement these successfully is important for optimising services. This article demonstrates how the complexity theory concept of 'self-organisation' can inform implementation, drawing on a process evaluation within a randomised controlled trial of the GREAT (Goal-oriented cognitive Rehabilitation in Early-stage Alzheimer's and related dementias: a multi-centre single-blind randomised controlled Trial) intervention which compared a cognitive rehabilitation intervention for people with dementia with usual treatment.

DESIGN

A process evaluation examined experiences of GREAT therapists and participants receiving the intervention, through thematic analysis of a focus group with therapists and interviews with participants and their carers. Therapy records of participants receiving the intervention were also analysed using adapted framework analysis. Analysis adopted a critical realist perspective and a deductive-inductive approach to identify patterns in how the intervention operated.

SETTING

The GREAT intervention was delivered through home visits by therapists, in eight regions in the UK.

PARTICIPANTS

Six therapists took part in a focus group, interviews were conducted with 25 participants and 26 carers, and therapy logs for 50 participants were analysed.

INTERVENTION

A 16-week cognitive rehabilitation programme for people with mild-to-moderate dementia.

RESULTS

'Self-organisation' of the intervention occurred through adaptations made by therapists. Adaptations included simplifying the intervention for people with greater cognitive impairment, and extending it to meet additional needs. Relational work by therapists produced an emergent outcome of 'social support'. Self-organised aspects of the intervention were less visible than formal components, but were important aspects of how it operated during the trial. This understanding can help to inform future implementation.

CONCLUSIONS

Researchers are increasingly adopting complexity theory to understand interventions. This study extends the application of complexity theory by demonstrating how 'self-organisation' was a useful concept for understanding aspects of the intervention that would have been missed by focusing on formal intervention components. Analysis of self-organisation could enhance future process evaluations and implementation studies.

TRIAL REGISTRATION NUMBER

ISRCTN21027481.

Confronting complexity and supporting transformation through health systems mapping: a case study

INTRODUCTION

Health systems are a complex web of interacting and interconnected parts; introducing an intervention, or the allocation of resources, in one sector can have effects across other sectors and impact the entire system. A prerequisite for effective health system reorganisation or transformation is a broad and common understanding of the current system amongst stakeholders and innovators. Chronic obstructive pulmonary disease (COPD) and heart failure (HF) are common chronic diseases with high health care costs that require an integrated health system to effectively treat.

STUDY DESCRIPTION

This case study documents the first phase of system transformation at a regional level in Ontario, Canada. In this first phase, visual representations of the health system in its current state were developed using a collaborative co-creation approach, and a focus on COPD and HF. Multiple methods were used including focus groups, open-ended questionnaires, and document review, to develop a series of graphical and visual representations; a health care ecosystem map.

RESULTS

The ecosystem map identified key sectoral components, inter-component interactions, and care requirements for patients with COPD and HF and inventoried current programs and services available to deliver this care. Main findings identified that independent system-wide navigation for this vulnerable patient group is limited, primary care is central to the accessibility of nearly half of the identified care elements, and resources are not equitably distributed. The health care ecosystem mapping helped to identify care gaps and illustrates the need to resource the primary care provider and the patient with system navigation resources and interdisciplinary team care.

CONCLUSION

The co-created health care ecosystem map brought a collective understanding of the health care system as it applies to COPD and HF. The map provides a blueprint that can be adapted to other disease states and health systems. Future transformation will build on this foundational work, continuing the robust interdisciplinary co-creation strategies, exploring predictive health system modelling and identifying areas for integration.

Strengthening prevention in communities through systems change: lessons from the evaluation of Healthy Families NZ

This article presents the findings from the first 3 years of the evaluation of Healthy Families NZ, a systems-change intervention to prevent chronic diseases in 10 communities. The initiative, which builds on existing prevention activities, aims to strengthen the health prevention system through evidence-driven action to enable people to make good food choices, be physically active, smoke-free and free from alcohol-related harm. Key investment areas are a dedicated systems thinking and acting health promotion workforce, and activating leaders who can influence transformational change. The evaluation to date has found the initiative is being implemented with integrity. Evidence indicates a shift towards greater action on prevention, and the prevention system being strengthened. Māori ownership has been enabled, and prioritizing equity has led teams to utilize methods that amplify diverse local perspectives. There is progress on developing a flexible workforce through adaptive learning, flexible resources, professional development and a responsive National team. There is also progress in activating local leadership and empowering local teams. The initiative design has explicitly taken into account the context of complexity within which it is being implemented. It has evolved to focus on action that can accelerate sharing information and practices within communities, and between policy and decision-makers. Healthy Families NZ and its evaluation have been refunded to 2022. This provides an important opportunity to gather further insight into effective ways to strengthen the community agency and trust needed to promote and deliver evidence-based action on prevention.

Evaluation of public health interventions from a complex systems perspective: A research methods review

INTRODUCTION

Applying a complex systems perspective to public health evaluation may increase the relevance and strength of evidence to improve health and reduce health inequalities. In this review of methods, we aimed to: (i) classify and describe different complex systems methods in evaluation applied to public health; and (ii) examine the kinds of evaluative evidence generated by these different methods.

METHODS

We adapted critical review methods to identify evaluations of public health interventions that used systems methods. We conducted expert consultation, searched electronic databases (Scopus, MEDLINE, Web of Science), and followed citations of relevant systematic reviews. Evaluations were included if they self-identified as using systems- or complexity-informed methods and if they evaluated existing or hypothetical public health interventions. Case studies were selected to illustrate different types of complex systems evaluation.

FINDINGS

Seventy-four unique studies met our inclusion criteria. A framework was developed to map the included studies onto different stages of the evaluation process, which parallels the planning, delivery, assessment, and further delivery phases of the interventions they seek to inform; these stages include: 1) theorising; 2) prediction (simulation); 3) process evaluation; 4) impact evaluation; and 5) further prediction (simulation). Within this framework, we broadly categorised methodological approaches as mapping, modelling, network analysis and 'system framing' (the application of a complex systems perspective to a range of study designs). Studies frequently applied more than one type of systems method.

CONCLUSIONS

A range of complex systems methods can be utilised, adapted, or combined to produce different types of evaluative evidence. Further methodological innovation in systems evaluation may generate stronger evidence to improve health and reduce health inequalities in our complex world.

Using extended complexity theory to test SMEs' adoption of Blockchain-based loan system

Blockchain-based loan system can be summed up as: information exchange between various government departments; information exchange between enterprises and various financial institutions; detection of the actual use of loans in the form of encrypted currency. This technology is supposed to reduce a lot of financing costs for SMEs on average. Therefore, this research extends complexity theory to discover the factors that affect the use of Blockchain loan systems by SMEs. Complexity, perceived risk, perceived fairness and reward sensitivity prove to have significant effects on usage intention. Complexity proves to have moderating effects on other relationships. This research may contribute to the system performance improvement and provide opportunities for SMEs to share information with financial institutions or individuals around the world, thereby providing investors with equal opportunities for competition.

Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators

BACKGROUND

Public health evaluation methods have been criticized for being overly reductionist and failing to generate suitable evidence for public health decision-making. A "complex systems approach" has been advocated to account for real world complexity. Qualitative methods may be well suited to understanding change in complex social environments, but guidance on applying a complex systems approach to inform qualitative research remains limited and underdeveloped. This systematic review aims to analyze published examples of process evaluations that utilize qualitative methods that involve a complex systems perspective and proposes a framework for qualitative complex system process evaluations.

METHODS AND FINDINGS

We conducted a systematic search to identify complex system process evaluations that involve qualitative methods by searching electronic databases from January 1, 2014-September 30, 2019 (Scopus, MEDLINE, Web of Science), citation searching, and expert consultations. Process evaluations were included if they self-identified as taking a systems- or complexity-oriented approach, integrated qualitative methods, reported empirical findings, and evaluated public health interventions. Two reviewers independently assessed each study to identify concepts associated with the systems thinking and complexity science traditions. Twenty-one unique studies were identified evaluating a wide range of public health interventions in, for example, urban planning, sexual health, violence prevention, substance use, and community transformation. Evaluations were conducted in settings such as schools, workplaces, and neighborhoods in 13 different countries (9 high-income and 4 middle-income). All reported some utilization of complex systems concepts in the analysis of qualitative data. In 14 evaluations, the consideration of complex systems influenced intervention design, evaluation planning, or fieldwork. The identified studies used systems concepts to depict and describe a system at one point in time. Only 4 evaluations explicitly utilized a range of complexity concepts to assess changes within the system resulting from, or co-occurring with, intervention implementation over time. Limitations to our approach are including only English-language papers, reliance on study authors reporting their utilization of complex systems concepts, and subjective judgment from the reviewers relating to which concepts featured in each study.

CONCLUSION

This study found no consensus on what bringing a complex systems perspective to public health process evaluations with qualitative methods looks like in practice and that many studies of this nature describe static systems at a single time point. We suggest future studies use a 2-phase framework for qualitative process evaluations that seek to assess changes over time from a complex systems perspective. The first phase involves producing a description of the system and identifying hypotheses about how the system may change in response to the intervention. The second phase involves following the pathway of emergent findings in an adaptive evaluation approach.

What prevents health policy being 'evidence-based'? New ways to think about evidence, policy and interventions in health

BACKGROUND

Evidence-based policy decision-making is a dominant paradigm in health but realizing this ideal has proven challenging.

SOURCES OF DATA

This paper conceptually maps health policy, policy studies and social science literature critically engaged with evidence and decision-making. No new data were generated or analysed in support of this review.

AREAS OF AGREEMENT

Barriers to evidence-based policy have been documented, with efforts made to increase the uptake of evidence.

AREAS OF CONTROVERSY

Evident complexities have been regarded as a problem of translation. However, this assumes that policy-making is a process of authoritative choice, and that 'evidence' is inherently valuable policy knowledge, which has been critiqued.

GROWING POINTS

Alternative accounts urge consideration of how evidence comes to bear on decisions made within complex systems, and what counts as evidence.

AREAS TIMELY FOR DEVELOPING RESEARCH

An 'evidence-making intervention' approach offers a framework for conceptualizing how evidence and interventions are made relationally in practices, thus working with the politics and contingencies of implementation and policy-making.

Applying a complex adaptive systems approach to the evaluation of a school-based intervention for intimate partner violence prevention in Mexico

Despite calls for evaluation practice to take a complex systems approach, there are few examples of how to incorporate complexity into real-life evaluations. This article presents the case for using a complex systems approach to evaluate a school-based intimate partner violence-prevention intervention. We conducted a post hoc analysis of qualitative evaluation data to examine the intervention as a potential system disruptor. We analysed data in relation to complexity concepts particularly relevant to schools: 'diverse and dynamic agents', 'interaction', 'unpredictability', 'emergence' and 'context dependency'. The data-two focus groups with facilitators and 33 repeat interviews with 14-17-year-old students-came from an evaluation of a comprehensive sexuality education intervention in Mexico City, which serves as a case study for this analysis. The findings demonstrate an application of complex adaptive systems concepts to qualitative evaluation data. We provide examples of how this approach can shed light on the ways in which interpersonal interactions, group dynamics, the core messages of the course and context influenced the implementation and outcomes of this intervention. This gender-transformative intervention appeared to disrupt pervasive gender norms and reshape beliefs about how to engage in relationships. An intervention comprises multiple dynamic and interacting elements, all of which are unlikely to be consistent across implementation settings. Applying complexity concepts to our analysis added value by helping reframe implementation-related data to focus on how the 'social' aspects of complexity influenced the intervention. Without examining both individual and group processes, evaluations may miss key insights about how the intervention generates change, for whom, and how it interacts with its context. A social complex adaptive systems approach is well-suited to the evaluation of gender-transformative interventions and can help identify how such interventions disrupt the complex social systems in which they are implemented to address intractable societal problems.

A System Dynamics and Participatory Action Research Approach to Promote Healthy Living and a Healthy Weight among 10-14-Year-Old Adolescents in Amsterdam: The LIKE Programme

This paper describes the design of the LIKE programme, which aims to tackle the complex problem of childhood overweight and obesity in 10–14-year-old adolescents using a systems dynamics and participatory approach. The LIKE programme focuses on the transition period from 10-years-old to teenager and was implemented in collaboration with the Amsterdam Healthy Weight Programme (AHWP) in Amsterdam-East, the Netherlands. The aim is to develop, implement and evaluate an integrated action programme at the levels of family, school, neighbourhood, health care and city. Following the principles of Participatory Action Research (PAR), we worked with our population and societal stakeholders as co-creators. Applying a system lens, we first obtained a dynamic picture of the pre-existing systems that shape adolescents’ behaviour relating to diet, physical activity, sleep and screen use. The subsequent action programme development was dynamic and adaptive, including quick actions focusing on system elements (quick evaluating, adapting and possibly catalysing further action) and more long-term actions focusing on system goals and/or paradigm change. The programme is supported by a developmental systems evaluation and the Intervention Level Framework, supplemented with routinely collected data on weight status and health behaviour change over a period of five years. In the coming years, we will report how this approach has worked to provide a robust understanding of the programme’s effectiveness within a complex dynamic system. In the meantime, we hope our study design serves as a source of inspiration for other public health intervention studies in complex systems.

Using implementation science theories and frameworks in global health

In global health, researchers and decision makers, many of whom have medical, epidemiology or biostatistics background, are increasingly interested in evaluating the implementation of health interventions. Implementation science, particularly for the study of public policies, has existed since at least the 1930s. This science makes compelling use of explicit theories and analytic frameworks that ensure research quality and rigour. Our objective is to inform researchers and decision makers who are not familiar with this research branch about these theories and analytic frameworks. We define four models of causation used in implementation science: intervention theory, frameworks, middle-range theory and grand theory. We then explain how scientists apply these models for three main implementation studies: fidelity assessment, process evaluation and complex evaluation. For each study, we provide concrete examples from research in Cuba and Africa to better understand the implementation of health interventions in global health context. Global health researchers and decision makers with a quantitative background will not become implementation scientists after reading this article. However, we believe they will be more aware of the need for rigorous implementation evaluations of global health interventions, alongside impact evaluations, and in collaboration with social scientists.

Health Inequality as a Large-Scale Outcome of Complex Social Systems: Lessons for Action on the Sustainable Development Goals

Action on the Sustainable Development Goals (SDGs) needs to become real and impactful, taking a “whole systems” perspective on levers for systems change. This article reviews what we have learned over the past century about the large-scale outcome of health inequality, and what we know about the behaviour of complex social systems. This combined knowledge provides lessons on the nature of inequality and what effective action on our big goals, like the SDGs, might look like. It argues that economic theories and positivist social theories which have dominated the last 150 years have largely excluded the nature of human connections to each other, and the environment. This exclusion of intimacy has legitimatised arguments that only value-free economic processes matter for macro human systems, and only abstract measurement constitutes valuable social science. Theories of complex systems provide an alternative perspective. One where health inequality is viewed as emergent, and causes are systemic and compounding. Action therefore needs to be intensely local, with power relationships key to transformation. This requires conscious and difficult intervention on the intolerable accumulation of resources; improved reciprocity between social groups; and reversal of system flows, which at present ebb away from the local and those already disadvantaged.

Using developmental evaluation to support knowledge translation: reflections from a large-scale quality improvement project in Indigenous primary healthcare

BACKGROUND

Developmental evaluation is a growing area of evaluation practice, advocated for informing the adaptive development of change initiatives in complex social environments. The utilisation focus, complexity perspective and systems thinking of developmental evaluation suggest suitability for evaluating knowledge translation initiatives in primary healthcare. However, there are few examples in the literature to guide its use in these contexts and in Indigenous settings. In this paper, we reflect on our experience of using developmental evaluation to implement a large-scale knowledge translation research project in Australian Aboriginal and Torres Strait Islander primary healthcare. Drawing on principles of knowledge translation and key features of developmental evaluation, we debate the key benefits and challenges of applying this approach to engage diverse stakeholders in using aggregated quality improvement data to identify and address persistent gaps in care delivery.

DISCUSSION

The developmental evaluation enabled the team to respond to stakeholder feedback and apply learning in real-time to successfully refine theory-informed research and engagement processes, tailor the presentation of findings to stakeholders and context, and support the project's dissemination and knowledge co-production aim. It thereby contributed to the production of robust, useable research findings for informing policy and system change. The use of developmental evaluation appeared to positively influence stakeholders' use of the project reports and their responses to the findings. Challenges included managing a high volume of evaluation data and multiple evaluation purposes, balancing facilitative sense-making processes and change with task-focused project management, and lack of experience in using this evaluation approach. Use of an embedded evaluator with facilitation skills and background knowledge of the project helped to overcome these challenges, as did similarities observed between features of developmental evaluation and continuous quality improvement.

CONCLUSION

Our experience of developmental evaluation confirmed our expectations of the potential value of this approach for strengthening improvement interventions and implementation research, and particularly for adapting healthcare innovations in Indigenous settings. In our project, developmental evaluation successfully encompassed evaluation, project adaptation, capacity development and knowledge translation. Further work is warranted to apply this approach more widely to improve primary healthcare initiatives and outcomes, and to evaluate implementation research.

The importance of systems thinking, context, and inclusion when studying the feasibility of expanding the FoodShare (SNAP) outreach program to rural communities of Northern Wisconsin

Evaluators often neglect to question whether a program has been appropriately designed for a new context prior to a feasibility study's commencement. In this paper, we document the results of a case study that closely examined context when determining the feasibility of implementing a FoodShare outreach program in rural northern Wisconsin. Using community-based participatory mixed methods, we examine how stakeholder engagement led to both study refinement and a more comprehensive understanding of community food security based on contextual factors and systems thinking. Lessons learned when conducting feasibility studies from this perspective are provided to improve evaluation practice.

Using mixed method approach in measuring effects of training in firms: Case study of the European Social Fund support

Public support of training in firms corresponds to the long-term importance of the quality of human capital in the competitiveness of firms and nations. Thus, the EU supports such training via the European Social Fund (ESF). The evaluation community evaluates the support by using either qualitative or quantitative methods. The simultaneous application of these two approaches is rare. The purpose of this paper is to combine quantitative (counterfactual impact evaluation) and qualitative (qualitative comparative analysis) methods in order to fill the methodological gap. Based on the combination of both approaches, it explores their strengths, complementarity and disadvantages to evaluate public support for employee training in the Czech Republic. The combination of methods makes it possible to identify not only the impacts but also their causes. Linking the ESF support to corporate competitiveness is crucial for demonstrating the effectiveness of public spending.

Applying pragmatic approaches to complex program evaluation: A case study of implementation of the New South Wales Get Healthy at Work program

ISSUE ADDRESSED

Complex health promotion programs, which can have multilevels of implementation and multi-components with nonlinear causal pathways, present many evaluation challenges. Traditional evaluation methods often fail to account for the complexity inherent in assessing these programs. In real-world settings, evaluations of complex programs are often beset by additional constraints of limited budgets and short timeframes. Determining whether a complex program is successful and how a program worked requires evaluators of complex programs to adopt a level of pragmatism.

METHODS

This paper describes a pragmatic evaluation approach used to evaluate the Get Healthy at Work workplace health promotion program, implemented in New South Wales, Australia. Using the program as a case study, we describe some key principles for applying a pragmatic evaluation approach and use these principles to develop an appropriate evaluation strategy.

RESULTS

The evaluation includes multiple research methods to assess program outputs and implementation; and identify emergent program impacts, within constrained resources. The evaluation was guided by epistemological flexibility, methodological comprehensiveness and operational practicality.

CONCLUSION

Health promotion programs, such as state-wide obesity prevention programs, require appropriate evaluation methods which address their inherent complexity amidst the real-world evaluation constraints, and focuses on the essential evaluation needs.

SO WHAT

The main complex program evaluation principles are applicable to other multilevel health promotion programs, challenged by methodological and practical or political constraints.

Wrestling with the complexity of evaluation for organizations at the boundary of science, policy, and practice

Boundary organizations are situated between science, policy, and practice and have a goal of supporting communication and collaboration among these sectors. They have been promoted as a way to improve the effectiveness of conservation efforts by building stronger relationships between scientists, policy makers, industry, and practitioners (Cook et al. 2013). Although their promise has been discussed in theory, the work of and expectations for boundary organizations are less defined in practice. Biodiversity conservation is characterized by complexity, uncertainty, dissent, and tight budgets, so boundary organizations face the challenging task of demonstrating their value to diverse stakeholders. We examined the challenges boundary organizations face when seeking to evaluate their work and thus aimed to encourage more productive conversations about evaluation of boundary organizations and their projects. Although no off-the-shelf solution is available for a given boundary organization, we identified 4 principles that will support effective evaluation for boundary organizations: engage diverse stakeholders, support learning and reflection, assess contribution to change, and align evaluation with assumption and values.

Complex adaptive systems: a new approach for understanding health practices

This article explores the potential of complex adaptive systems (CAS) theory to inform behaviour change research. A CAS describes a collection of heterogeneous agents interacting within a particular context, adapting to each other's actions. In practical terms, this implies that behaviour change is (1) socially and culturally situated; (2) highly sensitive to small baseline differences in individuals, groups, and intervention components; and (3) determined by multiple components interacting 'chaotically'. Two approaches to studying CAS are briefly reviewed. Agent-based modelling is a computer simulation technique that allows researchers to investigate 'what if' questions in a virtual environment. Applied qualitative research techniques, on the other hand, offer a way to examine what happens when an intervention is pursued in real-time, and to identify the sorts of rules and assumptions governing social action. Although these represent very different approaches to complexity, there may be scope for mixing these methods - for example, by grounding models in insights derived from qualitative fieldwork. Finally, I will argue that the concept of CAS offers one opportunity to gain a deepened understanding of health-related practices, and to examine the social psychological processes that produce health-promoting or damaging actions.

A Systems Thinking Approach to Understanding and Demonstrating the Role of Peer-Led Programs and Leadership in the Response to HIV and Hepatitis C: Findings From the W3 Project

The central role of community and peer-led programs has been a key characteristic of the Australian partnership response to HIV and hepatitis C since the beginning of the epidemics. Despite this, peer-led programs continue to have limited capacity to demonstrate their role and value as part of a multi-sectoral response. What makes one peer-led program a better investment than another? What role does the rest of the sector have in ensuring we gain the most value from these investments? To investigate this, we facilitated interactive systems thinking methods with 10 programs working within communities of people who inject drugs, gay men, sex workers and people living with HIV across Australia. This included articulating program theories in diagram and textual form to help us understand the role of peer-based programs promoting peer leadership within the Australian HIV and hepatitis C responses. Our aim was to develop a framework for monitoring and evaluation that could be applied to peer led programs at different levels and in different contexts. We found that for peer-led programs to fulfill their role, and to navigate the rapid changes occurring in the both epidemics, they need to: demonstrate the credibility of their peer and community insights; continually adapt to changing contexts and policy priorities in tandem with their communities; and maintain influence in both community and policy systems. We developed a framework of four key functions (Engagement, Alignment, Adaptation, and Influence) which peer-based programs need to demonstrate, which form the basis for identifying quality indicators. This article presents a new way of framing and monitoring investments in peer-led programs and peer eadership actions by these programs. If health policy is committed to strengthening the leadership shown by affected communities, then we need to understand, enhance, monitor and value the role of peer-led programs and peer leadership within the overall prevention system. We believe the W3 framework, drawing on systems thinking and modeling, can support funders, policy-makers and programs to achieve this.

Being pragmatic about healthcare complexity: our experiences applying complexity theory and pragmatism to health services research

BACKGROUND

The healthcare system has proved a challenging environment for innovation, especially in the area of health services management and research. This is often attributed to the complexity of the healthcare sector, characterized by intersecting biological, social and political systems spread across geographically disparate areas. To help make sense of this complexity, researchers are turning towards new methods and frameworks, including simulation modeling and complexity theory.

DISCUSSION

Herein, we describe our experiences implementing and evaluating a health services innovation in the form of simulation modeling. We explore the strengths and limitations of complexity theory in evaluating health service interventions, using our experiences as examples. We then argue for the potential of pragmatism as an epistemic foundation for the methodological pluralism currently found in complexity research. We discuss the similarities between complexity theory and pragmatism, and close by revisiting our experiences putting pragmatic complexity theory into practice.

CONCLUSION

We found the commonalities between pragmatism and complexity theory to be striking. These included a sensitivity to research context, a focus on applied research, and the valuing of different forms of knowledge. We found that, in practice, a pragmatic complexity theory approach provided more flexibility to respond to the rapidly changing context of health services implementation and evaluation. However, this approach requires a redefinition of implementation success, away from pre-determined outcomes and process fidelity, to one that embraces the continual learning, evolution, and emergence that characterized our project.

Factors associated with the implementation of community-based peer-led health promotion programs: A scoping review

Peer education is growing in popularity as a useful health promotion strategy. However, optimal conditions for implementing peer-led health promotion programs (HPPs) remain unclear. This scoping review aimed to describe factors that can influence implementation of peer-led HPPs targeting adult populations. Five databases were searched using the keywords "health promotion/prevention", "implementation", "peers", and related terms. Studies were included if they reported at least one factor associated with the implementation of community-based peer-led HPPs. Fifty-five studies were selected for the analysis. The method known as "best fit framework synthesis" was used to analyze the factors identified in the selected papers. Many factors included in existing implementation conceptual frameworks were deemed applicable to peer-led HPPs. However, other factors related to individuals, programs, and implementation context also emerged from the analysis. Based on this synthesis, an adapted theoretical framework was elaborated, grounded in a complex adaptive system perspective and specifying potential mechanisms through which factors may influence implementation of community-based peer-led HPPs. Further research is needed to test the theoretical framework against empirical data. Findings from this scoping review increase our knowledge of the optimal conditions for implementing peer-led HPPs and thereby maximizing the benefits of such programs.

Using Complexity Theory to Guide Medical School Evaluations

Contemporary medical school evaluations are narrow in focus and often do not consider the wider systems implications of the relationship between learning and teaching, research, clinical care, and community engagement. The result is graduates who lack the necessary knowledge and skills for the modern health care system and an educational system that is limited in its ability to learn and change.To address this issue, the authors apply complexity theory to medical school evaluation, using four key factors-nesting, diversity, self-organization, and emergent outcomes. To help medical educators apply this evaluation approach in their own settings, the authors offer two tools-a modified program logic model and sensemaking. In sensemaking, they use the organic metaphor of the medical school as a neuron situated within a complex neural network to enable medical educators to reframe the way they think about program evaluation. The authors then offer practical guidance for applying this model, including describing the example of addressing graduates' engagement in the health care system. The authors consider the input of teachers, the role of culture and curriculum, and the clinical care system in this example.Medical school evaluation is reframed as an improvement science for complex social interventions (medical school is such an intervention) in this model. With complexity theory's focus on emergent outcomes, evaluation takes on a new focus, reimagining medical students as reaching their future potential as change agents, who transform health systems and the lives of patients.

Making sense of the emerging conversation in evaluation about systems thinking and complexity science

In the last twenty years, a conversation has emerged in the evaluation field about the potential of systems thinking and complexity science (STCS) to transform the practice of evaluating social interventions. Documenting and interpreting this conversation are necessary to advance our understanding of the significance of using STCS in planning, implementing, and evaluating social interventions. Guided by a generic framework for evaluation practice, this paper reports on an inter-disciplinary literature review and argues that STCS raises some new ways of thinking about and carrying out the following six activities: 1) supporting social problem solving; 2) framing interventions and contexts; 3) selecting and using methods; 4) engaging in valuing; 5) producing and justifying knowledge; and 6) facilitating use. Following a discussion of these issues, future directions for research and practice are suggested.

Systems science and systems thinking for public health: a systematic review of the field

OBJECTIVES

This paper reports on findings from a systematic review designed to investigate the state of systems science research in public health. The objectives were to: (1) explore how systems methodologies are being applied within public health and (2) identify fruitful areas of activity.

DESIGN

A systematic review was conducted from existing literature that draws on or uses systems science (in its various forms) and relates to key public health areas of action and concern, including tobacco, alcohol, obesity and the social determinants of health.

DATA ANALYSIS

117 articles were included in the review. An inductive qualitative content analysis was used for data extraction. The following were systematically extracted from the articles: approach, methodology, transparency, strengths and weaknesses. These were then organised according to theme (ie, commonalities between studies within each category), in order to provide an overview of the state of the field as a whole. The assessment of data quality was intrinsic to the goals of the review itself, and therefore, was carried out as part of the analysis.

RESULTS

4 categories of research were identified from the review, ranging from editorial and commentary pieces to complex system dynamic modelling. Our analysis of each of these categories of research highlighted areas of potential for systems science to strengthen public health efforts, while also revealing a number of limitations in the dynamic systems modelling being carried out in public health.

CONCLUSIONS

There is a great deal of interest in how the application of systems concepts and approach might aid public health. Our analysis suggests that soft systems modelling techniques are likely to be the most useful addition to public health, and align well with current debate around knowledge transfer and policy. However, the full range of systems methodologies is yet to be engaged with by public health researchers.

Investigating combination HIV prevention: isolated interventions or complex system

Introduction

Treatment as prevention has mobilized new opportunities in preventing HIV transmission and has led to bold new UNAIDS targets in testing, treatment coverage and transmission reduction. These will require not only an increase in investment but also a deeper understanding of the dynamics of combining behavioural, biomedical and structural HIV prevention interventions. High-income countries are making substantial investments in combination HIV prevention, but is this investment leading to a deeper understanding of how to combine interventions? The combining of interventions involves complexity, with many strategies interacting with non-linear and multiplying rather than additive effects.

Discussion

Drawing on a recent scoping study of the published research evidence in HIV prevention in high-income countries, this paper argues that there is a gap between the evidence currently available and the evidence needed to guide the achieving of these bold targets. The emphasis of HIV prevention intervention research continues to look at one intervention at a time in isolation from its interactions with other interventions, the community and the socio-political context of their implementation. To understand and evaluate the role of a combination of interventions, we need to understand not only what works, but in what circumstances, what role the parts need to play in their relationship with each other, when the combination needs to adapt and identify emergent effects of any resulting synergies. There is little development of evidence-based indicators on how interventions in combination should achieve that strategic advantage and synergy. This commentary discusses the implications of this ongoing situation for future research and the required investment in partnership. We suggest that systems science approaches, which are being increasingly applied in other areas of public health, could provide an expanded vocabulary and analytic tools for understanding these complex interactions, relationships and emergent effects.

Conclusions

Relying on the current linear but disconnected approaches to intervention research and evidence we will miss the potential to achieve and understand system-level synergies. Given the challenges in sustaining public health and HIV prevention investment, meeting the bold UNAIDS targets that have been set is likely to be dependent on achieving systems level synergies.

What qualitative research can contribute to a randomized controlled trial of a complex community intervention

Using the case of a large-scale, multi-site Canadian Housing First research demonstration project for homeless people with mental illness, At Home/Chez Soi, we illustrate the value of qualitative methods in a randomized controlled trial (RCT) of a complex community intervention. We argue that quantitative RCT research can neither capture the complexity nor tell the full story of a complex community intervention. We conceptualize complex community interventions as having multiple phases and dimensions that require both RCT and qualitative research components. Rather than assume that qualitative research and RCTs are incommensurate, a more pragmatic mixed methods approach was used, which included using both qualitative and quantitative methods to understand program implementation and outcomes. At the same time, qualitative research was used to examine aspects of the intervention that could not be understood through the RCT, such as its conception, planning, sustainability, and policy impacts. Through this example, we show how qualitative research can tell a more complete story about complex community interventions.