Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators

How did the context of COVID-19 affect the implementation and mechanisms of participatory learning and action to address type 2 diabetes? Mixed-methods research in rural Bangladesh

OBJECTIVES

Research indicates the effectiveness of participatory interventions to address rapid rises in type 2 diabetes in low-income countries. Understanding their transferability to different contexts is a priority. We aimed to analyse how the COVID-19 post-lockdown context and adjustments to a participatory learning and action intervention affected theorised mechanisms of effect in rural Bangladesh and to examine the broader implications of this context and intervention adjustments for developing optimal contexts for participatory interventions.

DESIGN

Mixed methods using longitudinal qualitative and quantitative observation data, focus group discussions and interviews with group and community members and project personnel. We used descriptive content analysis, guided by realist evaluation research questions about context, implementation and mechanisms. We used team reflection to enhance the rigour of our analysis.

SETTING

Cluster-randomised trial in Alfadanga upazila, Faridpur district, in the central region of Bangladesh. The intervention was implemented between January 2020 and December 2022, during the COVID-19 pandemic.

PARTICIPANTS

Participatory group members, community members and project personnel (n=32). Structured observations of participatory groups (n=1820) and unstructured observations of groups and their environments (n=15).

INTERVENTIONS

Participatory learning and action community groups of men and women implemented by community-based facilitators.

RESULTS

Due to COVID-19, the participatory learning and action (PLA) intervention was not implemented as planned, which had major effects on the time available to develop the intervention with communities. Communities learnt about diabetes and were motivated to address its causes at an individual level, but community action was a more challenging mechanism to trigger. The post-pandemic context made it difficult to build community rapport, and strategies to engage communities through home visits were challenging. Communities' prior negative experience in working together and in working with non-governmental organisations led to low community cohesion and low motivation to address diabetes collectively. This also resulted in expectations that the implementing organisation would implement community actions and incentivise attendance at meetings. This misalignment of expectations further disabled relationship building, and community strategies addressing the social causes of diabetes were largely not enacted.

CONCLUSION

PLA has optimal effects when time is available to build trust and social cohesion. These are contextual elements and mechanisms that need to be activated to enable critical reflection and community action to develop an enabling environment to address type 2 diabetes.

TRIAL REGISTRATION NUMBER

ISRCTN42219712.

A Theoretically Informed Process Evaluation in Parallel to a Feasibility Study of a Complex Oral Health Intervention Using NICE Guidelines in a Care Home Setting

BACKGROUND

A theoretically informed process evaluation was undertaken in parallel to a study examining the feasibility of an oral health intervention based on an existing guideline for care homes. The objectives were to explore the factors that influenced the implementation of the intervention in order to understand the potential pathway to impact. The research team initially utilised Pfadenhauer et al.'s framework, which focuses on a number of different implementation factors: intervention characteristics, context, theory, process, strategy, agents, outcomes and setting.

METHODS

Nine semi-structured interviews were undertaken with care home managers and staff, predominantly within the intervention arm of the study. Interview schedules were originally based on Pfadenhauer et al.'s framework. These were coded and analysed using thematic analysis. Given the range of themes that emerged, the research team ran a reflexive workshop to determine whether Pfadenhauer et al.'s framework was able to capture and frame the authentic voice of those interviewed.

RESULTS

The research team found that a systems lens approach better fitted the data from the interviews, capturing the idiosyncrasy of the different settings and the importance of values and beliefs of the key stakeholders. It was clear that unlike the structure proposed by Pfaednhauer et al., many of the factors were interdependent and hierarchical in nature, that is, paradigm and goals within the care home had a direct impact on the system structure, which fed into how the care home was maintained, which led onto how the different actors (care home managers and staff) behaved. The process also highlighted key factors for intervention delivery: time poverty, competing needs, staff turnover, differences between shift patterns and between permanent and agency staff. Cognitive capacity of the residents and staff attitudes were also key.

CONCLUSIONS

Adding a reflexive workshop enabled the research to critically review the Pfadenhauer et al.'s framework and change to a systems lens approach, which better explained the interdependent and hierarchical nature of the findings. It also highlighted a number of key factors that could influence the pathway to impact for the intervention.

TRIAL REGISTRATION

ISRCTN10276613.

Operationalising participatory action research to evaluate early years' population health services

BACKGROUND

Early years interventions are critical to children's health and development and are emerging as core to public health programmes in the UK and globally. Evaluating such interventions is complex. The study reported in this article evaluates a place-based public health initiative 'A Better Start Southend' (ABSS) aimed at facilitating early years' development specifically. It centres on examining the access barriers and facilitators experienced by parents and young children, as perceived by health professionals providing these services.

METHODS

This paper illustrates the utilisation of participatory action research (PAR) approach, employing creative methods, including spider grams, service mapping and photovoice, with health professionals delivering ABSS services. PAR methods enabled exploration of community resources that facilitate or impede early childhood development in the local context.

RESULTS

Operationalising PAR yielded critical providers' perspectives on key challenges of delivering these programmes, and the factors that in their view impeded their uptake by families and hence effectiveness. The approach provided space for authentic knowledge production through critical reflexive enquiry, exchange, collaborative dialogue and transformation. Through the process, participants revealed the social and commercial determinants of childhood development and how these determined the reach and success of ABSS. Health professionals especially highlighted poor-quality housing, poor public transport, the cost-of-living crisis and harmful commercial marketing practices as key barriers to promoting good early childhood development. System-wide barriers were also reported and included poor resourcing of health and social care services, lack of culturally and linguistically accessible provisions, and exclusionary practices creating inequitable access to health for many families and children.

CONCLUSION

PAR is a potentially valuable tool for healthcare evaluations with the ability to generate nuanced reflexive perspectives and considerations that go beyond identifying the outcomes and gaps in interventions. It draws participants into a reflexive process to define pathways for change. Health professionals identified social inequities as the most significant barrier to promoting early childhood development. These inequities were not addressed in the design and implementation of the early year programme under study. The study supports the need for a multi-level, multi-systems and intersectional framework for place-based public health programmes to have the desired impact and reduce inequalities in access to early years interventions.

Maximizing Professional-Led Sexual Well-being Support in Cancer Care: Findings From a Qualitative Process Evaluation Following Healthcare Professionals' Engagement With an eLearning Resource

BACKGROUND

Provision of healthcare professional (HP)-led sexual support in cancer care is lacking, perpetuated by barriers including a lack of HP awareness of sexual concerns and strategies to help patients and partners cope. In response, the Maximizing Sexual Wellbeing|Cancer Care eLearning resource (MSW|CC) was developed and demonstrated efficacy in reducing HPs' attitudinal barriers to the provision of sexual support. However, the mechanisms for such change are not yet known.

OBJECTIVE

A qualitative process evaluation was used to explore mechanisms that influence HP engagement with MSW|CC and its adoption into practice.

METHODS

Semistructured interviews were conducted with HPs providing cancer care in Northern Ireland upon MSW|CC completion. Interviews were audio-recorded, transcribed verbatim, and thematically analyzed.

RESULTS

Seventeen participants were interviewed, and 4 key themes were identified: (1) MSW|CC raises HPs' awareness of the need for holistic sexual support as part of routine clinical care; (2) MSW|CC prepares and equips HPs to provide sexual support in cancer care; (3) MSW|CC is coherent, engaging, and acceptable; and (4) MSW|CC: moving forward.

CONCLUSION

Participants derived benefits from MSW|CC, deeming it an acceptable resource. Insights into MSW|CC mechanisms of success were identified, including language to address sexual concerns, alongside patient referral resources. Perceived MSW|CC shortcomings from an earlier study were confirmed as a software issue.

IMPLICATIONS FOR PRACTICE

Healthcare professionals require training and resources to enhance sexual support provision in cancer care. Proactive implementation strategies used to maximize HP engagement with MSW|CC could equip HPs with tools to improve patient sexual outcomes after cancer.

Processes of development related with the implementation of the Icelandic prevention model in a rural Canadian community

The Icelandic Prevention Model (IPM) is a sequential 10-step community-driven collaborative intervention that is designed to support the prevention of substance use in youth by establishing healthy developmental contexts. The IPM has been implemented across Iceland for over 20 years and is now being implemented in other countries. Recognizing the need to explore how to adapt the IPM to new contexts and document the implementation of the model, this paper describes a process evaluation of the first three steps of the IPM within a Canadian rural community to capture experiences during the early development. Specifically, this study addresses the following research questions: (1) What are the processes of development and contextual features that influence the implementation of the IPM within Lanark County, Ontario? and (2) What adaptations are needed to successfully implement the IPM in Canada? Semi-structured interviews were conducted to examine experiences and lessons learned through the implementation of the model. Thematic analyses were completed using QSR NVivo. A deductive and inductive approach was applied, whereby some interview guide questions were derived from the IPM implementation steps and others were more exploratory, examining context and processes of development. Nine interviews were conducted with key partners who were leading the implementation of the IPM. Themes highlighting cultural factors that influence implementation, processes of development related to community engagement, and themes relating to youth participation, fidelity issues, fundraising, health equity and challenges related to the COVID-19 pandemic were identified. This paper contributes new scientific knowledge related to implementation processes within upstream prevention of substance use and practical information that is useful for communities interested in implementing the IPM.

Evaluating a whole-school approach to addressing gender-based violence in Scottish secondary schools (Equally Safe at School): a study protocol for a type I hybrid effectiveness-implementation trial

INTRODUCTION

Equally Safe at School (ESAS) is a whole-school intervention to reduce gender-based violence (GBV) in secondary school. ESAS comprises self-assessment, student-led action group, two-tier staff training, curriculum enhancement and policy review. Schools set up key activities in Year 1 and embed them in Year 2. GBV, including sexual harassment, is common in secondary schools and disproportionately affects young women and lesbian, gay, bisexual, transgender and queer youth.

METHODS AND ANALYSIS

We will evaluate the effectiveness, cost-effectiveness, mechanisms of action and implementation of ESAS. We will recruit 36 schools across Scotland. The evaluation comprises three linked studies:Study 1: Pragmatic cluster randomised trial with 1:1 school allocation to either immediate ESAS intervention start (intervention schools) or 12-month delayed intervention start (control schools). Our primary outcome of student experience of sexual harassment will be measured at 12 months post-randomisation. Analysis of primary and secondary outcomes (student and school level) will be conducted on an intention to treat (ITT) basis comparing schools according to their original allocation.Study 2: Mixed-methods evaluation. Study 2A: Longitudinal follow-up will assess primary, secondary and intermediate outcomes at baseline, 12 months and 24 months of follow-up. Study 2B: Systems and realist-informed process evaluation will assess intervention and control school context, fidelity, dose and reach, acceptability and actor response, and how this varies by school and students. We will also assess implementation processes and mechanisms of action (beneficial or harmful), including if and how change is embedded over time, and if and how ESAS helps schools leverage other assets and resources.Study 3: Economic evaluation to assess the within-trial and longer term cost-effectiveness of ESAS.The methods include surveys in three out of six year groups (Years 2, 4 and 6) in all schools (baseline, 12 months and 24 months of follow-up); interviews with staff, students and other stakeholders; activity observations; brief surveys with key actors and analysis of trial documentation.

ETHICS AND DISSEMINATION

Ethical approval by University of Glasgow MVLS Ethics Committee (200220268). Findings will be disseminated via multiple channels to researchers, GBV and education sector stakeholders, study participants and the public.

TRIAL REGISTRATION NUMBER

ISRCTN29792495.

Interventions that challenge established and accepted clinical practice: lessons learnt from a process evaluation of the STOP-APE trial

Background

Developing and implementing interventions that change clinical practice can be challenging and complex. Such interventions can be particularly difficult when attempting to change established behaviours and practices. While extensive literature on implementation of interventions that focus on changing clinical practice exists, understanding of the difficulties involved in implementing interventions that go against accepted clinical practice is limited.

Objectives

To describe the challenges involved in delivering a complex intervention that goes against established clinical practice, using a clinical trial assessing the balance of benefits and risks of withholding anticoagulation for subsegmental pulmonary embolism as an example.

Design and methods

This study draws from a process evaluation conducted as part of a clinical trial. The evaluation utilised semistructured interviews with patients and clinicians during the trial's internal pilot phase to investigate the acceptability of withholding anticoagulant medication and participants' experiences within the trial. The data were analysed using the framework method.

Setting and participants

Eight patients with subsegmental pulmonary embolism (six females and two males) and three acute care clinicians (two males and one female) from five trial sites were interviewed.

Results

Our findings indicated that factors such as clinician equipoise, discomfort with certain patient characteristics, and effective patient communication are closely connected and significantly impact both the process of changing clinical practice and the conduct of the trial. Clinicians faced difficulties in approaching eligible patients for trial participation, especially when a diagnosis and treatment plan had already been provided by another clinician. The tension between maintaining clinical equipoise and addressing the needs of unwell patients further complicated decisions, particularly when withholding anticoagulation in those with severe symptoms or multiple comorbidities. Communication about the risks and benefits of non-medication strategies for pulmonary embolism was also challenging, with concerns about undermining patient trust. Patients, on the other hand, expressed considerable anxiety about not receiving anticoagulants, with their perspectives on study participation and treatment heavily influenced by their prior health experiences and ongoing medical conditions. The active involvement of clinicians in the consent process had a positive effect on patients' perceptions and experiences, with many feeling reassured in knowing they could contact clinical staff if needed.

Limitations

Small sample size of patients and clinicians across limited study sites; single method of data collection.

Conclusions

Our results highlight the multifaceted challenges when attempting to conduct studies that challenge accepted practices and norms. These complexities are deeply intertwined, influencing both clinical decision-making and patient recruitment for those studies.

Future work

Future research should focus on developing strategies to help clinicians maintain equipoise and communicate the risks and benefits of interventions, while also deepening the understanding of patients' experiences and perceptions to enhance recruitment strategies.

Ethical approval

Wales REC 6, Reference: 20/WA/0256, approved 30 September 2020.

Funding

This article presents independent research funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme as award number NIHR128073.

Using group model building to frame the commercial determinants of dietary behaviour in adolescence - findings from online system mapping workshops with adolescents, policymakers and public health practitioners in the Southwest of England

BACKGROUND

In England, 23% of children aged 11 start their teenage years living with obesity. An adolescent living with obesity is five times more likely to live with obesity in adult life. There is limited research and policy incorporating adolescents' views on how they experience the commercial determinants of dietary behaviour and obesity, which misses an opportunity to improve services and policies that aim to influence the prevalence of childhood obesity. This study reports the findings from online Group Model Building system mapping workshops in which we explored the mechanisms by which commercial drivers influence adolescents' dietary behaviour.

METHODS

We ran a series of 3 online Group Model Building workshops with adolescents and one Group Model Building workshop with policymakers and public health practitioners. Adolescents portrayed their views on how food and beverage industries influence what they choose to buy and eat in a system map, and then proposed a set of policy actions to promote healthier food environments. We shared the system map created by adolescents with policymakers and public health practitioners to reflect on how current policy interventions match adolescents' views on the most influential factors.

RESULTS

The system map contains 37 elements connected by 70 hypothesised causal links and five feedback loops. These elements were grouped into six themes that portray the complexity of factors that influence adolescents' food choices in their physical and digital environments, disproportionately encouraging the consumption of unhealthy products. Policymakers and public health practitioners reflected on the power and the deep level of influence food companies exert on adolescents' behaviour. They recognised that the coexisting influence of food marketing and social media on mental health and body image is not well reflected in current policy and research efforts.

CONCLUSIONS

This study highlights the need for public health policymaking processes to provide youth with a space to voice influential elements and consequences, thereby co-creating policies and designing interventions to buffer risk factors and increase well-being in this critical transitional stage.

Service evaluation of 'GP at Door' of accident and emergency services in Eastern England

AIM

We describe activity, outcomes, and benefits after streaming low urgency attenders to General practice services at Door of Accident and Emergency departments (GDAE).

BACKGROUND

Many attendances to A&Es are for non-urgent health problems that could be better met by primary care rather than urgent care clinicians. It is valuable to monitor service activity, outcomes, service user demographics, and potential benefits when primary care is co-located with A&E departments.

METHODS

As a service evaluation, we describe and analyse GDAE users, reasons for presentation, wait times, outcomes, and co-located A&E wait times at two hospitals in eastern England. Distributions of outcomes, wait times, reasons for attendance, deprivation, and age groups were compared for GDAE and usual A&E attenders at each site using Pearson chi-square tests and accelerated time failure modelling. Performance in a four-hour key performance indicator was descriptively compared for co-located and similar emergency departments.

FINDINGS

Each GDAE saw about 1025 patients per month. Wait times for usual accident and emergency (A&E) care are relatively short at only one site. Reattendances were common (about 11% of unique patients), 75% of GDAE attenders were seen within 1 hour of arrival, 7% of patients initially allocated to GDAE were referred back to A&E for further investigations, and 59% of GDAE patients were treated and discharged with no further treatment or referral required. Pain, injury, infection, or feeling generally unwell each comprised > 10% of primary reasons for attendance. At James Paget University Hospital, 4.3%, and at Queen Elizabeth Hospital, 16.1% of GDAE attendances led to referral to specialist health services. GDAE attenders were younger and more socially deprived than attenders to co-located A&Es. Patients were seen quickly at both GDAE sites, but there were differences in counts of specialist referrals and wait times. Process evaluation could illuminate reasons for differences between study sites.

Analysis of how a complex systems perspective is applied in studies on socioeconomic inequalities in health and health behaviour-a call for reporting guidelines

BACKGROUND

A complex systems perspective is gaining popularity in research on socioeconomic inequalities in health and health behaviour, though there may be a gap between its popularity and the way it is implemented. Building on our recent systematic scoping review, we aim to analyse the application of and reporting on complex systems methods in the literature on socioeconomic inequalities in health and health behaviour.

METHODS

Selected methods and results from the review are presented as a basis for in-depth critical reflection. A traffic light-based instrument was used to assess the extent to which eight key concepts of a complex systems perspective (e.g. feedback loops) were applied. Study characteristics related to the applied value of the models were also extracted, including the model evidence base, the depiction of the model structure, and which characteristics of model relationships (e.g. polarity) were reported on.

RESULTS

Studies that applied more key concepts of a complex systems perspective were also more likely to report the direction and polarity of relationships. The system paradigm, its deepest held beliefs, is seldom identified but may be key to recognize when designing interventions. A clear, complete depiction of the full model structure is also needed to convey the functioning of a complex system. We recommend that authors include these characteristics and level of detail in their reporting.

CONCLUSIONS

Above all, we call for the development of reporting guidelines to increase the transparency and applied value of complex systems models on socioeconomic inequalities in health, health behaviour and beyond.

The PET@home Toolkit: A Process Evaluation Study

The Dutch PET@home Toolkit was developed to mitigate challenges, foster conversations, and support planning concerning pets in long-term care at home. This process evaluation study aimed to evaluate the practical application of the toolkit on four topics: (1) satisfaction; (2) relevance; (3) feasibility; and (4) integration of the PET@home Toolkit materials in home care. Outcomes may be used to improve the toolkit materials to better align with the long-term care at home setting. Accounting for data saturation, professional caregivers (N = 6), clients (N = 2), and family caregivers (N = 2) who used toolkit materials participated in semi-structured interviews. Interviews were analyzed by two researchers in ATLAS.ti using an inductive-iterative approach. The researchers reached a consensus on themes and clustering within interview topics. This led to the identification of the following themes pertaining to each interview topic: satisfaction (general impression, suggestions for improvement); relevance (awareness, planning, pet-related aspects in practice, impact on healthcare quality); feasibility (healthcare practice, competence, quantity); and implementation (digitalization, task owner, piloting, timing). Several improvements were made to toolkit materials, such as providing clearer instructions for clients in the information booklet. Participants acknowledged the toolkit could lead to better planning while mitigating potential challenges concerning pets, which may lead to longer-lasting relationships between clients and their pets.

Health and wellbeing of survivors of sexual violence and abuse attending sexual assault referral centres in England: the MESARCH mixed-methods evaluation

Background

One million people in England and Wales experience sexual violence and abuse each year, with nearly half experiencing serious sexual offences; around 30,000 survivors access sexual assault referral centres.

Objectives

This research was commissioned by National Institute for Health and Care Research to evaluate access, interventions and care pathways for survivors, especially those provided through sexual assault referral centres.

Design, setting, participants

The sexual assault referral centres care pathway was investigated through six sub-studies. There were two Cochrane Reviews (4274 participants). Seventy-two providers and 5 survivors were interviewed at eight sites; the children and young people study involved 12 participants from two sexual assault referral centres. A cohort study involving three-wave data collection over 1 year (21 sites; 2602 service users screened, 337 recruited) used a multilevel modelling framework to explore risk factors for burden of post-traumatic stress disorder symptoms at baseline and change at 1 year. We analysed costs and outcomes and conducted a narrative analysis (41 survivors). We worked closely with survivors and prioritised the safety/welfare of participants and researchers.

Results

Cochrane Reviews identified large effects from psychosocial interventions for post-traumatic stress disorder and depression. Sexual assault referral centres delivered a high-quality frontline service for survivors but groups experiencing domestic abuse and some ethnic and cultural minorities were under-represented. The qualitative research emphasised inter-agency collaboration for survivor benefit. The cohort study identified a risk 'triad' of adverse childhood experiences, poor mental health and economic deprivation, which was associated with baseline trauma burden. There were important improvements in trauma symptoms a year later. These improvements were unrelated to different sexual assault referral centre models. Costs and other outcomes were also similar across models. Harmful policing and justice practices/procedures were identified by 25% of participants. In this context, trauma-competent interviewing techniques, regular/timely updates and conveying case decisions with care signalled good practice.

Limitations

The cohort study lacked a comparison group, reducing confidence in the finding that access to sexual assault referral centres explained the reduction observed in post-traumatic stress disorder.

Conclusions and future work

Barriers to access call for concerted efforts to implement trauma-informed universal health services. The risk 'triad' underscores the value of holistic approaches to care at sexual assault referral centres and timely follow-on care. Poor mental health was the main barrier to service access beyond sexual assault referral centres. The persistence of trauma symptoms a year after accessing sexual assault referral centres signals urgent need for tackling counselling wait-lists, expanding support options and commitment to lifelong care. Multidisciplinary evaluation of sexual assault referral centres for better health provides a foundation for advancing trauma-informed practices in the context of sexual violence and abuse.

Study registration

This study is registered as ISRCTN30846825 https://doi.org/10.1186/ISRCTN30846825.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: 16/117/04) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 35. See the NIHR Funding and Awards website for further award information.

Implementation of health-promoting retail initiatives in the Healthier Choices in Supermarkets Study-qualitative perspectives from a feasibility study

BACKGROUND

Improving food environments like supermarkets has the potential to affect customers' health positively. Scholars suggest researchers and retailers collaborate closely on implementing and testing such health-promoting interventions, but knowledge of the implementation of such interventions is limited. We explore the implementation of four health-promoting food retail initiatives selected and developed by a partnership between a research institution, a large retail group, and a non-governmental organisation.

METHODS

The four initiatives included downsizing of bags for pick'n' mix sweets and soda bottles at the check-out registers, shelf tags promoting healthier breakfast cereal options, and replacing a complimentary bun with a banana offered to children. The initiatives were implemented for 6 weeks (or longer if the store manager allowed it) in one store in Copenhagen, Denmark. Data were collected through observations, informal interviews with customers, and semi-structured interviews with retailers. We conducted a thematic analysis of transcripts and field notes inspired by process evaluation concepts and included quantitative summaries of selected data.

RESULTS

Two out of four initiatives were not implemented as intended. The implementation was delayed due to delivery issues, which also resulted in soda bottles not being downsized as intended. The maintenance of the shelf tags decreased over time. Retailers expressed different levels of acceptability towards the initiatives, with a preference for the complimentary banana for children. This was also the only initiative noticed by customers with both positive and negative responses. Barriers and facilitators of implementation fell into three themes: Health is not the number one priority, general capacity of retailers, and influence of customers and other stakeholders on store operation.

CONCLUSIONS

The retailers' interests, priorities, and general capacity influenced the initiative implementation. Retailers' acceptability of the initiatives was mixed despite their involvement in the pre-intervention phase. Our study also suggests that customer responses towards health-promoting initiatives, as well as cooperation with suppliers and manufacturers in the development phase, may be determining to successful implementation. Future studies should explore strategies to facilitate implementation, which can be applied prior to and during the intervention.

Healthcare quality improvement: It's time to update the Donabedian approach with a complex systems perspective

There is broad consensus that healthcare systems are complex systems, which as a result, face complex problems. From this perspective, quality of care can be conceptualised as an emergent outcome of the healthcare system, that is more than the sum of individual components of care (in terms of inputs and processes), and quality improvement as a complex systems problem. However, traditional approaches, such as Donabedian's structure/process/outcome framework, are rooted in a linear, reductionist perspective, that fails to recognise that quality of care is created in the context of complex healthcare systems, and the many interactions and uncertainties at play that shape quality of care and health outcomes. A paradigm shift is needed from a reductionist to a systems thinking approach if we are to better understand and improve quality of care. Such a shift begins with asking different research questions, situated within the system context, that focus on identifying how interventions may contribute to system improvement, as opposed to seeking to directly link interventions with quality of care outcomes. In contrast to traditional healthcare quality measures focusing on single components of the system in isolation, research needs to explicitly consider quality of care as an emergent system outcome and identify new indicators and methods of assessment that provide insight into how the healthcare system functions as an interconnected whole. It is an opportune moment to harness the energy of the international healthcare quality movement to drive the innovation needed in research and practice to adopt a systems thinking approach.

Mapping the ripple effects of a compassionate university for serious illness, death, and bereavement

Background

Compassionate communities have been put forward as a promising model for community-based support for people facing serious illness, caregiving, dying, and loss. In particular, educational institutions are increasingly acknowledged as potential settings to function as compassionate schools and compassionate workplaces, cultivating acceptance and validation of these experiences beyond the university setting.

Objectives

This paper investigates the activities and outcomes of a compassionate community initiative-the Compassionate University program at the Vrije Universiteit Brussel in Belgium.

Design

Ripple Effects Mapping was used to guide the focus group and individual interviews conducted with core team members responsible for the development and implementation of the Compassionate University program.

Methods

During the focus group and individual interviews, the core team members reflected on the program contributions, with their narratives visually depicted via a hand-drawn mind map. Qualitative data derived from this mind map were entered into XMIND mapping software and fine-tuned based on the focus group and individual interview transcripts and additional project records.

Results

Thematic analysis identified four outcome areas that encapsulate the key contributions of the Compassionate University program: (i) increased acceptance and integration of topics such as serious illness, death, and bereavement into existing practices; (ii) broader support for and formalization of compassionate procedures and policies; (iii) emergence of informal networks and internal collaboration on the topics; and (iv) diffusion of compassionate ideas beyond the university.

Conclusion

The Compassionate University program facilitates a cultural shift within the university environment, fostering greater acceptance of integrating topics such as serious illness, death, and bereavement into existing practices. Additionally, compassionate procedures and policies for students and staff have been formalized, and core team members are increasingly called upon to provide support on these matters. Notably, Compassionate University stands out as one of the pioneering initiatives in Europe, attracting different educational institutions seeking guidance on cultivating a more compassionate environment.

How can health systems approach reducing health inequalities? An in-depth qualitative case study in the UK

BACKGROUND

Addressing socioeconomic inequalities in health and healthcare, and reducing avoidable hospital admissions requires integrated strategy and complex intervention across health systems. However, the understanding of how to create effective systems to reduce socio-economic inequalities in health and healthcare is limited. The aim was to explore and develop a system's level understanding of how local areas address health inequalities with a focus on avoidable emergency admissions.

METHODS

In-depth case study using qualitative investigation (documentary analysis and key informant interviews) in an urban UK local authority. Interviewees were identified using snowball sampling. Documents were retrieved via key informants and web searches of relevant organisations. Interviews and documents were analysed independently based on a thematic analysis approach.

RESULTS

Interviews (n = 14) with wide representation from local authority (n = 8), NHS (n = 5) and voluntary, community and social enterprise (VCSE) sector (n = 1) with 75 documents (including from NHS, local authority, VCSE) were included. Cross-referenced themes were understanding the local context, facilitators of how to tackle health inequalities: the assets, and emerging risks and concerns. Addressing health inequalities in avoidable admissions per se was not often explicitly linked by either the interviews or documents and is not yet embedded into practice. However, a strong coherent strategic integrated population health management plan with a system's approach to reducing health inequalities was evident as was collective action and involving people, with links to a "strong third sector". Challenges reported include structural barriers and threats, the analysis and accessibility of data as well as ongoing pressures on the health and care system.

CONCLUSION

We provide an in-depth exploration of how a local area is working to address health and care inequalities. Key elements of this system's working include fostering strategic coherence, cross-agency working, and community-asset based approaches. Areas requiring action included data sharing challenges across organisations and analytical capacity to assist endeavours to reduce health and care inequalities. Other areas were around the resilience of the system including the recruitment and retention of the workforce. More action is required to embed reducing health inequalities in avoidable admissions explicitly in local areas with inaction risking widening the health gap.

A decade of change towards Value-Based Health Care at a Dutch University Hospital: a complexity-informed process study

BACKGROUND

While healthcare organizations in several countries are embracing Value-Based Health Care (VBHC), there are limited insights into how to achieve this paradigm shift. This study examines the decade-long (2012-2023) change towards VBHC in a pioneering Dutch university hospital.

METHOD

Through retrospective, complexity-informed process research, we study how a Dutch university hospital's strategy to implement VBHC evolved, how implementation outcomes unfolded, and the underlying logic behind these developments. Data include the hospital's internal documents (n = 10,536), implementation outcome indicators (n = 4), a survey among clinicians (n = 47), and interviews with individuals contributing to VBHC at the hospital level (n = 20).

RESULTS

The change towards VBHC is characterized by three sequential strategies. Initially, the focus was on deep change through local, tailored implementation of multiple VBHC elements. The strategy then transitioned to a hospital-wide program aimed at evolutionary change on a large scale, emphasizing the integration of VBHC into mainstream IT and policies. Recognizing the advantages and limitations of both strategies, the hospital currently adopts a "hybrid" strategy. This strategy delicately combines deep and broad change efforts. The strategy evolved based on accumulated insights, contextual developments and shifts in decision-makers. The complexity of change was downplayed in plans and stakeholder communication. By the end of 2023, 68 (sub)departments engaged in VBHC, enabled to discuss patients' responses to Patient Reported Outcomes Measures (PROMs) during outpatient care. However, clinicians' use of PROMs data showed limitations. While pioneers delved deeper into VBHC, laggards have yet to initiate it.

CONCLUSIONS

VBHC does not lend itself to linear planning and is not easily scalable. While there appears to be no golden standard for implementation, blending local and larger-scale actions appears advantageous. Local, deep yet harmonized and system-integrated changes culminate in large scale transformation. Embracing complexity and focusing on the ultimate aims of (re)institutionalization and (re)professionalization are crucial.

Applying dynamic systems theory and complexity theory methods in psychotherapy research: A systematic literature review

OBJECTIVE

Dynamic systems theory and complexity theory (DST/CT) is a framework explaining how complex systems change and adapt over time. In psychotherapy, DST/CT can be used to understand how a person's mental and emotional state changes during therapy incorporating higher levels of complexity. This study aimed to systematically review the variability of DST/CT methods applied in psychotherapy research.

METHODS

A primary studies search was conducted in the EBSCO and Web of Knowledge databases, extracting information about the analyzed DST/CT phenomena, employed mathematical methods to investigate these phenomena, descriptions of specified dynamic models, psychotherapy phenomena, and other information regarding studies with empirical data (e.g., measurement granularity).

RESULTS

After screening 38,216 abstracts and 4,194 full texts, N = 41 studies published from 1990 to 2021 were identified. The employed methods typically included measures of dynamic complexity or chaoticity. Computational and simulation studies most often employed first-order ordinary differential equations and typically focused on describing the time evolution of client-therapist dyadic influences. Eligible studies with empirical data were usually based on case studies and focused on data with high time intensity of within-session dynamics.

CONCLUSION

This review provides a descriptive synthesis of the current state of the proliferation of DST/CT methods in the psychotherapy research field.

A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance

The UK Medical Research Council's widely used guidance for developing and evaluating complex interventions has been replaced by a new framework, commissioned jointly by the Medical Research Council and the National Institute for Health Research, which takes account of recent developments in theory and methods and the need to maximise the efficiency, use, and impact of research.

Navigating One Health in research-for-development: Reflections on the design and implementation of the CGIAR Initiative on One Health

Adopting One Health approaches is key for addressing interconnected health challenges. Yet, how to best put One Health into practice in research-for-development initiatives aiming to 'deliver impacts' remains unclear. Drawing on the CGIAR Initiative on One Health - a global initiative to address zoonotic diseases, antimicrobial resistance, and food and water safety - we reflect on challenges during program conception and implementation, prompting us to suggest improvements in multisectoral collaboration, coordination, and communication. Our approach involves conducting a researcher-centered process evaluation, comprising individual interviews that are subsequently thematically analyzed and synthesized. The key takeaway is that limited time for planning processes and short program timelines compared to envisioned development impacts may impede research-for-development efforts. Yet, collaborative work can be successful when adequate time and resources are allocated for planning with minimal disruption throughout implementation. Additionally, due to the multifaceted nature of One Health initiatives, it is important to pay attention to co-benefits and trade-offs, where taking action in one aspect may yield advantages and disadvantages in another, aiding to identify sustainable One Health development pathways. Forming close partnerships with national governments and local stakeholders is essential not only to promote sustainability but also to ensure local relevance, enhancing the potential for meaningful impact. Finally, regularly assessing progress toward development goals is critical as development stands as an overarching objective.

Interventions for improving critical care in low- and middle-income countries: a systematic review

PURPOSE

To systematically review the typology, impact, quality of evidence, barriers, and facilitators to implementation of Quality Improvement (QI) interventions for adult critical care in low- and middle-income countries (LMICs).

METHODS

MEDLINE, EMBASE, Cochrane Library and ClinicalTrials.gov were searched on 1st September 2022. The studies were included if they described the implementation of QI interventions for adult critical care in LMICs, available as full text, in English and published after 2000. The risks of bias were assessed using the ROB 2.0/ROBINS-I tools. Intervention strategies were categorised according to a Knowledge Translation framework. Interventions' effectiveness were synthesised by vote counting and assessed with a binomial test. Barriers and facilitators to implementation were narratively synthesised using the Consolidated Framework for Implementation Research.

RESULTS

78 studies were included. Risk of bias was high. The most common intervention strategies were Education, Audit & Feedback (A&F) and Protocols/Guidelines/Bundles/Checklists (PGBC). Two multifaceted strategies improved both process and outcome measures: Education and A&F (p = 0.008); and PGBC with Education and A&F (p = 0.001, p < 0.001). Facilitators to implementation were stakeholder engagement, organisational readiness for implementation, and adaptability of interventions. Barriers were lack of resources and incompatibility with clinical workflows.

CONCLUSIONS

The evidence for QI in critical care in LMICs is sparse and at high risk of bias but suggests that multifaceted interventions are most effective. Co-designing interventions with and engaging stakeholders, communicating relative advantages, employing local champions and adapting to feedback can improve implementation. Hybrid study designs, process evaluations and adherence to reporting guidelines would improve the evidence base.

The Healthy Start scheme in England "is a lifeline for families but many are missing out": a rapid qualitative analysis

BACKGROUND

Healthy Start (HS) is a government scheme in England, Wales and Northern Ireland that offers a financial payment card and free vitamins to families experiencing low income. Pregnant women and families with children < 4 years can use the HS card to buy fruit, vegetables, cow's milk, infant formula and pulses. HS was fully digitalised in March 2022. While digitalisation has improved the user experience for many families, in the context of the cost-of-living crisis and increasing dietary inequalities, it is important to understand why HS is not reaching more families. This study aimed to (i) assess the perceptions and experiences of HS from stakeholders across the system including those who promote, implement and are eligible for HS, and (ii) identify recommendations to improve the scheme's effectiveness and uptake.

METHODS

The study design was a post-implementation rapid qualitative evaluation using stakeholder interviews. Data were collected between January and June 2023 via semi-structured interviews (50% online; 50% in person) with 112 stakeholders, including parents (n = 59), non-government organisations (n = 13), retailers (n = 11) and health and community professionals (n = 29) at national and local levels. Findings were confirmed by a sub-sample of participants.

RESULTS

Six core themes cut across stakeholders' perceptions and experiences, and stakeholders collectively outlined seven recommendations they felt could be acted upon to maximise uptake and efficiency of HS, with actions at both national and local levels. A novel finding from this study is that raising awareness about HS alone is unlikely to result automatically or universally in higher uptake rate. Recommendations include: continuing to provide this scheme that is universally valued; the need for many families to be provided with a helping hand to successfully complete the application; reframing of the scheme as a child's right to food and development to ensure inclusivity; improved leadership, coordination and accountability at both national and local levels.

CONCLUSIONS

HS provides benefits for child development and family wellbeing. The study's recommendations should be actioned by national and local governments to enable all families eligible for the scheme to benefit from this nutritional safety net.

Effectiveness and cost-effectiveness of a sustainable obesity prevention programme for preschool children delivered at scale 'HENRY' (Health, Exercise, Nutrition for the Really Young): protocol for the HENRY III cluster randomised controlled trial

INTRODUCTION

One-fifth of children start school already overweight or living with obesity, with rates disproportionately impacting those living in the most deprived areas. Social, environmental and biological factors contribute to excess weight gain and programmes delivered in early years settings aim to support families to navigate these in order to prevent obesity. One of these programmes (Health, Exercise and Nutrition for the Really Young, HENRY) has been delivered in UK community venues (hereon named 'centres') in high deprivation areas since 2008 and aims to help families to provide a healthy start for their preschool children. We aim to establish the effectiveness and cost-effectiveness of HENRY, including its potential role from a wider systems perspective.

METHODS AND ANALYSIS

This is a multicentre, open-labelled, two-group, prospective, cluster randomised controlled trial, with cost-effectiveness analysis, systems-based process evaluation and internal pilot. Primary analysis will compare body mass index (BMI) z-score at 12 months in children (n=984) whose parents have attended HENRY to those who have not attended. Secondary outcomes include parent and staff BMI and waist circumference, parenting efficacy, feeding, eating habits, quality of life, resource use and medium term (3 years) BMI z-scores (child and siblings). 82 centres in ~14 local authority areas will be randomised (1:1) to receive HENRY or continue with standard practice. Intention-to-treat analysis will compare outcomes using mixed effects linear regression. Economic evaluation will estimate a within-trial calculation of cost-per unit change in BMI z-score and longer-term trajectories to determine lifelong cost savings (long-term outcomes). A systems process evaluation will explore whether (and how) implementation of HENRY impacts (and is impacted by) the early years obesity system. An established parent advisory group will support delivery and dissemination.

ETHICS AND DISSEMINATION

Ethical approval has been granted by the University of York, Health Sciences' Research Governance Committee (HSRGC/2022/537/E). Dissemination includes policy reports, community resources, social media and academic outputs.

TRIAL REGISTRATION NUMBER

ISRCTN16529380.

Process evaluation of the Belgian one-month-without alcohol campaign 'Tournée Minérale': a mixed method approach

BACKGROUND

The Tournée Minérale campaign [TMC] is a mass media prevention campaign challenging Belgian adults to refrain from alcohol during one month. A process evaluation may help us better understand the effect of TMC and to formulate recommendations for future editions. The current study aimed to examine reach, experiences, perceived effectiveness and maintenance of TMC.

METHODS

A mixed method design was used to assess the process, using pre- and post-questionnaires (n = 49.022, 44.5 ± 13.1 years old, 37.0% men) and focus groups (n = 31, 47.3 ± 14.3 years old, 33.3% men).

RESULTS

Most campaign materials were considered useful and/or motivating. Facilitators for taking part in TMC were connectedness with other participants, stimulus control (e.g. removing alcohol at home) and a supportive social environment. Most difficulties were encountered with abstaining during social occasions as participants had to change a habit or find alternative non-alcoholic beverages. Participants reported both beneficial (e.g. sleeping better) and adverse effects (e.g. drinking more soft drinks).

CONCLUSIONS

Future editions of TMC should try to decrease perceived adverse effects (e.g. by providing attractive non-alcoholic beverages) and can benefit from having a forum where people can share experiences.

The current state of complex systems research on socioeconomic inequalities in health and health behavior-a systematic scoping review

BACKGROUND

Interest in applying a complex systems approach to understanding socioeconomic inequalities in health is growing, but an overview of existing research on this topic is lacking. In this systematic scoping review, we summarize the current state of the literature, identify shared drivers of multiple health and health behavior outcomes, and highlight areas ripe for future research.

METHODS

SCOPUS, Web of Science, and PubMed databases were searched in April 2023 for peer-reviewed, English-language studies in high-income OECD countries containing a conceptual systems model or simulation model of socioeconomic inequalities in health or health behavior in the adult general population. Two independent reviewers screened abstracts and full texts. Data on study aim, type of model, all model elements, and all relationships were extracted. Model elements were categorized based on the Commission on Social Determinants of Health framework, and relationships between grouped elements were visualized in a summary conceptual systems map.

RESULTS

A total of 42 publications were included; 18 only contained a simulation model, 20 only contained a conceptual model, and 4 contained both types of models. General health outcomes (e.g., health status, well-being) were modeled more often than specific outcomes like obesity. Dietary behavior and physical activity were by far the most commonly modeled health behaviors. Intermediary determinants of health (e.g., material circumstances, social cohesion) were included in nearly all models, whereas structural determinants (e.g., policies, societal values) were included in about a third of models. Using the summary conceptual systems map, we identified 15 shared drivers of socioeconomic inequalities in multiple health and health behavior outcomes.

CONCLUSIONS

The interconnectedness of socioeconomic position, multiple health and health behavior outcomes, and determinants of socioeconomic inequalities in health is clear from this review. Factors central to the complex system as it is currently understood in the literature (e.g., financial strain) may be both efficient and effective policy levers, and factors less well represented in the literature (e.g., sleep, structural determinants) may warrant more research. Our systematic, comprehensive synthesis of the literature may serve as a basis for, among other things, a complex systems framework for socioeconomic inequalities in health.

Process evaluation of school-based high-intensity interval training interventions for children and adolescents: a systematic review and meta-analysis of randomized controlled trials

BACKGROUND

Several systematic reviews have been published to investigate the effectiveness of high-intensity interval training (HIIT) in schools. However, there has been limited attention given to understanding the functioning of the intervention processes, which is of paramount importance for interpreting and translating the intervention effectiveness. The aim of this systematic review is to determine the extent to which process evaluation is measured in school-based HIIT interventions and to explore the effects of process evaluation and intervention characteristics on cardiorespiratory fitness (CRF), body composition, muscular strength, and blood pressure.

METHODS

A comprehensive search was conducted in SPORT Discus (EBSCOhost), Web of Science, Scopus, Medline (Ovid) and Cochrane Central Register of Controlled Trials. The extent to which process evaluation is measured was narratively reported, alongside with the guidance of process evaluation of complex interventions by UK Medical Research Council. Meta-analyses and meta-regressions were conducted to determine the effects of process evaluation and intervention characteristics to the intervention outcomes.

RESULTS

The literature search identified 77 studies reporting on 45 school-based HIIT interventions. In total, five interventions reported process evaluation in a section or in a separate study, and only one intervention adopted a process evaluation framework. On average, 6 out of 12 process evaluation measures were reported in all interventions. Subgroup analyses did not indicate any beneficial treatment effects for studies with process evaluation group, whereas all pooled data and studies without process evaluation group showed significant improvement for CRF and body composition.

CONCLUSION

Process evaluation is frequently omitted in the literature of school-based HIIT in children and adolescents. Although reporting of process evaluation measures may not directly associate with better intervention outcomes, it allows accurate interpretation of intervention outcomes, thereby enhancing the generalisability and dissemination of the interventions.

Real-life experiments in supermarkets to encourage healthy dietary-related behaviours: opportunities, challenges and lessons learned

BACKGROUND

Supermarkets are the primary source of food for many people yet their full potential as a setting to encourage healthy dietary-related behaviours remains underutilised. Sharing the experiences from research groups who have worked with supermarket chains to evaluate strategies that promote healthy eating could improve the efficiency of building such relationships and enhance the design quality of future research studies.

METHODS

A collective case study approach was used to synthesise experiences of engaging and sustaining research collaborations with national supermarket chains to test the effectiveness of health-focused in-store interventions. The collective narrative covers studies conducted in three high-income countries: Australia, the Netherlands and the United Kingdom.

RESULTS

We have distilled our experiences and lessons learned into six recommendations for conducting high quality public health research with commercial supermarket chains. These include: (i) using personal contacts, knowledge of supermarket activities and engaging executive management to establish a partnership and allowing time to build trust; (ii) using scientifically robust study designs with appropriate sample size calculations; (iii) formalising data exchange arrangements and allocating adequate resource for data extraction and re-categorisation; (iv) assessing effects at individual/households level where possible; (v) designing a mixed-methods process evaluation to measure intervention fidelity, dose and unintended consequences; and (vi) ensuring scientific independence through formal contract agreements.

CONCLUSIONS

Our collective experiences of working in non-financial partnerships with national supermarket chains could be useful for other research groups looking to develop and implement supermarket studies in an efficient manner. Further evidence from real-life supermarket interventions is necessary to identify sustainable strategies that can improve population diet and maintain necessary commercial outcomes.

Meta-evaluation of a whole systems programme, ActEarly: A study protocol

INTRODUCTION

Living in an area with high levels of child poverty predisposes children to poorer mental and physical health. ActEarly is a 5-year research programme that comprises a large number of interventions (>20) with citizen science and co-production embedded. It aims to improve the health and well-being of children and families living in two areas of the UK with high levels of deprivation; Bradford in West Yorkshire, and the London Borough of Tower Hamlets. This protocol outlines the meta-evaluation (an evaluation of evaluations) of the ActEarly programme from a systems perspective, where individual interventions are viewed as events in the wider policy system across the two geographical areas. It includes investigating the programme's impact on early life health and well-being outcomes, interdisciplinary prevention research collaboration and capacity building, and local and national decision making.

METHODS

The ActEarly meta-evaluation will follow and adapt the five iterative stages of the 'Evaluation of Programmes in Complex Adaptive Systems' (ENCOMPASS) framework for evaluation of public health programmes in complex adaptive systems. Theory-based and mixed-methods approaches will be used to investigate the fidelity of the ActEarly research programme, and whether, why and how ActEarly contributes to changes in the policy system, and whether alternative explanations can be ruled out. Ripple effects and systems mapping will be used to explore the relationships between interventions and their outcomes, and the degree to which the ActEarly programme encouraged interdisciplinary and prevention research collaboration as intended. A computer simulation model ("LifeSim") will also be used to evaluate the scale of the potential long-term benefits of cross-sectoral action to tackle the financial, educational and health disadvantages faced by children in Bradford and Tower Hamlets. Together, these approaches will be used to evaluate ActEarly's dynamic programme outputs at different system levels and measure the programme's system changes on early life health and well-being.

DISCUSSION

This meta-evaluation protocol presents our plans for using and adapting the ENCOMPASS framework to evaluate the system-wide impact of the early life health and well-being programme, ActEarly. Due to the collaborative and non-linear nature of the work, we reserve the option to change and query some of our evaluation choices based on the feedback we receive from stakeholders to ensure that our evaluation remains relevant and fit for purpose.

Knowledge exchange in crisis settings: A scoping review

BACKGROUND

Public health practice and efforts to improve the social determinants of health operate within a climate characterised by multiple and intersecting crises. This includes the Covid-19 pandemic as well as more protracted crises such as climate change and persistent social inequalities that impact health. We sought to understand and compare how knowledge exchange (KE) processes occur across different crises, and how knowledge on improving social determinants of health can be utilised at times of crisis to reduce health inequalities and strengthen public systems.

METHODS

We conducted a scoping review to understand how KE on improving social determinants of health can occur across different types of crises (e.g. environmental, pandemics, humanitarian). Relevant studies were identified through electronic searching of Medline, EMBASE, Global Health, Scopus and Web of Science databases.

RESULTS

We identified 86 studies for inclusion in the review. Most studies concerned pandemic or environmental crises. Fewer studies explored KE during technical (e.g. nuclear), terror-related or humanitarian crises. This may reflect a limitation of the searches. Few studies assessed KE as part of longer-term responses to social and economic impacts of crises, with studies more likely to focus on immediate response or early recovery stages. Exchange of research evidence or data with policy or practice contextual knowledge was common but there was variation in the extent that lay (public) knowledge was included as part of KE processes.

CONCLUSION

As ongoing crises continue with significant public health implications, KE processes should appropriately reflect the complexity inherent in crises and foreground health inequalities. Doing so could include the utilisation of systems or complexity-informed methods to support planning and evaluation of KE, a greater focus on KE to support action to address social determinants of health, and the inclusion of a plurality of knowledge-including lived experience-in planning and responding to crises.

Child and family experiences of a whole-systems approach to physical activity in a multiethnic UK city: a citizen science evaluation protocol

INTRODUCTION

Whole-systems approaches are being adopted to tackle physical inactivity. The mechanisms contributing to changes resulting from whole-systems approaches are not fully understood. The voices of children and families that these approaches are designed for need to be heard to understand what is working, for whom, where and in what context. This paper describes the protocol for the children and families' citizen science evaluation of the Join Us: Move, Play (JU:MP) programme, a whole-systems approach to increasing physical activity in children and young people aged 5-14 years in Bradford, UK.

METHODS AND ANALYSIS

The evaluation aims to understand the lived experiences of children and families' relationship with physical activity and participation in the JU:MP programme. The study takes a collaborative and contributory citizen science approach, including focus groups, parent-child dyad interviews and participatory research. Feedback and data will guide changes within this study and the JU:MP programme. We also aim to examine participant experience of citizen science and the suitability of a citizen science approach to evaluate a whole-systems approach. Data will be analysed using framework approach alongside iterative analysis with and by citizen scientists in the collaborative citizen science study.

ETHICS AND DISSEMINATION

Ethical approval has been granted by the University of Bradford: study one (E891-focus groups as part of the control trial, E982-parent-child dyad interviews) and study two (E992). Results will be published in peer-reviewed journals and summaries will be provided to the participants, through schools or directly. The citizen scientists will provide input to create further dissemination opportunities.

UK government's new placement legislation is a 'good first step': a rapid qualitative analysis of consumer, business, enforcement and health stakeholder perspectives

BACKGROUND

The current food system in England promotes a population diet that is high in fat, sugar and salt (HFSS). To address this, the UK government has implemented legislation to restrict the promotion of HFSS products in prominent locations (e.g. store entrances, checkouts) in qualifying retailers since October 2022. This study investigated the perceived impact of the legislation for affected stakeholders.

METHODS

A pre-implementation rapid qualitative evaluation of stakeholder interviews. One hundred eight UK stakeholders participated in the study including 34 consumers, 24 manufacturers and retailers, 22 local authority enforcement officers and 28 academic and charitable health representatives. A participatory conference was used to enable policy recommendations to be confirmed by stakeholders.

RESULTS

Stakeholders perceived the legislation to be a 'good first step' towards improving population diet but recognised this needed to be considered amongst a range of long-term obesity policies. Areas of further support were identified and these are presented as six recommendations for government to support the successful implementation of the legislation: (1) provide a free central HFSS calculator, (2) refine legislation to enhance intent and clarity, (3) conduct a robust evaluation to assess intended and unintended outcomes, (4) provide greater support for smaller businesses, (5) provide ring-fenced resources to local authorities and (6) create and communicate a long-term roadmap for food and health.

CONCLUSIONS

This legislation has the potential to reduce impulse HFSS purchases and makes a solid start towards creating healthier retail outlets for consumers. Immediate government actions to create a freely accessible HFSS calculator, support smaller businesses and provide additional resources to local authorities would support successful implementation and enforcement. Independent evaluation of the implementation of the legislation will enable monitoring of potential unintended consequences identified in this study and support refinement of the legislation. A long-term roadmap is necessary to outline strategies to support equal access to healthier and sustainable food across the whole food system within the next 20-30 years.

What do we want to get out of this? a critical interpretive synthesis of the value of process evaluations, with a practical planning framework

BACKGROUND

Process evaluations aim to understand how complex interventions bring about outcomes by examining intervention mechanisms, implementation, and context. While much attention has been paid to the methodology of process evaluations in health research, the value of process evaluations has received less critical attention. We aimed to unpack how value is conceptualised in process evaluations by identifying and critically analysing 1) how process evaluations may create value and 2) what kind of value they may create.

METHODS

We systematically searched for and identified published literature on process evaluation, including guidance, opinion pieces, primary research, reviews, and discussion of methodological and practical issues. We conducted a critical interpretive synthesis and developed a practical planning framework.

RESULTS

We identified and included 147 literature items. From these we determined three ways in which process evaluations may create value or negative consequences: 1) through the socio-technical processes of 'doing' the process evaluation, 2) through the features/qualities of process evaluation knowledge, and 3) through using process evaluation knowledge. We identified 15 value themes. We also found that value varies according to the characteristics of individual process evaluations, and is subjective and context dependent.

CONCLUSION

The concept of value in process evaluations is complex and multi-faceted. Stakeholders in different contexts may have very different expectations of process evaluations and the value that can and should be obtained from them. We propose a planning framework to support an open and transparent process to plan and create value from process evaluations and negotiate trade-offs. This will support the development of joint solutions and, ultimately, generate more value from process evaluations to all.

A scoping review of systems approaches for increasing physical activity in populations

INTRODUCTION

The past decade has increasingly seen systems approaches as a featured theme in public health studies and policy documents. This trend is evident in the area of physical activity, which is a significant global health risk factor that is addressed in WHO's Global Action Plan on Physical Activity. We undertook a comprehensive scoping review to characterize the application of systems approaches to physical activity, to develop a typology of the objectives, themes and methods of research papers that purported to apply systems thinking to this issue.

METHODS

We searched electronic databases (PubMed, Web of Science, Scopus and PsycINFO) for studies published during the period 2010-2021 that explicitly applied systems approaches or methods to investigate and/or address population physical activity. A framework using systems-based methodological approaches was adapted to classify physical activity studies according to their predominant approach, covering basic descriptive, complex analytical and advanced forms of practice. We selected case studies from retained studies to depict the current "state of the art".

RESULTS

We included 155 articles in our narrative account. Literature reporting the application of systems approaches to physical activity is skewed towards basic methods and frameworks, with most attention devoted to conceptual framing and predictive modelling. There are few well-described examples of physical activity interventions which have been planned, implemented and evaluated using a systems perspective. There is some evidence of "retrofitted" complex system framing to describe programmes and interventions which were not designed as such.

DISCUSSION

We propose a classification of systems-based approaches to physical activity promotion together with an explanation of the strategies encompassed. The classification is designed to stimulate debate amongst policy-makers, practitioners and researchers to inform the further implementation and evaluation of systems approaches to physical activity.

CONCLUSION

The use of systems approaches within the field of physical activity is at an early stage of development, with a preponderance of descriptive approaches and a dearth of more complex analyses. We need to see movement towards a more sophisticated research agenda spanning the development, implementation and evaluation of systems-level interventions.

Developing and implementing 20-mph speed limits in Edinburgh and Belfast: mixed-methods study

Background

Transport initiatives such as 20-mph (≈30-km/h) speed limits are anticipated to result in fewer road casualties and improve perceptions of safety, leading to increases in active travel. Lower speeds may also lead to more pleasant environments in which to live, work and play.

Objectives

The main objective was to evaluate and understand the processes and effects of developing and implementing 20-mph speed limits in Edinburgh and Belfast. The focus was on health-related outcomes (casualties and active travel) that may lead to public health improvements. An additional objective was to investigate the political and policy factors (conditions) that led to the decision to introduce the new speed limits.

Design

This was a mixed-methods study that comprised an outcome, process, policy and economic evaluation of two natural experiments.

Setting

The study was set in Edinburgh, Scotland, and Belfast, Northern Ireland, from 2000 to 2018.

Participants

The whole population of each city were participants, as well as stakeholders involved in implementation and decision-making processes.

Intervention

The intervention was the implementation of 20-mph legislation, signage, enforcement, and education and awareness-raising in Edinburgh (citywide) and Belfast (city centre).

Main outcome measures

The main outcomes measured were speed; number, type and severity of road collisions; perceptions; and liveability.

Data sources

The following data sources were used – routinely and locally collected quantitative data for speed, volume of traffic, casualties and collisions, and costs; documents and print media; surveys; interviews and focus groups; and Google Street View (Google Inc., Mountain View, CA, USA).

Results

Collisions and casualties – the overall percentage reduction in casualty rates was 39% (the overall percentage reduction in collision rates was 40%) in Edinburgh. The percentage reduction for each level of severity was 23% for fatal casualties, 33% for serious casualties and 37% for minor casualties. In Belfast there was a 2% reduction in casualties, reflecting differences in the size, reach and implementation of the two schemes. Perceptions – in Edinburgh there was an increase in two factors (support for 20 mph and rule-following after implementation) supported by the qualitative data. Liveability – for both cities, there was a small statistical increase in liveability. Speed – mean and median speeds reduced by 1.34 mph and 0.47 mph, respectively, at 12 months in Edinburgh, with no statistically significant changes in Belfast. History, political context, local policy goals, local priorities and leadership influenced decision-making and implementation in the two cities.

Limitations

There was no analysis of active travel outcomes because the available data were not suitable.

Conclusions

The pre-implementation period is important. It helps frame public and political attitudes. The scale of implementation and additional activities in the two cities had a bearing on the impacts. The citywide approach adopted by Edinburgh was effective in reducing speeds and positively affected a range of public health outcomes. The city-centre approach in Belfast (where speeds were already low) was less effective. However, the main outcome of these schemes was a reduction in road casualties at all levels of severity.

Future work

Future work should develop a statistical approach to public health interventions that incorporates variables from multiple outcomes. In this study, each outcome was analysed independently of each other. Furthermore, population measures of active travel that can be administered simply, inexpensively and at scale should be developed.

Study registration

This study is registered as ISRCTN10200526.

Funding

This project was funded by the National Institute for Health and Care Research (NIHR) Public Health Research programme and will be published in full in Public Health Research; Vol. 10, No. 9. See the NIHR Journals Library website for further project information.

Addressing alcohol-related harms in the local night-time economy: a qualitative process evaluation from a complex systems perspective

OBJECTIVES

English local authorities (LAs) are interested in reducing alcohol-related harms and may use discretionary powers such as the Late Night Levy (LNL) to do so. This study aims to describe how system stakeholders hypothesise the levy may generate changes and to explore how the system, its actors and the intervention adapt and co-evolve over time.

DESIGN

A process evaluation from a complex systems perspective, using qualitative methods.

SETTING

A London LA with high densities of residential and commercial properties, which implemented the LNL in 2014.

PARTICIPANTS

Data were generated through interviews with LNL implementers and alcohol consumers, observations in bars and during LNL patrols and documentary review.

INTERVENTION

The LNL allows LAs to charge late-night alcohol retailers an annual fee (£299-£4440) to manage and police the night-time economy (NTE).

RESULTS

When the LNL was being considered, stakeholders from different interest groups advanced diverse opinions about its likely impacts while rarely referencing supporting research evidence. Proponents of the levy argued it could reduce crime and anti-social behaviour by providing additional funds to police and manage the NTE. Critics of the levy hypothesised adverse consequences linked to claims that the intervention would force venues to vary their hours or close, cluster closing times, reduce NTE diversity and undermine public-private partnerships. In the first 2 years, levy-funded patrols developed relationships with the licensed trade and the public. The LNL did not undermine public-private partnerships and while some premises varied their hours, these changes did not undermine the intervention's viability, nor significantly cluster venue closing times, nor obviously damage the area's reputation for having a diverse NTE.

CONCLUSIONS

This study applies a framework for process evaluation from a complex systems perspective. The evaluation could be extended to measure alcohol-related outcomes and to consider the interplay between the national and local systems.

Challenges in Implementing the National Health Response to COVID-19 in Senegal

Since the beginning of the COVID-19 pandemic in Africa, many epidemiological or anthropological studies have been published. However, few studies have yet been conducted to understand the implementation of State interventions to fight the COVID-19 pandemic. In Senegal, the national response plan was planned before the country experienced its first official case of COVID-19 on 2 March 2020. This qualitative study, conducted in March and April 2021, based on 189 interviews, aims to understand how the national response has been implemented in several regions of Senegal. Implementation of the response to the pandemic was favoured by good preparation, capacity to adapt, responsiveness of health actors, and commitment for both the political and religious authorities. The implementation response was confronted by several constraining factors such as the coercive approach, the challenges of coordinating actors, and the lack of intersectoral response. The central level has sometimes used reflexivity processes to adapt its response, but it has remained highly politicized, centralized, directive, and with little involvement of civil society. In Senegal, the response to the pandemic has been implemented in a relatively political and directive, even coercive manner, without necessarily considering prior knowledge and the need to adapt it to local contexts and to involve civil society and community actors in the process.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43477-022-00053-4.

Navigating the river(s) of systems change: a multi-methods, qualitative evaluation exploring the implementation of a systems approach to physical activity in Gloucestershire, England

OBJECTIVES

Systems approaches aim to change the environments in which people live, through cross-sectoral working, by harnessing the complexity of the problem. This paper sought to identify: (1) the strategies which support the implementation of We Can Move (WCM), (2) the barriers to implementation, (3) key contextual factors that influence implementation and (4) impacts associated with WCM.

DESIGN

A multi-methods evaluation of WCM was completed between April 2019 and April 2021. Ripple Effects Mapping (REM) and semi-structured interviewers were used. Framework and content analysis were systematically applied to the dataset.

SETTING

WCM-a physical activity orientated systems approach being implemented in Gloucestershire, England.

PARTICIPANTS

31 stakeholder interviews and 25 stakeholders involved in 15 REM workshops.

RESULTS

A white-water rafting analogy was developed to present the main findings. The successful implementation of WCM required a facilitative, well-connected and knowledgeable guide (ie, the lead organisation), a crew (ie, wider stakeholders) who's vision and agenda aligned with WCM's purpose, and a flexible delivery approach that could respond to ever-changing nature of the river (ie, local and national circumstances). The context surrounding WCM further strengthened and hampered its implementation. Barriers included evaluative difficulties, a difference in stakeholder and organisational perspectives, misaligned expectations and understandings of WCM, and COVID-19 implications (COVID-19 also presented as a facilitative factor). WCM was said to strengthen cohesion and collaboration between partners, benefit other agendas and policies (eg, mental health, town planning, inequality), and improve physical activity opportunities and environments.

CONCLUSIONS

This paper is one of the first to evaluate a systems approach to increasing physical activity. We highlight key strategies and contextual factors that influenced the implementation of WCM and demonstrate some of the wider benefits from such approaches. Further research and methodologies are required to build the evidence base surrounding systems approaches in Public Health.

[Evaluation of the effects of a complex intervention]

Appraising the effects of a complex intervention is one step in a more broadly based research process, from the construction or modelling of the intervention to its actual deployment. It consists in measuring the effectiveness or impact of the intervention, i.e. analyzing its capacity to produce change. The aim of this article is to obtain unbiased measurement of the average effects of an intervention, based on a panel of predetermined parameters and on the assumption of a causal link between the intervention and the measured result. This article is consequently devoted to evaluation of the effects of a complex intervention and focuses on the methodological challenges of its three key stages : 1) modelling of the intervention and feasibility analysis, essential prerequisites ; 2) the choice of study design and of the effects to be measured, that is to say the methodological premises ; and 3) process analysis, carried out in parallel with the evaluation of effects, leading to an indispensable appraisal of the intervention implementation and of the context into which it is integrated. The article is illustrated by five intervention impact assessment projects. A specific objective when evaluating the effects of a complex intervention consists in (a) moving away from a simple search for causality involving the intervention and its effects and (b) toward understanding of the effectiveness mechanisms, once again taking into account the context and the actual conditions of implementation. The challenge is to embrace rather than limit the complexity of the intervention, this being an essential prerequisite for its successful deployment and eventual generalization.

Analyzing implementation of public health interventions : a need for rigor, and the challenges of stakeholder involvement

OBJECTIVES

This article shows how conceptual models can help to develop and evaluate public health interventions. It also reports on the challenges of getting stakeholders involved.

METHOD

The analysis is based on the reflexive approach applied by the authors during their participation in two public health intervention research (PHIR) projects, in France and in Burkina Faso.

RESULTS

In Paris, PHIR aimed to enable sub-Saharan immigrants to appropriate the existing means of prevention and sexual health care and to strengthen their empowerment in view of preserving their health. Evaluation was carried out using mixed methods. The intervention process theory is based on Ninacs' conceptual model of individual empowerment. The Consolidated Framework For Implementation Research (CFIR) was mobilized a posteriori to analyze the process. PHIR stemmed from collaboration between a research team and two associations. The different stakeholders were involved in the evaluation process, as were, at certain times, persons in highly precarious situations. In Ouagadougou, a community-based dengue vector control intervention was deployed to address an essential but neglected need. As regards evaluation, we opted for a holistic, mixed method approach (effectiveness and process). The contents of the intervention were determined based on tacit knowledge, a community preference survey and solid evidence. The theoretical framework of the intervention consisted in an eco-biological model of vector control. The implementation analysis combined an internal assessment of implementation fidelity with an external CFIR process analysis. All stakeholders were involved in the evaluation process.

DISCUSSION

Analysis confirmed not only the value of process evaluations in PHIR, but also the primordial importance of a rigorous approach. Stakeholder involvement is a major challenge to be addressed early in the planning of RISPs; with this in mind, effective and ethically sound assessment mechanisms need to be drawn up. Interdisciplinary evaluative approaches should be preferred, and the use of justified, relevant, and flexible frameworks is highly recommended.

CONCLUSION

Lessons learned for those wishing to engage in the process evaluation of a public health intervention are hereby presented.

Process evaluations of mental health and psychosocial support interventions for populations affected by humanitarian crises

BACKGROUND

Randomised controlled trials (RCTs) have been increasingly used to test the effectiveness of mental health and psychosocial support(MHPSS) interventions for populations affected by humanitarian crises. Process evaluations are often integrated within RCTs of psychological interventions to investigate the implementation of the intervention, the impact of context, and possible mechanisms of action. We aimed to explore limitations and strengths of how process evaluations are currently conceptualised and implemented within MHPSS RCTs specifically.

METHODS

In April-June 2021 we conducted semi-structured interviews with 24 researchers involved in RCTs of MHPSS interventions in 23 different countries. Participants were selected based on systematic reviews of MHPSS interventions, funders' databases, and personal networks. Data were analysed using codebook thematic analysis.

RESULTS

The conduct of process evaluations was characterized by high heterogeneity in perceived function, implementation outcomes assessed, and methods used. While process evaluations were overwhelmingly considered as an important component of an RCT, there were different opinions on their perceived quality. This could be explained by the varying prioritization of effectiveness data over implementation data, confusion around the nature of process evaluations, and challenges in the collection and analysis of process data in humanitarian settings. Various practical recommendations were made by participants to improve future process evaluations in relation to: (i) study design (e.g., embedding process evaluations in study protocol and overall study objectives); (ii) methods (e.g., use of mixed methods); and (iii) increased financial and human resources dedicated to process evaluations.

CONCLUSION

The current state of process evaluations in MHPSS RCTs is heterogeneous. The quality of process evaluations should be improved to strengthen implementation science of the growing number of evidence-informed MHPSS interventions.

Applying systems thinking to identify enablers and challenges to scale-up interventions for hypertension and diabetes in low-income and middle-income countries: protocol for a longitudinal mixed-methods study

INTRODUCTION

There is an urgent need to reduce the burden of non-communicable diseases (NCDs), particularly in low-and middle-income countries, where the greatest burden lies. Yet, there is little research concerning the specific issues involved in scaling up NCD interventions targeting low-resource settings. We propose to examine this gap in up to 27 collaborative projects, which were funded by the Global Alliance for Chronic Diseases (GACD) 2019 Scale Up Call, reflecting a total funding investment of approximately US$50 million. These projects represent diverse countries, contexts and adopt varied approaches and study designs to scale-up complex, evidence-based interventions to improve hypertension and diabetes outcomes. A systematic inquiry of these projects will provide necessary scientific insights into the enablers and challenges in the scale up of complex NCD interventions.

METHODS AND ANALYSIS

We will apply systems thinking (a holistic approach to analyse the inter-relationship between constituent parts of scaleup interventions and the context in which the interventions are implemented) and adopt a longitudinal mixed-methods study design to explore the planning and early implementation phases of scale up projects. Data will be gathered at three time periods, namely, at planning (TP), initiation of implementation (T0) and 1-year postinitiation (T1). We will extract project-related data from secondary documents at TP and conduct multistakeholder qualitative interviews to gather data at T0 and T1. We will undertake descriptive statistical analysis of TP data and analyse T0 and T1 data using inductive thematic coding. The data extraction tool and interview guides were developed based on a literature review of scale-up frameworks.

ETHICS AND DISSEMINATION

The current protocol was approved by the Monash University Human Research Ethics Committee (HREC number 23482). Informed consent will be obtained from all participants. The study findings will be disseminated through peer-reviewed publications and more broadly through the GACD network.

A Combined Theory of Change-Group Model Building Approach to Evaluating “Farm to Fork” Models for School Feeding in the Caribbean

There is a scarcity of research on building nutrition-sensitive value chains (NSVCs) to improve diets and nutrition outcomes of populations in the Caribbean. This study contributes to filling this research gap by outlining a participatory approach to evaluating a NSVC model for “farm to fork” (F2F) school feeding in the Eastern Caribbean Island of St. Kitts. Using a combined group model building (GMB) and theory of change (ToC) approach, policy actors and other stakeholders (n = 37) across the school feeding value chain were guided through a facilitated process to evaluate the ToC underlying a series of F2F interventions designed to enhance childhood nutrition. Stakeholders at the workshop engaged collaboratively to create a causal map of interconnected “system factors” that help explain behaviors contributing to unhealthy eating among children that extended well-beyond the original F2F project ToC that had been used to inform interventions. Through this facilitated GMB process, stakeholders proposed additional food system interventions, and identified multiple “impact pathways” and “mediating influences” underlying local availability and consumption of nutritious foods in local school environments. Workshop participants were also able to identify leverage points where community-level efforts, alongside research interventions, may ensure that initiatives for building local NSVCs are ultimately institutionalized. Results of this study suggest that developing NSVCs for school feeding and food systems in the Caribbean requires both locally driven innovation and the leveraging of system-wide resources, with lessons for project intervention strategies.

Evaluation proposal of a national community-based obesity prevention programme: a novel approach considering the complexity perspective

Community-based obesity prevention programmes are considered an important strategy to curb the obesity epidemic. The JOGG (Youth At a Healthy Weight) approach is a large-scale community-based programme for childhood obesity prevention in the Netherlands that has been implemented over the past ten years. Practice-based development of the programme, both at the national and local level, increasingly poses challenges for its evaluation. One considerable challenge is the increasing acknowledgement of the complexity in the JOGG-approach, characterized by (a) objectives that vary locally, (b) adaptions to the programme over time in response to a community's shifting needs, challenges and opportunities, and (c) emergent outcomes and non-linear causality.We propose an evaluation framework that highlights elements of the complex local practice, including the local programme theory, implementation, adaption, the influence of context and feedback loops and intended as well as emergent and unintended outcomes. By studying each of these elements in practice, we hope to learn about principles that guide effective obesity prevention across contexts. The results of the proposed evaluation will inform both practice and research.Considering complexity in evaluation is a relatively new challenge in public health and therefore an emergent research area. The proposed framework for complex evaluations allows to retrospectively evaluate a programme that was implemented and developed in practice, and enables us to learn from practice-based experiences. Following the ISBNPA Dare2Share initiative, we kindly invite other researchers in the field to share their ideas and experiences regarding integration of complexity in evaluation.

Identifying priorities for research on financial risk protection to achieve universal health coverage: a scoping overview of reviews

OBJECTIVES

Financial risk protection (FRP) is an indicator of the Sustainable Development Goal 3 universal health coverage (UHC) target. We sought to characterise what is known about FRP in the UHC context and to identify evidence gaps to prioritise in future research.

DESIGN

Scoping overview of reviews using the Arksey & O'Malley and Levac & Colquhoun framework and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews reporting guidelines.

DATA SOURCES

MEDLINE, PsycINFO, CINAHL-Plus and PAIS Index were systematically searched for studies published between 1 January 1995 and 20 July 2021.

ELIGIBILITY CRITERIA

Records were screened by two independent reviewers in duplicate using the following criteria: (1) literature review; (2) focus on UHC achievement through FRP; (3) English or French language; (4) published after 1995 and (5) peer-reviewed.

DATA EXTRACTION AND SYNTHESIS

Two reviewers extracted data using a standard form and descriptive content analysis was performed to synthesise findings.

RESULTS

50 studies were included. Most studies were systematic reviews focusing on low-income and middle-income countries. Study periods spanned 1990 and 2020. While FRP was recognised as a dimension of UHC, it was rarely defined as a concept. Out-of-pocket, catastrophic and impoverishing health expenditures were most commonly used to measure FRP. Pooling arrangements, expansion of insurance coverage and financial incentives were the main interventions for achieving FRP. Evidence gaps pertained to the effectiveness, cost-effectiveness and equity implications of efforts aimed at increasing FRP. Methodological gaps related to trade-offs between single-country and multicountry analyses; lack of process evaluations; inadequate mixed-methods evidence, disaggregated by relevant characteristics; lack of comparable and standardised measurement and short follow-up periods.

CONCLUSIONS

This scoping overview of reviews characterised what is known about FRP as a UHC dimension and found evidence gaps related to the effectiveness, cost-effectiveness and equity implications of FRP interventions. Theory-informed mixed-methods research using high-quality, longitudinal and disaggregated data is needed to address these objectives.

Critical factors that affect the functioning of a research and evaluation capacity building partnership: A causal loop diagram

Introduction

Public health policy and practice is strengthened by the application of quality evidence to decision making. However, there is limited understanding of how initiatives that support the generation and use of evidence in public health are operationalised. This study examines factors that support the internal functioning of a partnership, the Western Australian Sexual Health and Blood-borne Virus Applied Research and Evaluation Network (SiREN). SiREN aims to build research and evaluation capacity and increase evidence-informed decision making in a public health context.

Methods

This study was informed by systems concepts. It developed a causal loop diagram, a type of qualitative system model that illustrated the factors that influence the internal operation of SiREN. The causal loop diagram was developed through an iterative and participatory process with SiREN staff and management (n = 9) via in-depth semi-structured interviews (n = 4), workshops (n = 2), and meetings (n = 6).

Results

Findings identified critical factors that affected the functioning of SiREN. Central to SiREN’s ability to meet its aims was its capacity to adapt within a dynamic system. Adaptation was facilitated by the flow of knowledge between SiREN and system stakeholders and the expertise of the team. SiREN demonstrated credibility and capability, supporting development of new, and strengthening existing, partnerships. This improved SiREN’s ability to be awarded new funding and enhanced its sustainability and growth. SiREN actively balanced divergent stakeholder interests to increase sustainability.

Conclusion

The collaborative development of the diagram facilitated a shared understanding of SiREN. Adaptability was central to SiREN achieving its aims. Monitoring the ability of public health programs to adapt to the needs of the systems in which they work is important to evaluate effectiveness. The detailed analysis of the structure of SiREN and how this affects its operation provide practical insights for those interested in establishing a similar project.

There is no trustable data: the state and data accuracy of violence against women in Indonesia

The number of violence against women in Indonesia continues to increase every year, this can be seen from data released by the National Commission on Violence Against Women and the State Ministry of Women's Empowerment and Child Protection. The problem is, although the two state institutions regularly release data on violence against women, the data released does not nationally represent the data on violence. This study aims to find reasons on why data on violence in Indonesia cannot be used as material for policy making. By using ethnographic methods in the National Commission on Violence Against Women, the Ministry, and six partner institutions, this study found a number of facts why the data on violence in Indonesia is so unreliable. At the ministry level, the main constraints are with the forms that are difficult to fill in, the model of tiered bureaucracy that is useful for reporting, but with the higher the level, the less violence data there are, the limited number of registrar. Whereas National Commission on Violence Against Women data are constrained by a centralized and voluntary data collection model, that more than 65% of partners do not return the data collection forms. This results in both the Ministry and the Komnas Perempuan data being just the tip of the iceberg of the problem of violence against women in Indonesia.

A whole system approach to increasing children's physical activity in a multi-ethnic UK city: a process evaluation protocol

BACKGROUND

Engaging in regular physical activity requires continued complex decision-making in varied and dynamic individual, social and structural contexts. Widespread shortfalls of physical activity interventions suggests the complex underlying mechanisms of change are not yet fully understood. More insightful process evaluations are needed to design and implement more effective approaches. This paper describes the protocol for a process evaluation of the JU:MP programme, a whole systems approach to increasing physical activity in children and young people aged 5-14 years in North Bradford, UK.

METHODS

This process evaluation, underpinned by realist philosophy, aims to understand the development and implementation of the JU:MP programme and the mechanisms by which JU:MP influences physical activity in children and young people. It also aims to explore behaviour change across wider policy, strategy and neighbourhood systems. A mixed method data collection approach will include semi-structured interview, observation, documentary analysis, surveys, and participatory evaluation methods including reflections and ripple effect mapping.

DISCUSSION

This protocol offers an innovative approach on the use of process evaluation feeding into an iterative programme intended to generate evidence-based practice and deliver practice-based evidence. This paper advances knowledge regarding the development of process evaluations for evaluating systems interventions, and emphasises the importance of process evaluation.