The ENCOMPASS framework: a practical guide for the evaluation of public health programmes in complex adaptive systems

Qualitative systems mapping in promoting physical activity and cardiorespiratory fitness: Perspectives and recommendations

The purpose of this report is to provide a perspective on the use of qualitative systems mapping, provide examples of physical activity (PA) systems maps, discuss the role of PA systems mapping in the context of iterative learning to derive breakthrough interventions, and provide actionable recommendations for future work. Systems mapping methods and applications for PA are emerging in the scientific literature in the study of complex health issues and can be used as a prelude to mathematical/computational modeling where important factors and relationships can be elucidated, data needs can be prioritized and guided, interventions can be tested and (co)designed, and metrics and evaluations can be developed. Examples are discussed that describe systems mapping based on Group Model Building or literature reviews. Systems maps are highly informative, illustrate multiple components to address PA and physical inactivity issues, and make compelling arguments against single intervention action. No studies were identified in the literature scan that considered cardiorespiratory fitness the focal point of a systems maps. Recommendations for future research and education are presented and it is concluded that systems mapping represents a valuable yet underutilized tool for visualizing the complexity of PA promotion.

Development of an action programme tackling obesity-related behaviours in adolescents: a participatory system dynamics approach

System dynamics approaches are increasingly addressing the complexity of public health problems such as childhood overweight and obesity. These approaches often use system mapping methods, such as the construction of causal loop diagrams, to gain an understanding of the system of interest. However, there is limited practical guidance on how such a system understanding can inform the development of an action programme that can facilitate systems changes. The Lifestyle Innovations Based on Youth Knowledge and Experience (LIKE) programme combines system dynamics and participatory action research to improve obesity-related behaviours, including diet, physical activity, sleep and sedentary behaviour, in 10-14-year-old adolescents in Amsterdam, the Netherlands. This paper illustrates how we used a previously obtained understanding of the system of obesity-related behaviours in adolescents to develop an action programme to facilitate systems changes. A team of evaluation researchers guided interdisciplinary action-groups throughout the process of identifying mechanisms, applying the Intervention Level Framework to identify leverage points and arriving at action ideas with aligning theories of change. The LIKE action programme consisted of 8 mechanisms, 9 leverage points and 14 action ideas which targeted the system's structure and function within multiple subsystems. This illustrates the feasibility of developing actions targeting higher system levels within the confines of a research project timeframe when sufficient and dedicated effort in this process is invested. Furthermore, the system dynamics action programme presented in this study contributes towards the development and implementation of public health programmes that aim to facilitate systems changes in practice.

Developing and Evaluating a Data-Driven and Systems Approach to Health Promotion Among Vocational Students: Protocol for the Data Health Study

BACKGROUND

Vocational school students exhibit significant risk behaviors in terms of poor diet, frequent use of nicotine products, inadequate fruit and vegetable intake, low levels of physical activity, and poor mental health. This makes vocational students vulnerable to the development of noncommunicable diseases. Therefore, effective health promotion programs targeting vocational students are required.

OBJECTIVE

The Danish study "Data-driven and Systems Approach to Health Promotion Among Vocational Students" (Data Health) aims to develop, implement, and evaluate a systems approach to support vocational schools, municipalities, and local communities in implementing locally relevant health promotion actions among and for vocational students. This paper describes the Data Health program and how implementation and preliminary effectiveness will be evaluated.

METHODS

The Data Health program offers an iterative 5-step process to develop changes in the systems that shape health behavior and well-being among vocational students. The program will be implemented and evaluated in 8 Danish vocational schools in 4 municipalities. The implementation of the process and actions will be explored using a systems-based evaluation design that assesses contextual differences and the mechanisms through which the program leads to changes in the systems. Preliminary effectiveness at the individual level (students' self-reported health behavior and well-being) and organizational level (school organizational readiness reported by school staff) will be assessed using a quasi-experimental design, and cross-sectional data will be collected at all 8 schools simultaneously 4 times during the 2-year study period.

RESULTS

This study was launched in 2021, and data collection is expected to be completed in June 2024. The first results are expected to be submitted for publication in January 2024.

CONCLUSIONS

We expect that the Data Health study will make significant contributions to complex intervention research by contributing to the paucity of research studies that have used systems approaches in school settings. The study will also provide evidence of successful elements for systems change and effectiveness to determine whether a national scale-up can be recommended.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05308459; https://clinicaltrials.gov/study/NCT05308459.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/52571.

Using network analysis to identify leverage points based on causal loop diagrams leads to false inference

Network analysis is gaining momentum as an accepted practice to identify which factors in causal loop diagrams (CLDs)-mental models that graphically represent causal relationships between a system's factors-are most likely to shift system-level behaviour, known as leverage points. This application of network analysis, employed to quantitatively identify leverage points without having to use computational modelling approaches that translate CLDs into sets of mathematical equations, has however not been duly reflected upon. We evaluate whether using commonly applied network analysis metrics to identify leverage points is justified, focusing on betweenness- and closeness centrality. First, we assess whether the metrics identify the same leverage points based on CLDs that represent the same system but differ in inferred causal structure-finding that they provide unreliable results. Second, we consider conflicts between assumptions underlying the metrics and CLDs. We recognise six conflicts suggesting that the metrics are not equipped to take key information captured in CLDs into account. In conclusion, using betweenness- and closeness centrality to identify leverage points based on CLDs is at best premature and at worst incorrect-possibly causing erroneous identification of leverage points. This is problematic as, in current practice, the results can inform policy recommendations. Other quantitative or qualitative approaches that better correspond with the system dynamics perspective must be explored.

Test, evidence, transition projects in Scotland: developing the evidence needed for transition of effective interventions in cancer care from innovation into mainstream practice

BACKGROUND

A robust evidence base is required to assist healthcare commissioners and providers in selecting effective and sustainable approaches to improve cancer diagnosis and treatment. Such evidence can be difficult to build, given the fast-paced and highly pressured nature of healthcare delivery, the absence of incentives, and the presence of barriers in conducting pragmatic yet robust research evaluations. Cancer Research UK (CRUK) has played an active part in building the evidence base through its funding of programmes to identify, evaluate and scale-up innovative approaches across the UK. The aim of this paper is to describe and explain the research design and intended approach and activities for two cancer services improvement projects in Scotland funded by CRUK.

METHODS

A hybrid effectiveness-implementation study design will assess both the efficiency of the new pathways and their implementation strategies, with the aim of generating knowledge for scale-up. A range of implementation, service and clinical outcomes will be assessed as determined by the projects' Theories of Change (ToCs). A naturalistic case study approach will enable in-depth exploration of context and process, and the collection and synthesis of data from multiple sources including routine datasets, patient and staff surveys, in-depth interviews and observational and other data. The evaluations are informed throughout by a patient/public representatives' group, and by small group discussions with volunteer cancer patients.

DISCUSSION

Our approach has been designed to provide a holistic understanding of how (well) the improvement projects work (in relation to their anticipated outcomes), and how they interact with their wider contexts. The evaluations will help identify barriers, facilitators, and unanticipated consequences that can impact scalability, sustainability and spread. By opting for a pragmatic, participatory evaluation research design, we hope to inform strategies for scaling up successful innovations while addressing challenges in a targeted manner.

Comprehensive application of a systems approach to obesity prevention: a scoping review of empirical evidence

A systems approach to obesity prevention is increasingly urged. However, confusion exists on what a systems approach entails in practice, and the empirical evidence on this new approach is unclear. This scoping review aimed to identify and synthesise studies/programmes that have comprehensively applied a systems approach to obesity prevention in intervention development, delivery/implementation, and evaluation. By searching international databases and grey literature, only three studies (10 publications) met inclusion criteria, which might be explained partially by suboptimal reporting. No conclusion on the effectiveness of this approach can be drawn yet due to the limited evidence base. We identified common features shared by the included studies, such as measuring ongoing changes, in addition to endpoint outcomes, and supporting capacity building. Some facilitators and barriers to applying a comprehensive systems approach in practice were identified. More well-designed and reported studies are needed, especially from low- and middle-income countries.

Understanding the system dynamics of obesity-related behaviours in 10- to 14-year-old adolescents in Amsterdam from a multi-actor perspective

Introduction and Methods

To develop an understanding of the dynamics driving obesity-related behaviours in adolescents, we conducted systems-based analysis on a causal loop diagram (CLD) created from a multi-actor perspective, including academic researchers, adolescents and local stakeholders.

Results

The CLD contained 121 factors and 31 feedback loops. We identified six subsystems with their goals: (1) interaction between adolescents and the food environment, with profit maximisation as goal, (2) interaction between adolescents and the physical activity environment, with utility maximisation of outdoor spaces as goal, (3) interaction between adolescents and the online environment, with profit maximisation from technology use as goal, (4) interaction between adolescents, parenting and the wider socioeconomic environment, with a goal focused on individual parental responsibility, (5) interaction between healthcare professionals and families, with the goal resulting in treating obesity as an isolated problem, and (6) transition from childhood to adolescence, with the goal centring around adolescents' susceptibility to an environment that stimulates obesity-related behaviours.

Discussion

Analysis showed that inclusion of the researchers' and stakeholders' perspectives contributed to an understanding of how the system structure of an environment works. Integration of the adolescents' perspective enriched insights on how adolescents interact with that environment. The analysis further showed that the dynamics driving obesity-related behaviours are geared towards further reinforcing such behaviours.

Human Factors in Leveraging Systems Science to Shape Public Policy for Obesity: A Usability Study

Background: despite a broad consensus on their importance, applications of systems thinking in policymaking and practice have been limited. This is partly caused by the longstanding practice of developing systems maps and software in the intention of supporting policymakers, but without knowing their needs and practices. Objective: we aim to ensure the effective use of a systems mapping software by policymakers seeking to understand and manage the complex system around obesity, physical, and mental well-being. Methods: we performed a usability study with eight policymakers in British Columbia based on a software tool (ActionableSystems) that supports interactions with a map of obesity. Our tasks examine different aspects of systems thinking (e.g., unintended consequences, loops) at several levels of mastery and cover common policymaking needs (identification, evaluation, understanding). Video recordings provided quantitative usability metrics (correctness, time to completion) individually and for the group, while pre- and post-usability interviews yielded qualitative data for thematic analysis. Results: users knew the many different factors that contribute to mental and physical well-being in obesity; however, most were only familiar with lower-level systems thinking concepts (e.g., interconnectedness) rather than higher-level ones (e.g., feedback loops). Most struggles happened at the lowest level of the mastery taxonomy, and predominantly on network representation. Although participants completed tasks on loops and multiple pathways mostly correctly, this was at the detriment of spending significant time on these aspects. Results did not depend on the participant, as their experiences with the software were similar. The thematic analysis revealed that policymakers did not have a typical workflow and did not use any special software or tools in their policy work; hence, the integration of a new tool would heavily depend on individual practices. Conclusions: there is an important discrepancy between what constitutes systems thinking to policymakers and what parts of systems thinking are supported by software. Tools may be more successfully integrated when they include tutorials (e.g., case studies), facilitate access to evidence, and can be linked to a policymaker’s portfolio.

Promoting the health of vulnerable populations: Three steps towards a systems-based re-orientation of public health intervention research

This paper proposes a novel framework for the development of interventions in vulnerable populations. The framework combines a complex systems lens with syndemic theory. Whereas funding bodies, research organizations and reporting guidelines tend to encourage intervention research that (i) focuses on singular and predefined health outcomes, (ii) searches for generalizable cause-effect relationships, and (iii) aims to identify universally effective interventions, the paper suggests that a different direction is needed for addressing health inequities: We need to (i) start with exploratory analysis of population-level data, and (ii) invest in contextualized in-depth knowledge of the complex dynamics that produce health inequities in specific populations and settings, while we (iii) work with stakeholders at multiple levels to create change within systems.

Development and Validation of a Machine Learning Wrist-worn Step Detection Algorithm with Deployment in the UK Biobank

Background

Step count is an intuitive measure of physical activity frequently quantified in a range of health-related studies; however, accurate quantification of step count can be difficult in the free-living environment, with step counting error routinely above 20% in both consumer and research-grade wrist-worn devices. This study aims to describe the development and validation of step count derived from a wrist-worn accelerometer and to assess its association with cardiovascular and all-cause mortality in a large prospective cohort study.

Methods

We developed and externally validated a hybrid step detection model that involves self-supervised machine learning, trained on a new ground truth annotated, free-living step count dataset (OxWalk, n=39, aged 19-81) and tested against other open-source step counting algorithms. This model was applied to ascertain daily step counts from raw wrist-worn accelerometer data of 75,493 UK Biobank participants without a prior history of cardiovascular disease (CVD) or cancer. Cox regression was used to obtain hazard ratios and 95% confidence intervals for the association of daily step count with fatal CVD and all-cause mortality after adjustment for potential confounders.

Findings

The novel step algorithm demonstrated a mean absolute percent error of 12.5% in free-living validation, detecting 98.7% of true steps and substantially outperforming other recent wrist-worn, open-source algorithms. Our data are indicative of an inverse dose-response association, where, for example, taking 6,596 to 8,474 steps per day was associated with a 39% [24-52%] and 27% [16-36%] lower risk of fatal CVD and all-cause mortality, respectively, compared to those taking fewer steps each day.

Interpretation

An accurate measure of step count was ascertained using a machine learning pipeline that demonstrates state-of-the-art accuracy in internal and external validation. The expected associations with CVD and all-cause mortality indicate excellent face validity. This algorithm can be used widely for other studies that have utilised wrist-worn accelerometers and an open-source pipeline is provided to facilitate implementation.

Challenges in evaluating implementation and effectiveness in real-world settings: evaluation proposal for school-based health-promoting intervention

There are various research designs and approaches to investigate how health-promoting activities are implemented in complex, real-world systems, and to identify potential health effects that might occur following implementation. Although literature describes guidelines to perform and report about implementation research and effect evaluations, no specific guidelines exist on analysing and reporting about the combination of effectiveness data and implementation data collected as part of intervention evaluation in complex and diverse settings. This paper describes the evaluation of primary school-based health-promoting activities in complex systems. Furthermore, an approach for data categorization inspired by Rogers' Diffusion of Innovations theory is presented that can facilitate structuring the study's results and relating the degree of implementation to any impact on effectiveness outcomes that might be observed. Researchers interested in using this approach for data categorization have to ensure that the following three conditions are met: (i) data on an intervention's efficacy in a controlled setting with optimal implementation is available; (ii) key points that define an intervention's optimal implementation are available and (iii) an evaluation study is performed, collecting both effectiveness data and implementation data in a real-world context. This data categorization approach can be useful to generate more insight into an intervention's effectiveness under varying circumstances, and optimal support and advice can be provided to stakeholders to achieve maximum impact of population-based health-promoting interventions in complex, real-world systems. However, the proposed approach is a first suggestion and further testing and adaptation is necessary to increase its usefulness. Knowledge and experience sharing among researchers performing comparable research can increase the knowledge base regarding this subject.

The association between the presence of fast-food outlets and BMI: the role of neighbourhood socio-economic status, healthy food outlets, and dietary factors

Background

Evidence on the association between the presence of fast-food outlets and Body Mass Index (BMI) is inconsistent. Furthermore, mechanisms underlying the fast-food outlet presence-BMI association are understudied. We investigated the association between the number of fast-food outlets being present and objectively measured BMI. Moreover, we investigated to what extent this association was moderated by neighbourhood socio-economic status (NSES) and healthy food outlets. Additionally, we investigated mediation by frequency of fast-food consumption and amount of fat intake.

Methods

In this cross-sectional study, we used baseline data of adults in Lifelines (N = 149,617). Geo-coded residential addresses were linked to fast-food and healthy food outlet locations. We computed the number of fast-food and healthy food outlets within 1 kilometre (km) of participants’ residential addresses (each categorised into null, one, or at least two). Participants underwent objective BMI measurements. We linked data to Statistics Netherlands to compute NSES. Frequency of fast-food consumption and amount of fat intake were measured through questionnaires in Lifelines. Multivariable multilevel linear regression analyses were performed to investigate associations between fast-food outlet presence and BMI, adjusting for individual and environmental potential confounders. When exposure-moderator interactions had p-value < 0.10 or improved model fit (∆AIC ≥ 2), we conducted stratified analyses. We used causal mediation methods to assess mediation.

Results

Participants with one fast-food outlet within 1 km had a higher BMI than participants with no fast-food outlet within 1 km (B = 0.11, 95% CI: 0.01, 0.21). Effect sizes for at least two fast-food outlets were larger in low NSES areas (B = 0.29, 95% CI: 0.01, 0.57), and especially in low NSES areas where at least two healthy food outlets within 1 km were available (B = 0.75, 95% CI: 0.19, 1.31). Amount of fat intake, but not frequency of fast-food consumption, explained this association for 3.1%.

Conclusions

Participants living in low SES neighbourhoods with at least two fast-food outlets within 1 km of their residential address had a higher BMI than their peers with no fast-food outlets within 1 km. Among these participants, healthy food outlets did not buffer the potentially unhealthy impact of fast-food outlets. Amount of fat intake partly explained this association. This study highlights neighbourhood socio-economic inequalities regarding fast-food outlets and BMI.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-022-13826-1.

A Theoretical Perspective on Why Socioeconomic Health Inequalities Are Persistent: Building the Case for an Effective Approach

Despite policy intentions and many interventions aimed at reducing socioeconomic health inequalities in recent decades in the Netherlands and other affluent countries, these inequalities have not been reduced. Based on a narrative literature review, this paper aims to increase insight into why socioeconomic health inequalities are so persistent and build a way forward for improved approaches from a theoretical perspective. Firstly, we present relevant theories focusing on individual determinants of health-related behaviors. Thereafter, we present theories that take into account determinants of the individual level and the environmental level. Lastly, we show the complexity of the system of individual determinants, environmental determinants and behavior change for low socioeconomic position (SEP) groups and describe the next steps in developing and evaluating future effective approaches. These steps include systems thinking, a complex whole-system approach and participation of all stakeholders in system change.