Using theories of change to design monitoring and evaluation of community engagement in research: experiences from a research institute in Malawi

A youth advisory group on health and health research in rural Cambodia

Engaging young people in health research has been promoted globally. We explored the outcomes of youth advisory group on health and research engagement (YAGHRE) in rural Cambodia. In May 2021, the Mahidol Oxford Tropical Medicine Research Unit (MORU) partnered with a local health centre and a secondary school to establish a youth engagement group. Ten students underwent training and led health engagement activities in schools and communities. Activities were documented as field notes and audio-visual materials which underwent content analysis using theory of change supplemented by iterative discussions with YAGHRE members and stakeholders. Five major outcomes were identified: 1. Increased respect. Engagement activities developed based on input from students and stakeholders may have fostered greater respect. 2. Built trust and relationships. Frequent visits to MORU's laboratory and interactions with researchers appeared to contribute to the building of trust and relationship. 3. Improved health and research literacy. Learning new health and research topics, through participatory activities may have improved literacy; 4. Improved uptake of health and research interventions. Health promotional activities and communication with research participants potentially increased the uptake of interventions; 5. Improved community health. YAGHRE's health promotional interventions may have contributed in enhancing community's health.

Enhancing media communication of health research: an evaluation of the Journalist in Residence Programme at the Malawi Liverpool Wellcome Trust Clinical Research Programme

Background: In 2012, the Science Communication department at the Malawi-Liverpool Wellcome Trust Clinical Research Programme (MLW) established the Journalist in Residence Programme (JIR) to train journalists and give them support when reporting health research stories for MLW. However, the programme had not been evaluated since its inception. We assessed the impact of media training and support through this programme on the development of health research reporting in Malawi. Methods: Qualitative approaches were used to conduct in-depth interviews with five journalists, two editors, two consultants, five researchers, and three Ministry of Health officials. Two focus group discussions were also conducted with selected Community Advisory Group members in Blantyre and Chikwawa. All interviews and focus group discussions were recorded and transcribed verbatim. Data analysis was conducted using Nvivo11. Results: The JIR programme has a positive impact on the development of health research reporting in Malawi. The quantity and quality of health research stories generated from Malawi Liverpool Wellcome Trust research have increased after introducing the programme. Additionally, journalists are motivated to write health research stories as they get training and support through the programme. The health research work being implemented by Malawi-Liverpool Wellcome Trust is appreciated by policy makers and the public at large through stories published by journalists in the programme. Conclusions: This evaluation found that the JIR programme is a powerful tool for achieving the development of health research reporting in Malawi.

Organizational contexts, implementation process, and capacity outcomes of multicultural, multilingual Home-Based Programs in public initiatives: A Mixed-Methods study

AIMS

The study aim was to examine the impact of a home-based programme intervention on organizational contexts, implementation processes and organizational capacity outcomes from multicultural, multilingual participants working at community-based organizations.

DESIGN

This was a sequential exploratory, mixed-methods longitudinal study using community-based participatory research principles.

SAMPLE

Twenty participants from nine multicultural, multilingual community-based organizations were in this public health initiative's intervention to develop community-designed, home-based programmes.

METHODS

Capacity building providers delivered the intervention selected by the funders. Workshop outcomes were descriptively measured in April/May 2019. In April/May and November 2019, participants completed surveys about organizational contexts, implementation processes and organizational capacity outcomes, which were analysed with t-tests using the organization as the unit of analysis. Qualitative data were analysed using content analysis.

RESULTS

Seven programmes were new and two were modified. As workshop outcomes, 59% of participants reported increased overall implementation knowledge and 74% reported capacity building providers as the most helpful resource. After 6 to 7 months, no statistically significant changes were noted in organizational contexts, implementation processes or organizational capacity outcomes. Participants benefited from capacity building because they had programmes developed, formed partnerships with capacity building providers, gained implementation knowledge, and engaged in networking.

CONCLUSION

Participants reported excellent individual and organizational strengths. Many Initiative factors contributed to no statistical changes. Namely, there was no opportunity for baseline data; limited community-based organization engagement in the intervention model selection, timeline and processes; the Initiative's timeline did not fit participants' timeline; insufficient time to develop culturally and linguistically appropriate programmes; late literature review abstracts; lack of adequate, planful and paid capacity building time; and a contract requirement to have the programme due when it was not implementable. These Initiative design factors, as reported by participants, limited the Initiative's home-based programme development.

IMPACT

This study highlights the strengths of participants, community-based organizations and capacity building providers. Model selection, timeline and budget were identified as key factors for equitable implementation in multicultural, multilingual organizations.

Managing community engagement in research in Uganda: insights from practices in HIV/AIDS research

Background

Community engagement (CE) in research is valuable for instrumental and intrinsic reasons. Despite existing guidance on how to ensure meaningful CE, much of what it takes to achieve this goal differs across settings. Considering the emerging trend towards mandating CE in many research studies, this study aimed at documenting how CE is conceptualized and implemented, and then providing context-specific guidance on how researchers and research regulators in Uganda could think about and manage CE in research.

Methods

We conducted qualitative interviews and focus group discussions involving forty-one respondents who were experienced in HIV/AIDS biomedical research involving CE. Thirty-eight of these were directly or indirectly associated with Uganda’s leading research institution in the field of HIV/AIDS. They included Principal Investigators, Community Liaisons Officers, Research Ethics Committee members and Community Advisory Board Members. Three respondents were from Uganda National Council for Science and Technology. Data were collected between August 2019 and August 2020, using audio-taped focus group discussions and key informant interviews, transcribed and analyzed manually to generate themes and subthemes.

Results

Three major themes emerged: goals or value of CE; the means of CE, and, the evaluation of CE. Goals or value of CE generated four subthemes representing the overarching goals of CE: (1) Promote communities’ agency; (2) Generate and sustain trust; (3) Protect and promote communities’ rights and interests; and, (4) Help studies optimize participation in the form of enrolment and retention of participants. What usually comes under the nomenclatures of methods, strategies, and approaches of CE, such as town-hall meetings, sports events, drama, and the like, should simply be understood as the means of CE, and it is not desirable to hold pre-conceived and fixed ideas about the best means to conduct CE in research since a lot depend on the context. Finally, the study found that despite CE’s critical importance, which suggests the need to track and evaluate it, CE is currently intermittently evaluated, and for inadequate motivations.

Conclusions

Existing guidance on how to conduct robust CE in research is no substitute for creativity, flexibility, and reflexivity on the part of both researchers and research regulators.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12910-022-00797-6.

Monitoring and Evaluation of Patient Engagement in Health Product Research and Development: Co-Creating a Framework for Community Advisory Boards

PURPOSE

While patient engagement is becoming more customary in developing health products, its monitoring and evaluation to understand processes and enhance impact are challenging. This article describes a patient engagement monitoring and evaluation (PEME) framework, co-created and tailored to the context of community advisory boards (CABs) for rare diseases in Europe. It can be used to stimulate learning and evaluate impacts of engagement activities.

METHODS

A participatory approach was used in which data collection and analysis were iterative. The process was based on the principles of interactive learning and action and guided by the PEME framework. Data were collected via document analysis, reflection sessions, a questionnaire, and a workshop.

RESULTS

The tailored framework consists of a theory of change model with metrics explaining how CABs can reach their objectives. Of 61 identified metrics, 17 metrics for monitoring the patient engagement process and short-term outcomes were selected, and a "menu" for evaluating long-term impacts was created. Example metrics include "Industry representatives' understanding of patients' unmet needs;" "Feeling of trust between stakeholders;" and "Feeling of preparedness." "Alignment of research programs with patients' needs" was the highest-ranked metric for long-term impact.

CONCLUSIONS

Findings suggest that process and short-term outcome metrics could be standardized across CABs, whereas long-term impact metrics may need to be tailored to the collaboration from a proposed menu. Accordingly, we recommend that others adapt and refine the PEME framework as appropriate. The next steps include implementing and testing the evaluation framework to stimulate learning and share impacts.

'Working relationships' across difference - a realist review of community engagement with malaria research

Background: Community engagement (CE) is increasingly accepted as a critical aspect of health research, because of its potential to make research more ethical, relevant and well implemented. While CE activities linked to health research have proliferated in Low and Middle Income Countries (LMICs), and are increasingly described in published literature, there is a lack of conceptual clarity around how engagement is understood to 'work', and the aims and purposes of engagement are varied and often not made explicit. Ultimately, the evidence base for engagement remains underdeveloped. Methods: To develop explanations for how and why CE with health research contributes to the pattern of outcomes observed in published literature , we conducted a realist review of CE with malaria research - a theory driven approach to evidence synthesis. Results: We found that community engagement relies on the development of provisional 'working relationships' across differences, primarily of wealth, power and culture. These relationships are rooted in interactions that are experienced as relatively responsive and respectful, and that bring tangible research related benefits. Contextual factors affecting development of working relationships include the facilitating influence of research organisation commitment to and resources for engagement, and constraining factors linked to the prevailing 'dominant health research paradigm context', such as: differences of wealth and power between research centres and local populations and health systems; histories of colonialism and vertical health interventions; and external funding and control of health research. Conclusions: The development of working relationships contributes to greater acceptance and participation in research by local stakeholders, who are particularly interested in research related access to health care and other benefits. At the same time, such relationships may involve an accommodation of some ethically problematic characteristics of the dominant health research paradigm, and thereby reproduce this paradigm rather than challenge it with a different logic of collaborative partnership.

Developing a programme theory for a transdisciplinary research collaboration: Complex Urban Systems for Sustainability and Health

Background: Environmental improvement is a priority for urban sustainability and health and achieving it requires transformative change in cities. An approach to achieving such change is to bring together researchers, decision-makers, and public groups in the creation of research and use of scientific evidence. Methods: This article describes the development of a programme theory for Complex Urban Systems for Sustainability and Health (CUSSH), a four-year Wellcome-funded research collaboration which aims to improve capacity to guide transformational health and environmental changes in cities. Results: Drawing on ideas about complex systems, programme evaluation, and transdisciplinary learning, we describe how the programme is understood to "work" in terms of its anticipated processes and resulting changes. The programme theory describes a chain of outputs that ultimately leads to improvement in city sustainability and health (described in an 'action model'), and the kinds of changes that we expect CUSSH should lead to in people, processes, policies, practices, and research (described in a 'change model'). Conclusions: Our paper adds to a growing body of research on the process of developing a comprehensive understanding of a transdisciplinary, multiagency, multi-context programme. The programme theory was developed collaboratively over two years. It involved a participatory process to ensure that a broad range of perspectives were included, to contribute to shared understanding across a multidisciplinary team. Examining our approach allowed an appreciation of the benefits and challenges of developing a programme theory for a complex, transdisciplinary research collaboration. Benefits included the development of teamworking and shared understanding and the use of programme theory in guiding evaluation. Challenges included changing membership within a large group, reaching agreement on what the theory would be 'about', and the inherent unpredictability of complex initiatives.

Developing a programme theory for a transdisciplinary research collaboration: Complex Urban Systems for Sustainability and Health

Background: Environmental improvement is a priority for urban sustainability and health and achieving it requires transformative change in cities. An approach to achieving such change is to bring together researchers, decision-makers, and public groups in the creation of research and use of scientific evidence.

Methods: This article describes the development of a programme theory for Complex Urban Systems for Sustainability and Health (CUSSH), a four-year Wellcome-funded research collaboration which aims to improve capacity to guide transformational health and environmental changes in cities.

Results: Drawing on ideas about complex systems, programme evaluation, and transdisciplinary learning, we describe how the programme is understood to “work” in terms of its anticipated processes and resulting changes. The programme theory describes a chain of outputs that ultimately leads to improvement in city sustainability and health (described in an ‘action model’), and the kinds of changes that we expect CUSSH should lead to in people, processes, policies, practices, and research (described in a ‘change model’).

Conclusions: Our paper adds to a growing body of research on the process of developing a comprehensive understanding of a transdisciplinary, multiagency, multi-context programme. The programme theory was developed collaboratively over two years. It involved a participatory process to ensure that a broad range of perspectives were included, to contribute to shared understanding across a multidisciplinary team. Examining our approach allowed an appreciation of the benefits and challenges of developing a programme theory for a complex, transdisciplinary research collaboration. Benefits included the development of teamworking and shared understanding and the use of programme theory in guiding evaluation. Challenges included changing membership within a large group, reaching agreement on what the theory would be ‘about’, and the inherent unpredictability of complex initiatives.

Evaluation of patient engagement in medicine development: A multi-stakeholder framework with metrics

BACKGROUND

Patient engagement is becoming more customary in medicine development. However, embedding it in organizational decision-making remains challenging, partly due to lack of agreement on its value and the means to evaluate it. The objective of this project was to develop a monitoring and evaluation framework, with metrics, to demonstrate impact and enhance learning.

METHODS

A consortium of five patient groups, 15 biopharmaceutical companies and two academic groups iteratively created a framework in a multi-phase participatory process, including analysis of its application in 24 cases.

RESULTS

The framework includes six components, with 87 metrics and 15 context factors distributed among (sub)components: (a) Input: expectations, preparations, resources, representativeness of stakeholders; (b) Activities/process: structure, management, interactions, satisfaction; (c) Learnings and changes; (d) Impacts: research relevance, study ethics and inclusiveness, study quality and efficiency, quality of evidence and uptake of products, empowerment, reputation and trust, embedding of patient engagement; (e) Context: policy, institutional, community, decision-making contextual factors. Case study findings show a wide variation in use of metrics. There is no 'one size fits all' set of metrics appropriate for every initiative or organization. Presented sample sets of metrics can be tailored to individual situations.

CONCLUSION

Introducing change into any process is best done when the value of that change is clear. This framework allows participants to select what metrics they value and assess to what extent patient engagement has contributed.

PATIENT CONTRIBUTION

Five patient groups were involved in all phases of the study (design, conduct, interpretation of data) and in writing the manuscript.

Optimizing the development and evaluation of complex interventions: lessons learned from the BetterBirth Program and associated trial

BACKGROUND

Despite extensive efforts to develop and refine intervention packages, complex interventions often fail to produce the desired health impacts in full-scale evaluations. A recent example of this phenomenon is BetterBirth, a complex intervention designed to implement the World Health Organization's Safe Childbirth Checklist and improve maternal and neonatal health. Using data from the BetterBirth Program and its associated trial as a case study, we identified lessons to assist in the development and evaluation of future complex interventions.

METHODS

BetterBirth was refined across three sequential development phases prior to being tested in a matched-pair, cluster randomized trial in Uttar Pradesh, India. We reviewed published and internal materials from all three development phases to identify barriers hindering the identification of an optimal intervention package and identified corresponding lessons learned. For each lesson, we describe its importance and provide an example motivated by the BetterBirth Program's development to illustrate how it could be applied to future studies.

RESULTS

We identified three lessons: (1) develop a robust theory of change (TOC); (2) define optimization outcomes, which are used to assess the effectiveness of the intervention across development phases, and corresponding criteria for success, which determine whether the intervention has been sufficiently optimized to warrant full-scale evaluation; and (3) create and capture variation in the implementation intensity of components. When applying these lessons to the BetterBirth intervention, we demonstrate how a TOC could have promoted more complete data collection. We propose an optimization outcome and related criteria for success and illustrate how they could have resulted in additional development phases prior to the full-scale trial. Finally, we show how variation in components' implementation intensities could have been used to identify effective intervention components.

CONCLUSION

These lessons learned can be applied during both early and advanced stages of complex intervention development and evaluation. By using examples from a real-world study to demonstrate the relevance of these lessons and illustrating how they can be applied in practice, we hope to encourage future researchers to collect and analyze data in a way that promotes more effective complex intervention development and evaluation.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT02148952; registered on May 29, 2014.

A theory of change for community interventions to prevent domestic violence against women and girls in Mumbai, India

Background: We describe the development of a theory of change for community mobilisation activities to prevent violence against women and girls. These activities are part of a broader program in urban India that works toward primary, secondary, and tertiary prevention of violence and includes crisis response and counselling and medical, police, and legal assistance. Methods: The theory of change was developed in five phases, via expert workshops, use of primary data, recurrent team meetings, adjustment at further meetings and workshops, and a review of published theories. Results: The theory summarises inputs for primary and secondary prevention, consequent changes (positive and negative), and outcomes. It is fully adapted to the program context, was designed through an extended consultative process, emphasises secondary prevention as a pathway to primary prevention, and integrates community activism with referral and counselling interventions. Conclusions: The theory specifies testable causal pathways to impact and will be evaluated in a controlled trial.

A theory of change for community interventions to prevent domestic violence against women and girls in Mumbai, India

Background: We describe the development of a theory of change for community mobilisation activities to prevent violence against women and girls. These activities are part of a broader program in urban India that works toward primary, secondary, and tertiary prevention of violence and includes crisis response and counselling and medical, police, and legal assistance. Methods: The theory of change was developed in five phases, via expert workshops, use of primary data, recurrent team meetings, adjustment at further meetings and workshops, and a review of published theories. Results: The theory summarises inputs for primary and secondary prevention, consequent changes (positive and negative), and outcomes. It is fully adapted to the program context, was designed through an extended consultative process, emphasises secondary prevention as a pathway to primary prevention, and integrates community activism with referral and counselling interventions. Conclusions: The theory specifies testable causal pathways to impact and will be evaluated in a controlled trial.

Evolution of a programme to engage school students with health research and science in Kenya

Facilitating mutually-beneficial educational activities between researchers and school students is an increasingly popular way for research institutes to engage with communities who host health research, but these activities have rarely been formally examined as a community or public engagement approach in health research. The KEMRI-Wellcome Trust Research Programme (KWTRP) in Kilifi, Kenya, through a Participatory Action Research (PAR) approach involving students, teachers, researchers and education stakeholders, has incorporated 'school engagement' as a key component into their community engagement (CE) strategy. School engagement activities at KWTRP aim at strengthening the ethical practice of the institution in two ways: through promoting an interest in science and research among school students as a form of benefit-sharing; and through creating forums for dialogue aimed at promoting mutual understanding between researchers and school students. In this article, we provide a background of CE in Kilifi and describe the diverse ways in which health researchers have engaged with communities and schools in different parts of the world. We then describe the way in which the KWTRP school engagement programme (SEP) was developed and scaled-up. We conclude with a discussion about the challenges, benefits and lessons learnt from the SEP implementation and scale-up in Kilifi, which can inform the establishment of SEPs in other settings.