Building evaluation capacity in Ontario's public health units: promising practices and strategies

Examining the competencies required by evaluation capacity builders in community-based organizations

Increasing demand for evidence generated through program evaluation has led many community-based organizations (CBOs) to seek external support for evaluation capacity building (ECB). However, studies have yet to explore the essential competencies required by evaluation capacity builders working in the community sector. Our qualitative study aimed to examine the perceptions of ECB practitioners (n = 12) regarding essential competencies for building evaluation capacity in this sector. Our findings reveal that ECB practice requires competencies not found in known evaluation competency frameworks, such as instructional design, knowledge of organizational change models, motivating stakeholders, and understanding of the community sector. Our findings provide valuable information to help guide future education and training related to building the evaluation capacity of community organizations.

Laying a Solid Foundation for the Next Generation of Evaluation Capacity Building: Findings from an Integrative Review

Evaluation capacity building (ECB) continues to attract the attention and interest of scholars and practitioners. Over the years, models, frameworks, strategies, and practices related to ECB have been developed and implemented. Although ECB is highly contextual, the evolution of knowledge in this area depends on learning from past efforts in a structured approach. The purpose of the present article is to integrate the ECB literature in evaluation journals. More specifically, the article aims to answer three questions: What types of articles and themes comprise the current literature on ECB? How are current practices of ECB described in the literature? And what is the current status of research on ECB? Informed by the findings of the review, the article concludes with suggestions for future ECB practice and scholarship.

Validating the evaluation capacity scale among practitioners in non-governmental organizations

The growing emphasis on demonstrating the effectiveness of social services through evaluation has heightened demand for nongovernmental organization (NGO) practitioners to enhance evaluation capacity. However, a lack of validated instruments in the NGO context has hampered efforts to assess NGO practitioners' current evaluation capacity and understand how capacity-building activities could be tailored to meet NGO practitioners' actual needs and enhance their evaluation capacity. Hence, this study aims to develop the Evaluation Capacity Scale (ECS), a self-reporting instrument of NGO practitioners' capacity to conduct an effective evaluation of their service programs. Validation data was derived from 439 NGO practitioners who attended the Jockey Club MEL Institute Project in Hong Kong, China. Exploratory factor analysis of the ECS revealed three factors-evaluation mindset, evaluation implementation, and evaluation communication-and confirmatory factor analysis further validated this three-factor structure. Moreover, MANCOVA analysis demonstrated the ECS's predictive validity. Overall, the ECS demonstrated satisfactory convergent validity, high internal consistency reliability, and predictive validity, and its factor structure was supported in subgroups based on gender, age, and level of education. Theoretical and practical implications of the findings are discussed.

Utilizing Causal Loop Diagramming to Explore a Research and Evaluation Capacity Building Partnership

The capacity to engage in research, evaluation and evidence-informed decision-making supports effective public health policy and practice. Little is known about partnership-based approaches that aim to build capacity across a system or how to evaluate them. This study examines the impacts of a research and evaluation capacity building partnership called the Western Australian Sexual Health and Blood-borne Virus Applied Research and Evaluation Network (hereafter, SiREN). SiREN aims to strengthen capacity across a system of clinical and medical services and government and non-government organizations. These organizations are connected through their shared aim of preventing and managing sexually transmissible infections and blood-borne viruses. To examine SiREN, systems concepts and methods were used. Data were collected from SiREN organizational documents (n = 42), a survey tool (n = 104), in-depth interviews (n = 17), a workshop and three meetings with SiREN stakeholders and used to develop two causal loop diagrams. Findings show engagement with SiREN was influenced by a complex interplay of contextual (e.g., organizational capacity) and process (e.g., presence of trusting relationships) factors. SiREN contributed to system level changes, including increased resources for research and evaluation, the development of networks and partnerships that led to more efficient responses to emerging health issues, evidence sharing, and sustainable research and evaluation practice. The use of causal loop diagrams enabled the identification of key leverage points that SiREN can use for continuous improvement or evaluation. The focus on how contextual factors influenced SiREN's ability to create change provides valuable information for researchers, policymakers or practitioners seeking to develop a similar partnership.

Critical factors that affect the functioning of a research and evaluation capacity building partnership: A causal loop diagram

Introduction

Public health policy and practice is strengthened by the application of quality evidence to decision making. However, there is limited understanding of how initiatives that support the generation and use of evidence in public health are operationalised. This study examines factors that support the internal functioning of a partnership, the Western Australian Sexual Health and Blood-borne Virus Applied Research and Evaluation Network (SiREN). SiREN aims to build research and evaluation capacity and increase evidence-informed decision making in a public health context.

Methods

This study was informed by systems concepts. It developed a causal loop diagram, a type of qualitative system model that illustrated the factors that influence the internal operation of SiREN. The causal loop diagram was developed through an iterative and participatory process with SiREN staff and management (n = 9) via in-depth semi-structured interviews (n = 4), workshops (n = 2), and meetings (n = 6).

Results

Findings identified critical factors that affected the functioning of SiREN. Central to SiREN’s ability to meet its aims was its capacity to adapt within a dynamic system. Adaptation was facilitated by the flow of knowledge between SiREN and system stakeholders and the expertise of the team. SiREN demonstrated credibility and capability, supporting development of new, and strengthening existing, partnerships. This improved SiREN’s ability to be awarded new funding and enhanced its sustainability and growth. SiREN actively balanced divergent stakeholder interests to increase sustainability.

Conclusion

The collaborative development of the diagram facilitated a shared understanding of SiREN. Adaptability was central to SiREN achieving its aims. Monitoring the ability of public health programs to adapt to the needs of the systems in which they work is important to evaluate effectiveness. The detailed analysis of the structure of SiREN and how this affects its operation provide practical insights for those interested in establishing a similar project.

Chronic disease prevention evaluation in Ontario's public health system: a qualitative needs assessment

OBJECTIVES

Building evaluation capacity for chronic disease prevention (CDP) is a critical step in ensuring the effectiveness of CDP programming over time. In this article, we highlight the findings of the qualitative arm of a mixed-methods needs assessment designed to assess the gaps and areas of strength within Ontario's public health system with respect to CDP evaluation.

METHODS

We conducted 29 interviews and focus groups with representatives from 25 public health units (PHUs) and analyzed the data using thematic analysis. We sought to understand what gaps and challenges exist in the Ontario public health system around CDP evaluation.

RESULTS

Challenges facing Ontario's PHUs in CDP evaluation include variation and centralization of capacity to evaluate, as well as competing priorities limiting the development of evaluative thinking. Participating PHUs identified the need for evaluation capacity building (ECB) strategies grounded in an understanding of the unique contexts in which they work and a desire for guidance in conducting a complex and thoughtful evaluation. Moving forward, PHUs noted a desire for a strong system of knowledge sharing and consultation across the public health system, including through strengthening existing partnerships with community collaborators.

CONCLUSION

These results support the case for ECB strategies that are adaptive and context-sensitive and equip PHUs with the skills required to evaluate complex CDP programming.