Understanding the factors that influence health promotion evaluation: The development and validation of the evaluation practice analysis survey

Laying a Solid Foundation for the Next Generation of Evaluation Capacity Building: Findings from an Integrative Review

Evaluation capacity building (ECB) continues to attract the attention and interest of scholars and practitioners. Over the years, models, frameworks, strategies, and practices related to ECB have been developed and implemented. Although ECB is highly contextual, the evolution of knowledge in this area depends on learning from past efforts in a structured approach. The purpose of the present article is to integrate the ECB literature in evaluation journals. More specifically, the article aims to answer three questions: What types of articles and themes comprise the current literature on ECB? How are current practices of ECB described in the literature? And what is the current status of research on ECB? Informed by the findings of the review, the article concludes with suggestions for future ECB practice and scholarship.

How practitioner, organisational and system-level factors act to influence health promotion evaluation capacity: Validation of a conceptual framework

The need to improve the practice and quality of evaluation in the health promotion and disease prevention field is widely recognised. In order to plan, implement and evaluate health promotion evaluation capacity building efforts, there is a need to better understand the practitioner, organisational and system-level determinants of evaluation capacity and practice. This study aimed to assess the validity Evaluation Practice Analysis Survey (EPAS) constructs using confirmatory factor analysis and validate a conceptual framework of health promotion evaluation capacity using path analysis. Experienced Australian health promotion practitioners completed the survey (n = 219). Twenty-one of the original 23 EPAS scales were assessed as reliable and valid. The final model was found to have good fit (χ²₁₄ = 18.72, p = 0.18, root mean square error of approximation = 0.04, 90% CI 0.00-0.82, Comparative Fit Index = 1.00, standardised root mean square residual = 0.04). This model supports the role of the organisation in facilitating evaluation practice through leadership, culture, systems, support and resources. It builds on existing frameworks from other fields to incorporate political, funding and administrative factors. This study provides an evidence-based model of evaluation capacity that organisations, funders and policy makers can use to plan and implement more effective evaluation capacity building strategies within organisations and the wider prevention field.

An assessment of program evaluation methods and quality in Australian prevention agencies

ISSUE ADDRESSED

This study aimed to examine evaluation methods and quality in Australian health promotion agencies and the factors associated with this. The evidence base for prevention strategies is limited, with the evidence generated through program evaluation by health promotion and disease prevention agencies lacking rigour. Despite the need to improve the quality of evaluation, there is limited evidence of what influences evaluation quality in the prevention field.

METHODS

Data were collected using the Evaluation Practice Analysis Survey and an audit and appraisal of evaluation reports. Descriptive analysis was used to examine evaluation characteristics and multivariable regression was used to explore the association between evaluation and organisational attributes and evaluation quality.

RESULTS

In total, 392 evaluation reports were reviewed from 78 government and non-government agencies. Process evaluation was conducted most frequently, followed by impact evaluation. Overall evaluation quality was low (median 24.5%). In multivariable regression analysis, only two factors were associated with evaluation quality: health promotion budget (ratio of geometric means 1.53 [95% CI 1.02-2.29]); and, conducting statewide or national prevention programs (1.38 [95% CI 1.05-1.82]).

CONCLUSIONS

The findings show that the potential to improve evaluation quality is greatest in smaller organisations that deliver health promotion at a local or regional scale. SO WHAT?: By improving the rigour of existing evaluation, there is opportunity to build the evidence base for prevention strategies, which highlights the importance of embedding the enablers of program learning and evidence generation within health promotion and prevention organisations.