Involving program stakeholders in reviews of evaluators' recommendations for program revisions

The extent to which the recommendations issued after the periodic evaluation of university programs are targeted, specific, measurable and unequivocal: An exploratory analysis

BACKGROUND AND STUDY OBJECTIVES

Each year, Quebec universities conduct a periodic evaluation of their study programs and issue recommendations to improve the quality and relevance of these latter. Although these evaluations are commonplace in higher education institutions, there is little information about them and the resulting recommendations. To compensate this lack of convincing data, our exploratory analysis will look at the quality of expression of these recommendations.

METHODOLOGY

Experts were solicited to evaluate whether the recommendations were targeted, specific, measurable and unequivocal, four strategic characteristics of good formulations.

RESULTS

Our results reveal major issues. Almost all of the recommendations do not indicate who is to carry them out. Also, the desired improvement is not clearly identified, thus leaving unanswered major questions about how they should be implemented. Another weak point was the absence of indicators for measuring the accomplishment of the recommendations. The last part of this study presents the best practices for issuing better recommendations, since these latter are key in the improvement process and the development of action plans. This study opens the way for research that will examine the root causes of the problem and make it possible to establish best practices for writing recommendations.

Participation by different stakeholders in participatory evaluation of health promotion: a literature review

Participatory evaluation has been increasingly used in health promotion (HP) and various forms of participatory evaluation have been put into practice. Simultaneously, the concept of participation has become more important for evaluation research in general, which is equally diverse and the subject of various discourses. This study addresses the issue of how the concept of participation has been established in HP evaluation practice. An analytical framework was developed, which served as a basis for a literature review, but can also be used as a general framework for analyzing and planning the scope of participation by various stakeholders within different phases of participatory evaluation. Three dimensions of participation, which refer to decision making (decision power, deliberation) and action processes are distinguished. The results show that only a few articles discussed participatory evaluation processes and participatory (evaluation) research was largely put forth by participatory (action) research in communities. The articles analyzed referred mostly to three stakeholder groups - evaluators, program staff and beneficiaries - and to participation processes in the initial evaluation phases. The application of the framework revealed that decision power seems to be held predominantly by program staff, evaluators seem to be more involved in action processes and beneficiaries in deliberation processes.

Program implementers' evaluation of the project P.A.T.H.S.: findings based on different datasets over time

This paper integrates the evaluation findings based on program implementers in nine datasets collected from 2005 to 2009 (244 schools and 7,926 implementers). Using consolidated data with schools as the unit of analysis, results showed that program implementers generally had positive perceptions of the program, themselves, and benefits of the program, with more than four-fifths of the implementers regarding the program as beneficial to the program participants. The subjective outcome evaluation instrument was found to be internally consistent. Multiple regression analyses revealed that perceived qualities of the program and program implementers predicted perceived effectiveness of the program. In conjunction with evaluation findings based on other sources, the present study provides support for the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) in Hong Kong.

Optimizing an immersion ESL curriculum using analytic hierarchy process

The main purpose of this study is to fill a substantial knowledge gap regarding reaching a uniform group decision in English curriculum design and planning. A comprehensive content-based course criterion model extracted from existing literature and expert opinions was developed. Analytical hierarchy process (AHP) was used to identify the relative importance of course criteria for the purpose of tailoring an optimal one-week immersion English as a second language (ESL) curriculum for elementary school students in a suburban county of Taiwan. The hierarchy model and AHP analysis utilized in the present study will be useful for resolving several important multi-criteria decision-making issues in planning and evaluating ESL programs. This study also offers valuable insights and provides a basis for further research in customizing ESL curriculum models for different student populations with distinct learning needs, goals, and socioeconomic backgrounds.

The unique character of involvement in multi-site evaluation settings

As the number of large federal programs increases, so, too, does the need for a more complete understanding of how to conduct evaluations of such complex programs. The research literature has documented the benefits of stakeholder participation in smaller-scale program evaluations. However, given the scope and diversity of projects in multi-site program evaluations, traditional notions of participatory evaluation do not apply. The purpose of this research is to determine the ways in which stakeholders are involved in large-scale, multi-site STEM evaluations. This article describes the findings from a survey of 313 program leaders and evaluators and from follow-up interviews with 12 of these individuals. Findings from this study indicate that attendance at meetings and conferences, planning discussions within the project related to use of the program evaluation, and participation in data collection should be added to the list of activities that foster feelings of evaluation involvement among stakeholders. In addition, perceptions of involvement may vary according to breadth or depth of evaluation activities, but not always both. Overall, this study suggests that despite the contextual challenges of large, multi-site evaluations, it is feasible to build feelings of involvement among stakeholders.