The Relationships Between State Health Department Practitioners' Perceptions of Organizational Supports and Evidence-Based Decision-Making Skills

OBJECTIVES

Evidence-based decision making (EBDM) allows public health practitioners to implement effective programs and policies fitting the preferences of their communities. To engage in EBDM, practitioners must have skills themselves, their agencies must engage in administrative evidence-based practices (A-EBPs), and leaders must encourage the use of EBDM. We conducted this longitudinal study to quantify perceptions of individual EBDM skills and A-EBPs, as well as the longitudinal associations between the 2.

METHODS

An online survey completed among US state health department practitioners in 2016 and 2018 assessed perceptions of respondents' skills in EBDM and A-EBPs. We used χ2 tests, t tests, and linear regressions to quantify changes over time, differences by demographic characteristics, and longitudinal associations between individual skills and A-EBPs among respondents who completed both surveys (N = 336).

RESULTS

Means of most individual EBDM skills and A-EBPs did not change significantly from 2016 to 2018. We found significant positive associations between changes in A-EBPs and changes in EBDM skill gaps: for example, a 1-point increase in the relationships and partnerships score was associated with a narrowing of the EBDM skill gap (β estimate = 0.38; 95% CI, 0.15-0.61). At both time points, perceived skills and A-EBPs related to financial practices were low.

CONCLUSIONS

Findings from this study can guide the development and dissemination of initiatives designed to simultaneously improve individual and organizational capacity for EBDM in public health settings. Future studies should focus on types of strategies most effective to build capacity in particular types of agencies and practitioners, to ultimately improve public health practice.

Patient navigation programs in Alberta, Canada: an environmental scan

BACKGROUND

Patient navigation is a complex intervention that has garnered substantial interest and investment across Canada. We conducted an environmental scan to understand the landscape of patient navigation programs within the health care system in Alberta, Canada.

METHODS

We included patient navigation programs within Alberta Health Services (AHS) and Alberta's Primary Care Networks (PCNs). Key informants were asked in October 2016 to identify existing programs and their corresponding program contacts. These program contacts were invited to complete a telephone-based survey from October 2016 to July 2017, to provide program descriptions and eligibility criteria, and to identify gaps in navigation. Programs were included if they engaged patients on an individual basis, and either facilitated continuity of care or promoted patient and family empowerment. We tabulated results and calculated summary statistics for program characteristics.

RESULTS

Ninety-five potentially eligible programs were identified by key informants. The response rate to the study survey was 73% (n = 69). After excluding programs not meeting inclusion criteria, we included a total of 58 programs in the study: 43 AHS programs and 15 PCN programs. Nearly all programs (93%, n = 54) delivered navigation via an individual acting as a navigator. A minority of programs also included nonnavigator components, such as Web-based resources (7%, n = 4) and process or structural changes to facilitate navigation (22%, n = 13). Certain patient subgroups were particularly well-served by patient navigation; these included patients with cancer, substance use disorders or mental health concerns, and pediatric patients. Gaps identified in navigation fell under 4 domains: awareness, resources, geographic distribution and integration.

INTERPRETATION

Patient navigation programs are common and have extended beyond cancer care, from which the construct originated; however, gaps include a lack of awareness and inequitable access to the programs. These findings will be of interest to those developing and implementing patient navigation interventions in Alberta and other jurisdictions.

An immersive clinical experience to create sustainable clinical learning opportunities for nurse practitioner students

ABSTRACT

A significant increase in nurse practitioner program enrollment has resulted in a critical need for committed preceptors willing to offer clinical learning opportunities for students. Academic practice partnerships have been proposed as a strategy to address the preceptorship crisis. This project evaluated the impact of a longitudinal immersive clinical experience designed for nurse practitioner students in collaboration with two federally qualified health centers and an underserved clinic within an academic health center. The study used a pre- and postsurvey method to determine the impact of an immersive clinical experience on preceptor perceptions of benefits and rewards, support, and commitment to the role. A statistically significant increase in preceptor perception of support was noted in three areas. Participants reported increased opportunity to share information with other preceptors, improved support from nursing faculty to help identify students' performance problems, and greater clarity in faculty responsibilities related to the preceptor role. Increases in perceptions of benefits and rewards and commitment to the preceptor role did not reach statistical significance. Immersive clinical experiences have the potential to improve the levels of preceptor support, commitment, and perceived opportunities for benefits and rewards, which may facilitate the development of high-quality, sustainable clinical learning opportunities for nurse practitioner students.

Methods to measure effects of social accountability interventions in reproductive, maternal, newborn, child, and adolescent health programs: systematic review and critique

BACKGROUND

There is no agreed way to measure the effects of social accountability interventions. Studies to examine whether and how social accountability and collective action processes contribute to better health and healthcare services are underway in different areas of health, and health effects are captured using a range of different research designs.

OBJECTIVES

The objective of our review is to help inform evaluation efforts by identifying, summarizing, and critically appraising study designs used to assess and measure social accountability interventions' effects on health, including data collection methods and outcome measures. Specifically, we consider the designs used to assess social accountability interventions for reproductive, maternal, newborn, child, and adolescent health (RMNCAH).

DATA SOURCES

Data were obtained from the Cochrane Library, EMBASE, MEDLINE, SCOPUS, and Social Policy & Practice databases.

ELIGIBILITY CRITERIA

We included papers published on or after 1 January 2009 that described an evaluation of the effects of a social accountability intervention on RMNCAH.

RESULTS

Twenty-two papers met our inclusion criteria. Methods for assessing or reporting health effects of social accountability interventions varied widely and included longitudinal, ethnographic, and experimental designs. Surprisingly, given the topic area, there were no studies that took an explicit systems-orientated approach. Data collection methods ranged from quantitative scorecard data through to in-depth interviews and observations. Analysis of how interventions achieved their effects relied on qualitative data, whereas quantitative data often raised rather than answered questions, and/or seemed likely to be poor quality. Few studies reported on negative effects or harms; studies did not always draw on any particular theoretical framework. None of the studies where there appeared to be financial dependencies between the evaluators and the intervention implementation teams reflected on whether or how these dependencies might have affected the evaluation. The interventions evaluated in the included studies fell into the following categories: aid chain partnership, social audit, community-based monitoring, community-linked maternal death review, community mobilization for improved health, community reporting hotline, evidence for action, report cards, scorecards, and strengthening health communities.

CONCLUSIONS

A wide range of methods are currently being used to attempt to evaluate effects of social accountability interventions. The wider context of interventions including the historical or social context is important, as shown in the few studies to consider these dimensions. While many studies collect useful qualitative data that help illuminate how and whether interventions work, the data and analysis are often limited in scope with little attention to the wider context. Future studies taking into account broader sociopolitical dimensions are likely to help illuminate processes of accountability and inform questions of transferability of interventions. The review protocol was registered with PROSPERO (registration # CRD42018108252).

Development of a National Dietary Behaviors Questionnaire for EFNEP Adult Participants

OBJECTIVE

Develop and establish the reliability and validity of dietary behavior evaluation questions for the Expanded Food and Nutrition Education Program (EFNEP).

DESIGN

A mixed-methods study using cognitive interviews, expert panels, test/retest reliability, and pretests/posttests.

SETTING

14 states across the US.

PARTICIPANTS

A convenience sample of low-income EFNEP or EFNEP-eligible participants for cognitive interviews (n = 111), reliability testing (n = 181), and sensitivity to change testing (n = 382).

MAIN OUTCOMES MEASURES

Indicators of face and content validity, temporal reliability, and sensitivity to change.

ANALYSIS

Questions interpreted as intended in cognitive interviews, intraclass correlation coefficient and Spearman rank-order correlation for reliability testing; paired t tests or Wilcoxon signed-rank tests for sensitivity to change; and exploratory factor analyses to identify possible scales.

RESULTS

Cognitive interviews resulted in 3 rounds of question revisions; reliability value ranges were 0.48-0.77 for intraclass correlation coefficient and 0.43-0.77 for Spearman rank-order correlation. For sensitivity to change, 9 items had evidence of change (P < 0.05) between pretests and posttests, whereas 5 items had evidence for change after removing those with little room to change. Two scales were identified: diet quality and non-cheese dairy.

CONCLUSIONS AND IMPLICATIONS

The EFNEP's new dietary behavior evaluation questions demonstrated face and content validity, moderate to strong reliability, and sensitivity to detect self-reported behavior changes among low-income, diverse populations (culturally, racially/ethnically, and level of education) across 14 states. Nutrition education programs targeting similar behaviors with English speaking clients could consider this dietary behavior questionnaire.

Professionals' experiences of using an improvement programme: applying quality improvement work in preschool contexts

BACKGROUND

Improvement work can be used in preschools to enrich outdoor environment for children's better health. Effective improvement work can facilitate the necessary changes, but little is known about professionals' experiences of participation in improvement interventions. The aim was to evaluate how preschool staff experience quality improvement work, using the Breakthrough Series Collaborative improvement programme, to enhance outdoor environments.

METHODS

An improvement intervention using a breakthrough collaborative was performed at 9 preschools in Sweden and examined with a longitudinal mixed method design. Staff completed questionnaires on 4 occasions (n=45 participants) and interviews took place after the intervention (n=16 participants).

RESULTS

The intervention was successful in the sense that the staff were content with the learning seminars, and they had triggered physical changes in the outdoor environment. They integrated the quality improvement work with their ordinary work and increasingly involved the children. The staff tested improvement tools but did not find them entirely appropriate for their work, because they preferred existing methods for reflection.

CONCLUSIONS

The challenges in quality improvement work seem to be similar across contexts. Using the Breakthrough Series Collaborative in a public health intervention is promising but needs to be integrated with preunderstandings, current reflections and quality tools and models.

Fundamental principles of an effective diabetic retinopathy screening program

BACKGROUND

Diabetic retinopathy (DR) is the leading cause of blindness among working-age adults worldwide. Early detection and treatment are necessary to forestall vision loss from DR.

METHODS

A working group of ophthalmic and diabetes experts was established to develop a consensus on the key principles of an effective DR screening program. Recommendations are based on analysis of a structured literature review.

RESULTS

The recommendations for implementing an effective DR screening program are: (1) Examination methods must be suitable for the screening region, and DR classification/grading systems must be systematic and uniformly applied. Two-field retinal imaging is sufficient for DR screening and is preferable to seven-field imaging, and referable DR should be well defined and reliably identifiable by qualified screening staff; (2) in many countries/regions, screening can and should take place outside the ophthalmology clinic; (3) screening staff should be accredited and show evidence of ongoing training; (4) screening programs should adhere to relevant national quality assurance standards; (5) studies that use uniform definitions of risk to determine optimum risk-based screening intervals are required; (6) technology infrastructure should be in place to ensure that high-quality images can be stored securely to protect patient information; (7) although screening for diabetic macular edema (DME) in conjunction with DR evaluations may have merit, there is currently insufficient evidence to support implementation of programs solely for DME screening.

CONCLUSION

Use of these recommendations may yield more effective DR screening programs that reduce the risk of vision loss worldwide.

Evaluation of the Use of Cancer Registry Data for Comparative Effectiveness Research

Importance

Researchers often analyze cancer registry data to assess for differences in survival among cancer treatments. However, the retrospective, nonrandomized design of these analyses raises questions about study validity.

Objective

To examine the extent to which comparative effectiveness analyses using observational cancer registry data produce results concordant with those of randomized clinical trials.

Design, Setting, and Participants

In this comparative effectiveness study, a total of 141 randomized clinical trials referenced in the National Comprehensive Cancer Network Clinical Practice Guidelines for 8 common solid tumor types were identified. Data on participants within the National Cancer Database (NCDB) diagnosed between 2004 and 2014, matching the eligibility criteria of the randomized clinical trial, were obtained. The present study was conducted from August 1, 2017, to September 10, 2019. The trials included 85 118 patients, and the corresponding NCDB analyses included 1 344 536 patients. Three Cox proportional hazards regression models were used to determine hazard ratios (HRs) for overall survival, including univariable, multivariable, and propensity score-adjusted models. Multivariable and propensity score analyses controlled for potential confounders, including demographic, comorbidity, clinical, treatment, and tumor-related variables.

Main Outcomes and Measures

The main outcome was concordance between the results of randomized clinical trials and observational cancer registry data. Hazard ratios with an NCDB analysis were considered concordant if the NDCB HR fell within the 95% CI of the randomized clinical trial HR. An NCDB analysis was considered concordant if both the NCDB and clinical trial P values for survival were nonsignificant (P ≥ .05) or if they were both significant (P < .05) with survival favoring the same treatment arm in the NCDB and in the randomized clinical trial.

Results

Analyses using the NCDB-produced HRs for survival were concordant with those of 141 randomized clinical trials in 79 univariable analyses (56%), 98 multivariable analyses (70%), and 90 propensity score models (64%). The NCDB analyses produced P values concordant with randomized clinical trials in 58 univariable analyses (41%), 65 multivariable analyses (46%), and 63 propensity score models (45%). No clinical trial characteristics were associated with concordance between NCDB analyses and randomized clinical trials, including disease site, type of clinical intervention, or severity of cancer.

Conclusions and Relevance

The findings of this study suggest that comparative effectiveness research using cancer registry data often produces survival outcomes discordant with those of randomized clinical trial data. These findings may help provide context for clinicians and policy makers interpreting observational comparative effectiveness research in oncology.

The Public Health Uniform National Data System (PHUND$): A Platform for Monitoring Fiscal Health and Sustainability of the Public Health System

CONTEXT

Leaders of government agencies are responsible for stewardship over taxpayer investments. Stewardship strengthens agency performance that is critical to improving population health. Most industries, including health care, and public enterprises, such as education, have policies for uniform data reporting and financial systems for the application of theoretical analytical techniques to organizations and entire systems. However, this is not a mainstreamed practice in local and state government public health.

PROGRAM

The Public Health Uniform National Data System (PHUND$) is a financial information system for local health departments that advances the application of uniform practices to close financial analytical gaps. A 10-year retrospective overview on the development, implementation, and utility of PHUND$ is provided and supported by documented program and agency improvements to validate the analytical features and demonstrate a best practice.

RESULTS

Benefits found from utilizing PHUND$ included reducing financial risks, supporting requests for increased revenues, providing comparative analysis, isolating drivers of costs and deficits, increasing workforce financial management skills, enhancing decision-making processes, and fostering agency sustainability to support continuous improvements in quality and population health. The PHUND$ financial data definitions in the data dictionary provided the structure needed for standardized data collection and confirmed the feasibility of a standardized public health chart of accounts.

CONCLUSION

PHUND$ analysis provided evidence on the relationship between financial and operational performance, as well as informing strategies for managing risks and improving quality. Such analysis is critical to identifying financial and operational problems and essential to mitigating financial crisis, avoiding disruption of services, and fostering agency sustainability. PHUND$ additionally serves as an instrument that can guide development of standards that measure for agency sound financial management systems.

A tale of two cascades: promoting a standardized tool for monitoring progress in HIV prevention

INTRODUCTION

To achieve significant progress in global HIV prevention from 2020 onward, it is essential to ensure that appropriate programmes are being delivered with high quality and sufficient intensity and scale and then taken up by the people who most need and want them in order to have both individual and public health impact. Yet, currently, there is no standard way of assessing this. Available HIV prevention indicators do not provide a logical set of measures that combine to show reduction in HIV incidence and allow for comparison of success (or failure) of HIV prevention programmes and for monitoring progress in meeting global targets. To redress this, attention increasingly has turned to the prospects of devising an HIV prevention cascade, similar to the now-standard HIV treatment cascade; but this has proven to be a controversial enterprise, chiefly due to the complexity of primary prevention.

DISCUSSION

We address a number of core issues attendant with devising prevention cascades, including: determining the population of interest and accounting for the variability and fluidity of HIV-related risk within it; the fact that there are multiple HIV prevention methods, and many people are exposed to a package of them, rather than a single method; and choosing the final step (outcome) in the cascade. We propose two unifying models of prevention cascades-one more appropriate for programme managers and monitors and the other for researchers and programme developers-and note their relationship. We also provide some considerations related to cascade data quality and improvement.

CONCLUSIONS

The HIV prevention field has been grappling for years with the idea of developing a standardised way to regularly assess progress and to monitor and improve programmes accordingly. The cascade provides the potential to do this, but it is complicated and highly nuanced. We believe the two models proposed here reflect emerging consensus among the range of stakeholders who have been engaging in this discussion and who are dedicated to achieving global HIV prevention goals by ensuring the most appropriate and effective programmes and methods are supported.

It's a Marathon, Not a Sprint: Rapid Evaluation of Competency-Based Medical Education Program Implementation

PURPOSE

Despite the broad endorsement of competency-based medical education (CBME), myriad difficulties have arisen in program implementation. The authors sought to evaluate the fidelity of implementation and identify early outcomes of CBME implementation using Rapid Evaluation to facilitate transformative change.

METHOD

Case-study methodology was used to explore the lived experience of implementing CBME in the emergency medicine postgraduate program at Queen's University, Canada, using iterative cycles of Rapid Evaluation in 2017-2018. After the intended implementation was explicitly described, stakeholder focus groups and interviews were conducted at 3 and 9 months post-implementation to evaluate the fidelity of implementation and early outcomes. Analyses were abductive, using the CBME core components framework and data-driven approaches to understand stakeholders' experiences.

RESULTS

In comparing planned with enacted implementation, important themes emerged with resultant opportunities for adaption. For example, lack of a shared mental model resulted in frontline difficulty with assessment and feedback and a concern that the granularity of competency-focused assessment may result in "missing the forest for the trees," prompting the return of global assessment. Resident engagement in personal learning plans was not uniformly adopted, and learning experiences tailored to residents' needs were slow to follow.

CONCLUSIONS

Rapid Evaluation provided critical insights into the successes and challenges of operationalizing CBME. Implementing the practical components of CBME was perceived as a sprint, while realizing the principles of CBME and changing culture in postgraduate training was a marathon requiring sustained effort in the form of frequent evaluation and continuous faculty and resident development.

Development and validation of an instrument to assess institutionalization of health promotion in faith-based organizations

Institutionalization of health promotion interventions occurs when the organization makes changes to support the program as a component of its routine operations. To date there has not been a way to systematically measure institutionalization of health promotion interventions outside of healthcare settings. The purpose of the present study was to develop and evaluate the initial psychometric properties of an instrument to assess institutionalization (i.e., integration) of health activities into faith-based organizations (i.e., churches). This process was informed by previous institutionalization models led by a team of experts and a community-based advisory panel. We recruited African American church leaders (N = 91) to complete a 22-item instrument. An exploratory factor analysis revealed four factors: 1) Organizational Structures (e.g., existing health ministry, health team), 2) Organizational Processes (e.g., records on health activities; instituted health policy), 3) Organizational Resources (e.g., health promotion budget; space for health activities), and 4) Organizational Communication (e.g., health content in church bulletins, discussion of health within sermons) that explained 62.3 % of the variance. The measure, the Faith-Based Organization Health Integration Inventory (FBO-HII), had excellent internal consistency reliability (α = .89) including the subscales (α = .90, .82, .81, and .87). This measure has promising initial psychometric properties for assessing institutionalization of health promotion interventions in faith-based settings.

Expanding evaluator competency research: Exploring competencies for program evaluation using the context of non-formal education

The overlap of competencies between general program evaluation and specific contexts or content will always be reality because evaluators may need unique competencies to answer evaluation questions for particular contexts or content areas. Limited research exists that explores the essential competencies required by professionals who use evaluation as one part of their job portfolio, which leaves unanswered questions regarding the applicability of current evaluator competency models in such settings. We used a modified three-round Delphi technique to identify evaluator competencies for non-formal educators in Cooperative Extension (CE). Our panelists identified 36 competencies in the non-formal educational programming context for CE educators that they considered important to be included in evaluation capacity building efforts. We categorized our 36 identified competencies from the Delphi study into the five competency domains proposed by the American Evaluation Association. Our findings provide information to help guide professional development among non-formal educators related to program evaluation.

The whole elephant: Defining evaluation

Definitions help us understand the characteristics of an object or phenomenon and are a necessary precursor to understanding what a good version of it looks like. Evaluation as a field has resisted a common definition (Crane, 1988; Morell & Flaherty, 1978; M. F. Smith, 1999), which has implications for marketing, training, practice, and quality assurance. In this position paper, I describe the benefits and challenges of not having a clear, agreed-upon definition, then propose and explore the implications of two definitions for the evaluation profession based on values and valuation as the core of evaluation practice. The purpose is to describe a possible way forward through definition that would increase our professional profile, power, and contribution to social justice. The paper concludes with implications for evaluator competencies and evaluation education and questions for further research.

What are the characteristics of the best type 1 diabetes patient education programmes (from diagnosis to long-term care), do they improve outcomes and what is required to make them more effective?

The last 20 years have witnessed a marked change in approaches to the management of type 1 diabetes in the UK. This is exemplified by National Institute of Health and Care Excellence (NICE) guidance which acknowledges that reaching and maintaining target glucose depends on people with type 1 diabetes effectively implementing flexible intensive insulin therapy. The guidance emphasizes that successful self-management requires the acquisition of complex skills and is best achieved by participation in high-quality structured education. Controlled trials and other research have shown that programmes teaching self-management can lower glucose levels while reducing hypoglycaemia, improve psychological outcomes and are highly cost-effective. An important principle of successful programmes is therapeutic education in which learning becomes a partnership between the professional and the person with diabetes who learns to fit diabetes into his/her everyday life. Other recommended elements of programmes include a written curriculum, group teaching by a professional multidisciplinary team and quality assurance. Yet many participants struggle post-course to implement and maintain skills, and overall HbA_1c levels, particularly in the UK, remain far from target. Recent studies have identified the barriers to sustained effective self-management and concluded that even high-quality programmes generally lack critical components. These include incorporating evidence from behaviour change research, exploiting the promise of new technologies in reducing the burden of self-management, and providing structured professional support once people have completed the training. Studies are currently underway to evaluate structured training courses which have added these elements and examine whether they can lower glucose to levels closer to target without impairing quality of life.

What do we know about evaluator education? A review of peer-reviewed publications (1978-2018)

This article reviews peer-reviewed writings on evaluator education from 1978 to 2018. Despite the topic's presumed importance, scholars to date have not extensively addressed it in peer-reviewed publications. The article first describes the methods used to select articles and the conceptual framework for their analysis. It then presents the content of 64 articles selected for review, divided into two major categories: research studies, and reflective case narratives. We further divide research studies into program directories and empirical studies; the case narratives provide information on programs, curriculum, and instruction and minimally on students/faculty. The article concludes with thoughts on next steps for research on evaluator education.

Development of the Health Awareness and Behaviour Tool (HABiT): reliability and suitability for a Canadian older adult population

BACKGROUND

Determining the effectiveness of community-based health promotion and disease prevention programs requires an appropriate data collection tool. This study aimed to develop a comprehensive health questionnaire for older adults, called the HABiT, and evaluate its reliability, content validity, and face validity in assessing individual health-related items (e.g., health status, healthcare utilization) and five specific scales: knowledge, current health behaviors (risk factors), health-related quality of life (HRQoL), perceived risk and understanding, and self-efficacy.

METHODS

Iterative survey development and evaluation of its psychometric properties in a convenience sample of 28 older adults (≥ 55 years old), half from a low-income population. Following item generation, the questionnaire was assessed for content validity (expert panel), face validity (participant feedback), internal consistency of each scale (Cronbach's alpha), and test-retest reliability for each item and scale (Pearson's r and phi correlations, as appropriate).

RESULTS

Questions were drawn from 15 sources, but primarily three surveys: Canadian Community Health Survey, Canadian Diabetes Risk Questionnaire (CANRISK), and a survey by the Canadian Hypertension Education Program. Expert consensus was attained for item inclusion and representation of the desired constructs. Participants completing the questionnaire deemed the questions to be clear and appropriate. Test-retest reliability for many individual items was moderate-to-high, with some exceptions for items that can reasonably change in a short period (e.g., perceived day-to-day stress). Of the five potential scales evaluated, two had acceptable internal consistency (Cronbach's alpha ≥ 0.60) and a subset of one scale also had acceptable internal consistency. Test-retest reliability was high (correlation ≥ 0.80) for all scales and sub-scales.

CONCLUSIONS

The HABiT is a reliable and suitable comprehensive tool with content and face validity that can be used to evaluate health promotion and chronic disease prevention programs in older adults, including low-income older adults. Some noted limitations are discussed. Data collected using this tool also provides a diabetes risk score, health literacy score, and quality-adjusted life years (QALYs) for economic analysis.

Using Evidence to Drive Impact: Developing the FP Goals Impact Matrix

When designing a family planning (FP) strategy, decision‐makers can choose from a wide range of interventions designed to expand access to and develop demand for FP. However, not all interventions will have the same impact on increasing modern contraceptive prevalence (mCP). Understanding the existing evidence is critical to planning successful and cost‐effective programs. The Impact Matrix is the first comprehensive summary of the impact of a full range of FP interventions on increasing mCP using a single comparable metric. It was developed through an extensive literature review with input from the wider FP community, and includes 138 impact factors highlighting the range of effectiveness observed across categories and subcategories of FP interventions. The Impact Matrix is central to the FP Goals model, used to project scenarios of mCP growth that help decision‐makers set realistic goals and prioritize investments. Development of the Impact Matrix, evidence gaps identified, and the contribution to FP Goals are discussed.

Guidance for Male Mentors to Support the Safety and Success of Female Mentees

Women constitute the majority of students at US schools and colleges of pharmacy but make up only a minority of senior faculty members. In academic pharmacy, male mentors play an essential role in supporting the professional success of female mentees. These male mentors must recognize their role in creating safe and equitable environments for female mentees to learn and collaborate. They should respect the unique experiences and concerns of female mentees, particularly regarding sexual violence and harassment, and conduct themselves in a manner that prioritizes safety. They should also strive to develop the same interpersonal connections with female mentees that they work to develop with male mentees, realizing that maintaining inequitable personal distance from women undermines their professional success. Specific suggestions for balancing the potentially competing concepts of mentee safety and personal connection are offered by the authors, a male faculty mentor and female student mentee pair.

Development and assessment of an academic performance enrichment program for low-performing, first-year pharmacy students

Pharmacy school applications have steadily declined over the past several years. Thus pharmacy schools are not only searching for effective means to increase enrollment of qualified candidates, but are also focusing on the development of programs to improve academic performance and retention of enrolled students. To address the needs of struggling first-year pharmacy students enrolled in an Integrated Biological Sciences (BSI) course, an academic performance enrichment program (APEP) was developed. The program was designed to improve academic success by engaging low-performing students with the aims of improving their time management skills, study skills, metacognition, and understanding of BSI course material. The APEP consisted of structured tutoring sessions twice per week, which were required for all students with a course grade ≤73.5% at any point during the semester. To assess program effectiveness, performance improvement on BSI exams by the APEP students were compared with that of non-APEP students in the same class and to those in the previous 3 yr. Student perceptions of the program were also evaluated via an online survey. The APEP was deemed effective in that a greater percentage of students were able to improve their exam scores and to a greater extent by attending the APEP sessions compared with non-APEP students in the same class and with low-performing students in previous years when the APEP did not exist. Furthermore, APEP students believed the program was effective in meeting its aims. In conclusion, the APEP was effective in improving academic performance of low-performing students in BSI.

Integrated care pathway for rectal cancer treatment: cross-sectional post-implementation study using a logic model framework

BACKGROUND

Management of rectal cancer has become more complex with multimodality therapy (neoadjuvant chemoradiotherapy and surgery) and this has led to the need to organize multidisciplinary teams. The aim of this study was to report on the planning, implementation and evaluation of an integrated care pathway for neoadjuvant treatment of middle and lower rectal cancer.

DESIGN AND SETTING

This was a cross-sectional post-implementation study that was carried out at a public university cancer center.

METHODS

The Framework for Program Evaluation in Public Health of the Centers for Disease Control and Prevention (CDC) was used to identify resources and activities; link results from activities and outcomes with expected goals; and originate indicators and outcome measurements.

RESULTS

The logic model identified four activities: stakeholders' engagement, clinical pathway development, information technology improvements and training programs; and three categories of outcomes: access to care, effectiveness and organizational outcomes. The measurements involved 218 patients, among whom 66.3% had their first consultation within 15 days after admission; 75.2% underwent surgery < 14 weeks after the end of neoadjuvant treatment and 72.7% completed the treatment in < 189 days. There was 100% adherence to the protocol for the regimen of 5-fluorouracil and leucovorin.

CONCLUSIONS

The logic model was useful for evaluating the implementation of the integrated care pathways and for identifying measurements to be made in future outcome studies.

Rising to the Challenge: Residency Programs' Experience With Implementing Milestones-Based Assessment

BACKGROUND

Changes to assessment efforts following the shift to milestones-based assessment in the ACGME Next Accreditation System have not been fully characterized.

OBJECTIVE

This study describes themes in initial milestones-based assessment practices with the goal of informing continued implementation and optimization of milestones-based assessment.

METHODS

Semistructured interviews were conducted with 15 residency program leaders in 6 specialties at 8 academic medical centers between August and December 2016. We explored what was retained, what was added, and what was changed from pre-milestones assessment efforts. We also examined the perceived impact of the shift to milestones-based assessment on the programs. Thematic analysis began after the first 5 interviews and ended once thematic sufficiency was reached. Two additional authors reviewed the codes, offered critical input, and informed the formation and naming of the final themes.

RESULTS

Three themes were identified: (1) program leaders faced challenges to effective implementation; (2) program leaders focused on adaptability and making milestones work in what felt like a less than ideal situation for them; and (3) despite challenges, program leaders see value and utility in their efforts to move to milestones-based assessment. We describe a number of strategies that worked for programs during the transition, with perceived benefits acknowledged.

CONCLUSIONS

While adaptation to milestones has occurred and benefits are noted, negative impacts and challenges (eg, perceived lack of implementation guidance and faculty development resources) persist. There are important lessons learned (eg, utilizing implementation experiences formatively to improve curricula and assessment) in the transition to milestones-based assessment.

Pathways to becoming an internal evaluator: Perspectives from the Australian non-government sector

There is a lack of clarity around intra-organisational evaluation roles and pathways into these roles in non-government organisations (NGOs). This article presents three auto-narratives from the authors who are working as internal evaluators in the NGO sector. We examine this phenomenon of role ambiguity by exploring our evaluation journeys and struggles to find identities in the formal evaluation community. Findings from the auto-narratives identify implications for the evaluation field regarding professionalisation. This article explores how aspects of professionalisation, such as clarification of roles and tasks of internal evaluators, could facilitate their recruitment, assess credibility and guide career trajectory. Elucidating internal evaluation career pathways contributes to the evaluation discipline by providing information relevant for evaluation capacity building, evaluator training, and the professionalisation movement.

Effectiveness of universal programmes for the prevention of suicidal ideation, behaviour and mental ill health in medical students: a systematic review and meta-analysis

QUESTION

A growing body of work suggests that medical students may be particularly at risk of mental ill health, suicidal ideation and behaviour, resulting in recent calls to develop interventions to prevent these outcomes. However, few reviews have synthesised the current evidence base regarding the effectiveness of these interventions and provided guidance to improve future intervention efforts.

STUDY SELECTION AND ANALYSIS

The authors conducted a systematic review to identify studies of any design reporting the effectiveness of any universal intervention to address these outcomes in medical students. Embase, MEDLINE and PsycINFO databases were searched from their respective start dates until 1 December 2017.

FINDINGS

Data from 39 studies were included. Most investigated the effectiveness of relatively brief interventions designed to reduce stress; most commonly using mindfulness-based or guided meditation approaches. Only one implemented an intervention specifically designed to address suicidal ideation; none investigated the effectiveness of an intervention specifically designed to address suicidal behaviour. Five investigated the effects of curriculum-level changes. Overall, there was limited evidence of an effect for these programmes at both the postintervention and longest follow-up assessment on depression, anxiety and stress.

CONCLUSIONS

Relatively brief, individually focused, mindfulness-based interventions may be effective in reducing levels of anxiety, depression and stress in medical students in the short term. Effects on suicidal ideation and behaviour, however, remain to be determined. There has been a significant lack of attention on organisational-level stressors associated with medical education and training.

Assessing Quality of Program Environments for Children and Youth with Autism: Autism Program Environment Rating Scale (APERS)

The purpose of this study was to examine the psychometric properties of the Autism Program Environment Rating Scale (APERS), an instrument designed to assess quality of program environments for students with autism spectrum disorder. Data sets from two samples of public school programs that provided services to children and youth with autism spectrum disorder were utilized. Cronbach alpha analyses indicated high coefficients of internal consistency for the total APERS and moderate levels for item domains for the first data set, which was replicated with the second data set. A factor analysis of the first data set indicated that all domain scores loaded on one main factor, in alignment with the conceptual model, with this finding being replicated in the second data set. Also, the APERS was sensitive to changes resulting from a professional development program designed to promote program quality.

Assessing research impact: Australian Research Council criteria and the case of Family Wellbeing research

Researchers worldwide are increasingly reporting the societal impact of their research as part of national research productivity assessments. However, the challenges they encounter in developing their impact case studies against specified government assessment criteria and how pitfalls can be mitigated are not reported. This paper examines the key steps taken to develop an Aboriginal Family Wellbeing (FWB) empowerment research impact case study in the context of an Australian Research Council (ARC) pilot research impact assessment exercise and the challenges involved in applying the ARC criteria. The requirement that researchers demonstrate how their institutions support them to conduct impactful research has the potential to create supportive environments for researchers to be more responsive to the needs of users outside academia. However, the 15-year reference period for the associated research underpinning the reported impact and the focus on researcher's current institutional affiliation constitute potential constraints to demonstrating the true impact of research. For researchers working with Indigenous people, relationships that build over long periods of time, irrespective of university affiliation, are critical to conducting impactful research. A more open-ended time-frame, with no institutional restrictions for the 'associated research' provides the best opportunity to demonstrate the true benefits of research not only for Indigenous people but for Australian society more broadly.

Development and Testing of Program Evaluation Instruments for the iCook 4-H Curriculum

OBJECTIVE

To develop and test the validity of program outcome evaluation instruments for cooking, eating, and playing together for obesity prevention during iCook 4-H.

DESIGN

Instrument development for both youth and adults through pre-post testing of items newly constructed and compiled to address key curriculum constructs. Testing occurred throughout program intervention and dissemination to determine dimensionality, internal consistency and test-retest reliability, and validity.

SETTING

A 5-state out-of-school program in cooperative extension and other community sites.

PARTICIPANTS

Youths aged 9-10 years; adults were main food preparers; the first phase involved 214 dyads and the second phase, 74 dyads.

MAIN OUTCOME MEASURE(S)

Youth measures were cooking skills, culinary self-efficacy, physical activity, and openness to new foods. Adult measures were cooking together, physical activity, and eating together.

ANALYSIS

Exploratory factor analysis to determine initial scale structure and confirmatory factor analysis to confirm factor structures. Longitudinal invariance tests to see whether the factor structure held over time. Test-retest reliability was determined by Pearson r and internal consistency was determined by coefficient Ω and Cronbach α. Validity testing was determined by Pearson r correlations.

RESULTS

Youth cooking skills, openness to new foods, and adult eating together and cooking together showed strong evidence for dimensionality, reliability, and validity. Youth physical activity and adult physical activity measures showed strong evidence for dimensionality and validity but not reliability. The youth culinary self-efficacy measure showed strong evidence for reliability and validity but weaker evidence for dimensionality.

CONCLUSIONS AND IMPLICATIONS

Program outcome evaluation instruments for youths and adults were developed and tested to accompany the iCook 4-H curriculum. Program leaders, stakeholders, and administrators may monitor outcomes within and across programs and generate consistent reporting.

Constructing care cascades for active tuberculosis: A strategy for program monitoring and identifying gaps in quality of care

The cascade of care is a model for evaluating patient retention across sequential stages of care required to achieve a successful treatment outcome. This approach was first used to evaluate HIV care and has since been applied to other diseases. The tuberculosis (TB) community has only recently started using care cascade analyses to quantify gaps in quality of care. In this article, we describe methods for estimating gaps (patient losses) and steps (patients retained) in the care cascade for active TB disease. We highlight approaches for overcoming challenges in constructing the TB care cascade, which include difficulties in estimating the population-level burden of disease and the diagnostic gap due to the limited sensitivity of TB diagnostic tests. We also describe potential uses of this model for evaluating the impact of interventions to improve case finding, diagnosis, linkage to care, retention in care, and post-treatment monitoring of TB patients.

Evaluating program effects: Conceptualizing and demonstrating a typology

Outcome evaluation is very important for program evaluation and has been becoming increasingly so in the age of accountability. Typically, outcome evaluation is conducted for a single program from a single perspective. However, in a real-life situation, many programs exist in a system, and the effects could be viewed from various perspectives. The authors illustrate a typology of program effects in a system. It moves from the paradigm of a single program's single effect to that of a set of programs' multiple effects. Methodological implications are discussed.

Explaining experiences of community-dwelling older adults with a pro-active comprehensive geriatric assessment program - a thorough evaluation by interviews

INTRODUCTION

Pro-active assessment programs are increasingly used to improve care for older adults. These programs include comprehensive geriatric tailored to individual patient preferences. Evidence for the effects of these programs on patient outcomes is nevertheless scarce or ambiguous. Explaining these dissatisfying results is difficult due to the multi-component nature of the programs. The objective of the current study was to explore and explain the experience of older adults participating in a pro-active assessment program, to help to clarify the effects.

METHODS

Semi-structured in-depth interviews were held with 25 participants of a pro-active assessment program for frail community-dwelling adults aged 65+. This study was part of an evaluation study on the effects of the program. Transcripts were analysed with thematic analysis and cross-case analysis.

RESULTS

The participants' mean age was 78.5 (SD 6.9) and 56% was female. The majority of the participants were satisfied with the program but based this on communication aspects, since only a few of them expressed real program benefits. Participant experiences could be clustered in six themes: (1) All participants expressed the need for a holistic view which was covered in the program, (2) the scope of the CGA was broader than expected or unclear, (3) the program delivered unexpected but valued help, (4) participants described a very low sense of ownership, (5) timing of the program implementation or the CGA was difficult and(6), participants and care workers had a different view on what to consider as a problem. These experiences could be explained by three program components: the degree of (the lack of) integration of the program within usual care, the pro-active screening method and the broader than expected, but appreciated multi-domain approach.

CONCLUSION

Older adults' need for a holistic view is covered by this outpatient assessment program. However, their engagement and the correct timing of the program are hampered by the pro-active recruitment and the limited integration of the program within existing care. Furthermore, satisfaction seems an insufficient guiding factor when evaluating CGA programs for older adults because it does not reflect the impact of the program.

Evaluation of a portfolio-based course on self-development for pre-medical students in Korea

PURPOSE

We have developed and operated a portfolio-based course aimed at strengthening pre-medical students' capabilities for self-management and self-improvement. In order to determine the effectiveness of the course and to establish future operational strategies, we evaluated the course and the students' learning experience.

METHODS

The subjects of this study were 97 students of a pre-medical course "Self-development and portfolio I" in 2019. Their learning experience was evaluated through the professor's assessment of portfolios they had submitted, and the program was evaluated based on the responses of 68 students who completed a survey. The survey questionnaire included 32 items. Descriptive statistics were reported for quantitative data, including the mean and standard deviation. Opinions collected from the open-ended question were grouped into categories.

RESULTS

The evaluation of students' portfolios showed that only 6.2% of the students' portfolios were well-organized, with specific goals, strategies, processes, and self-reflections, while most lacked the basic components of a portfolio (46.4%) or contained insufficient content (47.4%). Students' responses to the survey showed that regular portfolio personality assessments (72.1%), team (64.7%), and individual (60.3%) activities were felt to be more appropriate as educational methods for this course, rather than lectures. Turning to the portfolio creation experience, the forms and components of the portfolios (68.2%) and the materials provided (62.2%) were felt to be appropriate. However, students felt that individual autonomy needed to be reflected more (66.7%) and that this course interfered with other studies (42.5%).

CONCLUSION

The findings of this study suggest that standardized samples, guidelines, and sufficient time for autonomous portfolio creation should be provided. In addition, education on portfolio utilization should be conducted in small groups in the future.

Incorporating public values into evaluative criteria: Using crowdsourcing to identify criteria and standards

At its core, evaluation involves the generation of value judgments. These evaluative judgments are based on comparing an evaluand's performance to what the evaluand is supposed to do (criteria) and how well it is supposed to do it (standards). The aim of this four-phase study was to test whether criteria and standards can be set via crowdsourcing, a potentially cost- and time-effective approach to collecting public opinion data. In the first three phases, participants were presented with a program description, then asked to complete a task to either identify criteria (phase one), weigh criteria (phase two), or set standards (phase three). Phase four found that the crowd-generated criteria were high quality; more specifically, that they were clear and concise, complete, non-overlapping, and realistic. Overall, the study concludes that crowdsourcing has the potential to be used in evaluation for setting stable, high-quality criteria and standards.

Evaluation of a Noyce program: Development of teacher leaders in STEM education

The development of teacher leaders in science, technology, engineering, and mathematics has become a focus as demand grows on the national scale to improve student learning in these disciplines. As teachers' role in leadership continues to be redefined, research and professional development in teacher leadership will continue to evolve. Given the lack of a clear conceptualization of teacher leadership in the empirical literature, there is a clear methodological challenge for evaluators who are charged with assessing the impact of teacher leadership professional development programs. This paper describes how both the Utilization-Focused Evaluation and Theory-Driven Evaluation frameworks were used concurrently to design evaluation methods that were effective for assessing the impact of a dynamic teacher leadership program. The evaluation is specifically situated within the context of a Robert Noyce Scholarship Program, which aimed to grow veteran science teachers into teacher leaders. The paper describes how the evaluation frameworks used guided the evaluation methods, provides illustrative evaluation results, and states lessons learned from the author's experiences working within this context. This paper aims to provide an example of evaluation methods that could be replicated by evaluators' working within a Noyce or teacher leadership context.

Evaluation capacity building-Results and reflections across two years of a multisite empowerment evaluation in an HIV prevention context

As the need for rigorous evidence of program efficacy increases, integrating evaluation activities into program implementation is becoming crucial. As a result, external evaluators are placing increased focus on evaluation capacity building as a practice. However, empirical evidence of how to foster evaluation capacity in different contexts remains limited. This study presents findings from an evaluation capacity survey conducted within a multisite Empowerment Evaluation initiative, in which an external evaluator worked with 20 project teams at diverse community agencies implementing HIV prevention projects. Survey results revealed representatives from project teams (n = 33) reported significantly higher overall evaluation capacity after engaging with the external evaluator on planning and implementing their evaluation. Improvements differed across organization type, intervention type, staff position, and reported engagement on various activities throughout the course of the evaluation. Results indicated empowerment evaluation and other stakeholder-focused evaluation approaches are broadly applicable when evaluation capacity building is a desired outcome, particularly when able to engage project staff in the planning of the evaluation and in delivering technical assistance services. Accordingly, efforts should be made by program funders, staff, and evaluators to encourage active engagement starting in the early stages of program and evaluation planning.

Sustaining quality in the community: trends in the performance of a structured diabetes care programme in primary care over 16 years

AIM

To examine the quality of care delivered by a structured primary care-led programme for people with Type 2 diabetes mellitus in 1999-2016.

METHODS

The Midland Diabetes Structured Care Programme provides structured primary care-led management. Trends over time in care processes were examined (using a chi-squared trend test and age- and gender-adjusted logistic regression). Screening and annual review attendance were reviewed. A composite of eight National Institute for Health and Care Excellence-recommended processes was used as a quality indicator. Participants who were referred to diabetes nurse specialists were compared with those not referred (Student's t-test, Pearson's chi-squared test, Wilcoxon-Mann-Whitney test). Proportions achieving outcome targets [HbA_1c ≤58 mmol/mol (7.5%), blood pressure ≤140/80 mmHg, cholesterol <5.0 mmol/l] were calculated.

RESULTS

Data were available for people with diabetes aged ≥18 years: 1998/1999 (n=336); 2003 (n=843); 2008 (n=988); and 2016 (n=1029). Recording of some processes improved significantly over time (HbA_1c , cholesterol, blood pressure, creatinine), and in 2016 exceeded 97%. Foot assessment and annual review attendance declined. In 2016, only 29% of participants had all eight National Institute for Health and Care Excellence processes recorded. A higher proportion of people with diabetes who were referred to a diabetes nurse specialist had poor glycaemic control compared with those not referred. The proportions meeting blood pressure and lipid targets increased over time.

CONCLUSIONS

Structured primary care led to improvements in the quality of care over time. Poorer recording of some processes, a decline in annual review attendance, and participants remaining at high risk suggest limits to what structured care alone can achieve. Engagement in continuous quality improvement to target other factors, including attendance and self-management, may deliver further improvements.

Effectiveness of the Gold Standard Programme (GSP) for smoking cessation on smokers with and without a severe mental disorder: a Danish cohort study

OBJECTIVES

We compared the effectiveness of an intensive smoking cessation intervention among smokers with and without a severe mental disorder (SMD) and identified factors associated with successful quitting. The main hypothesis was that smokers with an SMD would be less likely to stay continuously smoke-free for 6 months.

DESIGN

A prospective cohort study.

SETTING

In all, 302 smoking cessation clinics in Denmark from municipal clinics, pharmacies, hospitals, midwives, primary care facilities and other private providers who reported data to the national Danish Smoking Cessation Database from 2006 to 2016 participated in this study.

PARTICIPANTS

A total of 38 293 patients from the Danish Smoking Cessation Database. Patients with an SMD were identified by linking data to the Danish National Patient Register. Diagnoses of organic mental disorders (F0 chapter) or intellectual disabilities (F7 chapter) were not included. Smokers ≥18 years old who were attending a Gold Standard Programme (GSP) with planned follow-up were included. Smokers not wanting contact after 6 months were excluded.

INTERVENTIONS

A comprehensive manual-based smoking cessation intervention comprising five meetings over a 6-week period (the GSP).

MAIN OUTCOME MEASURES

Self-reported continuous abstinence at the 6-month follow-up.

RESULTS

In all, 69% of the participants participated in the follow-up after 6 months. The overall rate of successful quitting was high but significantly lower in SMD smokers (29% vs 38%; OR 0.74; 95% CI 0.68 to 0.80). Variables associated with successful quitting were compliance (defined as attending ≥75% of the planned meetings), older age and male gender as well as not being disadvantaged, heavy smoking or recommendation of intervention by health professionals.

CONCLUSIONS

Only 29% of smokers with an SMD successfully quit smoking which was significantly lower than the 38% of smokers without an SMD. Compliance was the most important predictor for successful quitting.

Systematic iteration between model and methodology: A proposed approach to evaluating unintended consequences

This article argues that evaluators could better deal with unintended consequences if they improved their methods of systematically and methodically combining empirical data collection and model building over the life cycle of an evaluation. This process would be helpful because it can increase the timespan from when the need for a change in methodology is first suspected to the time when the new element of the methodology is operational. The article begins with an explanation of why logic models are so important in evaluation, and why the utility of models is limited if they are not continually revised based on empirical evaluation data. It sets the argument within the larger context of the value and limitations of models in the scientific enterprise. Following will be a discussion of various issues that are relevant to model development and revision. What is the relevance of complex system behavior for understanding predictable and unpredictable unintended consequences, and the methods needed to deal with them? How might understanding of unintended consequences be improved with an appreciation of generic patterns of change that are independent of any particular program or change effort? What are the social and organizational dynamics that make it rational and adaptive to design programs around single-outcome solutions to multi-dimensional problems? How does cognitive bias affect our ability to identify likely program outcomes? Why is it hard to discern change as a result of programs being embedded in multi-component, continually fluctuating, settings? The last part of the paper outlines a process for actualizing systematic iteration between model and methodology, and concludes with a set of research questions that speak to how the model/data process can be made efficient and effective.

Aid and good governance: Examining aggregate unintended effects of aid

Although donors generally aim to improve governance in recipient countries by various means, critics claim that the aggregate effect of large aid flows is the deterioration of governance. Aid is said to weaken domestic accountability, sustain authoritarian regimes, increase political instability, weaken government capacities, and increase corruption. Conducting a systematic search in Web of Science, this paper reviews the empirical evidence for these unintended aggregate effects of aid on the political, administrative, and judicial dimensions of good governance. It finds that the negative effects of aid on governance are much exaggerated. The aggregate effect of aid on democracy has become more positive after the Cold War, and the effect of aid on government capacity and on reducing corruption has also improved over time. Furthermore, most studies show a positive effect of aid on political stability. These findings imply that donor intentions matter: donors that are serious about their intended effects on governance are able to mitigate the possible negative unintended effects of their aid.

The effects of stakeholder involvement on perceptions of an evaluation's credibility

This article presents a study of the effects of stakeholder involvement on perceptions of an evaluation's credibility. Crowdsourced members of the public and a group of educational administrators read a description of a hypothetical program and two evaluations of the program: one conducted by a researcher and one conducted by program staff (i.e. program stakeholders). Study participants were randomly assigned versions of the scenario with different levels of stakeholder credibility and types of findings. Results showed that both samples perceived the researcher's evaluation findings to be more credible than the program staff's, but that this difference was significantly reduced when the program staff were described to be highly credible. The article concludes with implications for theory and research on evaluation dissemination and stakeholder involvement.

Evaluating the implementation of Check & Connect in various school settings: Is intervention fidelity necessarily associated with positive outcomes?

There are numerous school dropout prevention programs. However, few of them have undergone a rigorous implementation evaluation to understand their effects. This research presents two studies that evaluated the intervention fidelity and differential effects of Check & Connect (C&C), a targeted school dropout prevention program aimed at promoting student engagement and achievement. A total of 145 elementary school students (Study 1) and 200 secondary school students (Study 2) from two French-Canadian school boards (regional districts grouping elementary and secondary schools) received the C&C intervention for two years. In both studies, a clinical monitoring form was used to compare the intervention fidelity of each program component and active ingredient with what was initially planned. The relation between intervention fidelity and the effects of C&C on student engagement and achievement was analyzed using multiple linear regressions. Overall, the results show that intervention fidelity varies across elementary and secondary schools from one component to another and from one site to another. Furthermore, the association between the fidelity of each component and positive outcomes varies, depending on the implementation site. This evaluation supports the relevance of every component of C&C to favor engagement and academic achievement among at-risk elementary and secondary school students, while suggesting that the importance of certain program components may vary, depending on contextual influences on implementation and outcomes.

The human factor in development cooperation: An effective way to deal with unintended effects

Development policy and implementation are a human endeavor. Too often, however, the human factor is relegated to an input or an externality in a quasi-technical process for transforming public funds into measurable results. Within the Weberian rational-legal order, policies and bureaucracies are impersonal and objective. Policy objectives tend to get depersonalized and the human stories get filtered out of impact evaluations. This article, on the basis of case descriptions, argues that following the human stories in development policy and implementation can offer surprising insights into why at times policies may work or not work. The article explores how the idiosyncrasies of individuals' agency impact on achievement of policy outcomes and what the unintended effects are. It describes how the human factor can give rise to beneficial unplannable, unforeseeable, and thus unintended policy outcomes. This article argues that, instead of negating this, policy makers ought to embrace the human factor of development cooperation.

Reported credibility techniques in higher education evaluation studies that use qualitative methods: A research synthesis

This synthesis study examined the reported use of credibility techniques in higher education evaluation articles that use qualitative methods. The sample included 118 articles published in six leading higher education evaluation journals from 2003 to 2012. Mixed methods approaches were used to identify key credibility techniques reported across the articles, document the frequency of these techniques, and describe their use and properties. Two broad sets of techniques were of interest: primary design techniques (i.e., basic), such as sampling/participant recruitment strategies, data collection methods, analytic details, and additional qualitative credibility techniques (e.g., member checking, negative case analyses, peer debriefing). The majority of evaluation articles reported use of primary techniques although there was wide variation in the amount of supporting detail; most of the articles did not describe the use of additional credibility techniques. This suggests that editors of evaluation journals should encourage the reporting of qualitative design details and authors should develop strategies yielding fuller methodological description.

Validation of the evaluation capacity in organizations questionnaire

The purpose of this study was to test the construct validity of the Evaluation Capacity in Organizations Questionnaire (ECOQ). Conceptually, the ECOQ examines the role of evaluation in organizational development and, most notably in organizational learning. In this model, evaluation capacity building (ECB) initiatives are assumed to contribute to the development of a culture of systematic self-assessment and reflection, which, in turn, leads to increased organizational learning. Our sample consisted of internal evaluators within the federal, provincial or municipal government, not-for-profit organizations, private firms, and colleges or universities in Canada. Exploratory factor analysis (EFA) and latent path analysis (LPA) were conducted to better understand the underlying structural aspect of the organizational capacity to do and use evaluation construct as measured by the ECOQ. The results of our study indicate that the ECOQ effectively assesses an organization's capacity to do and use evaluation. Furthermore, evidence provided by the LPA statistical analysis suggests that an organization's capacity to learn is enhanced by the relationships among the various factors. Implications of using a validated model of an organization's capacity to do and use evaluations in both research and practice are discussed.

Measuring unintended effects in peacebuilding: What the field of international cooperation can learn from innovative approaches shaped by complex contexts

Capturing unintended impacts has been a persistent struggle in all fields of international development, and the field of peacebuilding is no exception. However, because peacebuilding focuses on relationships in complex contexts, the field of peacebuilding has, by necessity, made efforts towards finding practical ways to reflect upon both the intended and unintended effects of this work. To explore what lessons can be learned from the peacebuilding field, this study examines the evaluations of Search for Common Ground, a peacebuilding organisation working in over 35 countries across the world. Analysis focuses on 96 evaluations completed between 2013 and 2016 in 24 countries across Africa, Asia, and the MENA regions that found unintended effects. Programmes focusing on women, youth, and radio were most effective at identifying and explaining unintended effects, likely because the project design guided broader lines of questioning from the beginning. The paper argues that OECD-DAC guidelines are not enough on their own to guide evaluators into exploration of unintended effects, and teams instead need to work together to decide where, when and how they will look for them. Different approaches were also used to capture positive and negative outcomes, suggesting that evaluators need to decide at what level they are evaluating and how to tie effects back to the project's contribution. This study explores evaluation techniques and approaches used to understand impact in complex contexts in the peacebuilding field, and draws on lessons learned for the benefit of other fields dealing with similar complexities in international development and cooperation among actors.