Growing stronger together: Sharing a story of culturally responsive evaluation with indigenous families and communities

BACKGROUND

Gaps exist in understanding how to create and conduct culturally responsive evaluations. This information is particularly critical when working with evidence-based programs and when involving populations that have and continue to experience oppression and trauma.

OBJECTIVE

We share our story of developing and carrying out a culturally responsive evaluation of an EBP, Strengthening Families Program (SFP), with Indigenous families.

PARTICIPANTS AND SETTING

A collective storytelling approach was used based on reflections from the evaluation team and key implementation staff.

METHODS

We used a collective storytelling approach, organizing the content around six previously identified principles of Indigenous research (Tsosie et al., 2022).

RESULTS

Emerging themes, supported by quotes throughout, illustrate the importance of organizing the integration of culture into the evaluation through the six principles of Indigenous research: respect, relationship, relevance, reciprocity, responsibility, and representation.

CONCLUSION

Working toward a culturally responsive evaluation allowed for the creation of more meaningful connections with Indigenous community partners and families. It also acknowledged insights that partners and families bring to the work and encouraged multi-directional learning to occur between evaluators, partners, and families.

Growing stronger together: Implementing the Strengthening Families Program with Indigenous communities

BACKGROUND

Child welfare agencies commonly seek to use evidence-based programs (EBPs) for their demonstrated results. Challenges remain in adapting programs to fit for Indigenous populations. We suggest that relationality holds promise as a guide in the implementation of EBPs with Indigenous families and children.

OBJECTIVE

We provide the story of a culturally integrated implementation of the EBP, Strengthening Families Program (SFP), with Indigenous families.

PARTICIPANTS AND SETTING

Insights from the staff who implemented SFP, project leadership and a community steering committee were brought together to create the collective implementation story.

METHODS

A relational approach was used in thematic analysis with a focus on the three Rs - responsibility, respect, and reciprocity- that support Indigenous knowledge organization.

RESULTS

Findings offer insight into cultural integrations in the implementation of SFP. The program centered Indigenous and community identities through meals, gifts, parenting practice examples and discussions tailored by each group of families and staff. Practices related to responsibility, respect and reciprocity each proved to be essential concepts in the relationship building among caregivers, children, SFP staff, project leadership, and community supporters that led to program success.

CONCLUSION

Cultural integration created a space that reflected Indigenous knowledge relationality. It respected the uniqueness among groups of families who participated in the evidence-based SFP. Our story supports the importance of having Indigenous staff and group leaders to guide cultural integration in relationship with tribal communities.

Facilitating implementation of primary care mental health over time and across organizational contexts: a qualitative study of role and process

BACKGROUND

Healthcare organizations have increasingly utilized facilitation to improve implementation of evidence-based practices and programs (e.g., primary care mental health integration). Facilitation is both a role, related to the purpose of facilitation, and a process, i.e., how a facilitator operationalizes the role. Scholars continue to call for a better understanding of this implementation strategy. Although facilitation is described as dynamic, activities are often framed within the context of a staged process. We explored two understudied characteristics of implementation facilitation: 1) how facilitation activities change over time and in response to context, and 2) how facilitators operationalize their role when the purpose of facilitation is both task-focused (i.e., to support implementation) and holistic (i.e., to build capacity for future implementation efforts).

METHODS

We conducted individual monthly debriefings over thirty months with facilitators who were supporting PCMHI implementation in two VA networks. We developed a list of facilitation activities based on a literature review and debriefing notes and conducted a content analysis of debriefing notes by coding what activities occurred and their intensity by quarter. We also coded whether facilitators were "doing" these activities for sites or "enabling" sites to perform them.

RESULTS

Implementation facilitation activities did not occur according to a defined series of ordered steps but in response to specific organizational contexts through a non-linear and incremental process. Amount and types of activities varied between the networks. Concordant with facilitators' planned role, the focus of some facilitation activities was primarily on doing them for the sites and others on enabling sites to do for themselves; a number of activities did not fit into one category and varied across networks.

CONCLUSIONS

Findings indicate that facilitation is a dynamic and fluid process, with facilitation activities, as well as their timing and intensity, occurring in response to specific organizational contexts. Understanding this process can help those planning and applying implementation facilitation to make conscious choices about the facilitation role and the activities that facilitators can use to operationalize this role. Additionally, this work provides the foundation from which future studies can identify potential mechanisms of action through which facilitation activities enhance implementation uptake.

Recruitment and engagement in disease self-management programs: Special concerns for rural residents reporting depression and/or anxiety

•

Poorer health outcomes are correlated with depression/anxiety in a rural population.

•

Electronic modes of recruitment engage those reporting depression/anxiety into CDSME.

•

CDSME increases patient activation regardless of history of depression/anxiety.

Chronic disease self-management education (CDSME) programs benefit individuals with chronic diseases, including mental health conditions, by improving health-related outcomes and increasing engagement with the health care system. Recruiting individuals with a history of mental health conditions to participate in CDSME is challenging, particularly in rural, underserved areas. Hence, it is important to understand factors associated with the presence of mental health conditions, and impacts of CDSME on patient engagement. This project identifies individual and program-level characteristics, as well as recruitment characteristics, associated with reporting a history of depression and/or anxiety. It also assesses factors related to program engagement and the relationship between completing CDSME and patient activation. Data were collected during CDSME workshops offered in 2019 in a rural region of New York. Of the 421 enrollees who completed survey instruments, 162 reported a history of depression and/or anxiety. Univariate analyses indicated that those reporting a history of depression and/or anxiety were younger, female, in poorer health, had more comorbidities, were Medicaid beneficiaries, and had lower patient activation scores. They also heard about and signed up for the workshop through the internet at higher rates than those not reporting a history of depression and/or anxiety. Multivariable logistic regression modeling indicated age, self-rated health, and number of comorbidities were independent predictors of reporting a history of depression and/or anxiety. Among CDSME completers, patient activation significantly improved regardless of history of depression and/or anxiety. Engaging individuals with mental health conditions in CDSME requires a multimodal recruitment strategy incorporating electronic marketing and registration.

Parent- and teacher-reported long-term effects of parent training on child conduct problems in families with child protection and other support services: a randomized controlled trial

BACKGROUND

This randomized controlled trial (RCT) evaluated the long-term effectiveness of the Incredible Years^® (IY) Parenting Program in modifying children's externalizing problems among families in Child Protection Services (CPS) and using other special support services. We also examined whether parent-reported effects of the IY^® generalize to the daycare/school setting as reported by teachers.

METHODS

Participants in the study were 3-7-year-old children with behavioural problems (N = 102 at baseline, N = 89 at one-year follow-up). Participants were randomized to intervention (N = 50) and control groups (N = 52) after the baseline assessment. The intervention group received 19-week IY^® Parenting Program. The effectiveness of the intervention was analyzed using linear mixed model.

RESULTS

Our previously reported pre-post intervention effects on CBCL (Child Behavior Checklist) and ECBI (Eyberg Child Behavior Inventory) were not sustained to the one-year follow-up. Child conduct problems decreased from baseline to follow-up in both intervention and control groups. The positive changes were not observed at daycare/school from baseline to post-intervention or to the one-year follow-up, and there were no significant differences in changes between the groups.

CONCLUSIONS

Evidence-based parenting program IY^® seems to be an effective intervention for child conduct problems in the short term in families in the CPS context, but sustaining the positive effects and generalizing them to the daycare/school context are challenging.

TRIAL REGISTRATION

The trial is registered in the ClinicalTrials.gov registry (NCT03239990), Registered August 4th, 2017; https://clinicaltrials.gov/ct2/results?cond=&term=NCT03239990&cntry=&state=&city=&dist=.

Changes in practitioners' attitudes, perceived training needs and self-efficacy over the implementation process of an evidence-based parenting program

BACKGROUND

Evidence-based family support programs such as the Triple P - Positive Parenting Program have the potential to enhance the well-being of children and families. However, they cannot achieve their expected outcomes if insufficient attention is paid to the implementation process. It has been demonstrated that practitioners' attitudes towards evidence-based programs (EBPs), perceived training needs and self-efficacy for working with parents influence implementation outcomes (e.g., program acceptability, adoption, adherence and sustainability). At the same time, the experience of being involved in the implementation process of an EBP could enhance practitioners' perceptions of the initiative. This study aimed to assess changes in practitioner's attitudes, perceived training needs and self-efficacy over a two-year EBP implementation process, in interaction with their appraisal of their organization's capacity to implement the EPB.

METHODS

In the province of Quebec, Canada, Triple P was implemented and evaluated in two communities. Ninety-nine practitioners from various organizations completed questionnaires shortly before their training in Triple P and two years later.

RESULTS

Findings show that practitioners who displayed more initial skepticism regarding their organization's capacity to implement the program reported greater improvements in attitudes over time, while practitioners who showed more optimism at baseline reported a greater decrease in their perceived training needs. Practitioners' self-efficacy increased moderately regardless of perceived organizational capacity.

CONCLUSIONS

These results are encouraging given that more positive perceptions of EBPs could foster the systematic use of these programs in communities, for the potential benefit of a greater number of families.

Barriers and facilitators to implementing evidence-based health innovations in low- and middle-income countries: A systematic literature review

The unsuccessful implementation of health innovations occurs frequently, leading to missed opportunities where improvements could have been made on various aspects of a health system. The purpose of this study is to identify, assess and synthesise the facilitators and barriers to sustainably implementing evidence-based health innovations in a low- and middle-income country (LMIC) context. To identify the LMIC specific facilitators and barriers, a systematic literature review was conducted. 79 studies were analysed, and the implementation barriers and facilitators identified in each study were extracted. The extracted barriers and facilitators were categorised and synthesized into one of seven concepts: context, innovation, relations and networks, institutions, knowledge, actors or resources. There were no substantial variations between the frequency that each concept was identified as a facilitator or barrier to implementation. However, resources, which includes time, human, financial and physical resources, was the most frequently mentioned concept; emphasising the need to focus on the resource situation in LMICs. This study contributes to the growing literature that aims to inform health system planners and evaluators in LMICs on effectively and sustainably implementing evidence-based health innovations.

Addressing the Adherence-Adaptation Debate: Lessons from the Replication of an Evidence-Based Sexual Health Program in School Settings

Whether high adherence to programs is necessary to achieve program outcomes is an area of great debate. The objectives of this study were to determine the frequency, type, and rationale of adaptations made in the implementation of an evidence-based program and to determine program outcomes for intervention program participants, as compared to comparison participants, by the level of adaptations. A total of 1608 participants in 45 classrooms participated. Percent adaptations was calculated by classroom. Thematic qualitative analysis was used to categorize types and rationales for adaptations. Program outcomes by level of adaptations were determined using logistic regression analyses and mean differences. Propensity score matching methods were used to create comparability between adaptation subgroup participants and comparison participants. Adaptations ranged from 2 to 97% across classrooms, with mean adaptations of 63%. Thematic analysis revealed that the adaptations made were related to delivery of content, rather than to the content itself and in response to participant needs and setting constraints. Program outcomes did not appear to be reduced for the high-adaptation subgroup. Understanding both rationale (intent) and type of adaptation made is crucial to understanding the complexity of adaptations. These finding support the argument for allowing facilitators some flexibility and autonomy to adapt the delivery of prescribed content to participant needs and setting constraints.

Influence of an Implementation Support Intervention on Barriers and Facilitators to Delivery of a Substance Use Prevention Program

Implementation support interventions have helped organizations implement programs with quality and obtain intended outcomes. For example, a recent randomized controlled trial called Preparing to Run Effective Programs (PREP) showed that an implementation support intervention called Getting To Outcomes (GTO) improved implementation of an evidence-based substance use prevention program (CHOICE) run in community-based settings. However, more information is needed on how these interventions affect organizational barriers and facilitators of implementation. This paper aims to identify differences in implementation facilitators and barriers in sites conducting a substance use prevention program with and without GTO. PREP is a cluster-randomized controlled trial testing GTO, a two-year implementation support intervention, in Boys & Girls Clubs. The trial compares 15 Boys & Girls Club sites implementing CHOICE (control group), a five-session evidence-based alcohol and drug prevention program, with 14 Boys & Girls Club sites implementing CHOICE supported by GTO (intervention group). All sites received CHOICE training. Intervention sites also received GTO manuals, training, and onsite technical assistance to help practitioners complete implementation best practices specified by GTO (i.e., GTO steps). During the first year, technical assistance providers helped the intervention group adopt, plan, and deliver CHOICE, and then evaluate and make quality improvements to CHOICE implementation using feedback reports summarizing their data. Following the second year of CHOICE and GTO implementation, all sites participated in semi-structured interviews to identify barriers and facilitators to CHOICE implementation using the Consolidated Framework for Implementation Research (CFIR). This paper assesses the extent to which these facilitators and barriers differed between intervention and control group. Intervention sites had significantly higher average ratings than control sites for two constructs from the CFIR process domain: planning and reflecting and evaluating. At the same time, intervention sites had significantly lower ratings on the culture and available resources constructs. Findings suggest that strong planning, evaluation, and reflection-likely improved with GTO support-can facilitate implementation even in the face of perceptions of a less desirable implementation climate. These findings highlight that implementation support, such as GTO, is likely to help low-resourced community-based organizations improve program delivery through a focus on implementation processes. TRIAL REGISTRATION: This project is registered at ClinicalTrials.gov with number NCT02135991 (URL: https://clinicaltrials.gov/show/NCT02135991). The trial was first registered May 12, 2014.

Tapping into Community Resiliency in Rural Adolescent Pregnancy Prevention: An Implementation Sciences Approach

Although rural youth experience marked inequities in adolescent pregnancy, there is little guidance for implementing evidence-based programs (EBPs) in rural settings. When implementation occurs in rural communities, it frequently focuses on deficits, rather than strengths or capacity for growth. Using the consolidated framework for implementation research (CFIR), we describe a resiliency-focused implementation of two middle school EBPs in rural Midwestern communities, including the intervention, outer and inner settings, individuals, implementation processes and preliminary outcomes. Data included program staff interviews, feedback from local partners, community meetings notes, and participant surveys. Using the CFIR, we describe the engagement of rural communities themselves in a resilience-based implementation of adolescent pregnancy prevention EPBs. Communities self-described as rural, traditional and religious. They identified adolescent pregnancy, substance use, and academic success as priorities. To address infrastructure needs and build on local strengths, funds were used to hire local partners to implement the program. As small communities, stakeholders were closely networked and wanted to address local needs. Local partners selected the EBP based upon community values and priorities. Champions, including local partner organizations and schools were locally based and were well connected. Intensive training of local staff and piloting with adaptation assured fidelity and sustainability, while increasing community implementation skills and comfort. In Clinton County, enrollment was 1946 with students receiving the program in 6th, 7th, and/or 8th grades. In Southern Indiana, 7275 students received the program once in either 6th, 7th, or 8th. We conclude that the CFIR can facilitate the implementation of a community resilience-focused adolescent pregnancy prevention intervention in rural communities.

Sustaining an Evidence-Based Program Over Time: Moderators of Sustainability and the Role of the Getting to Outcomes® Implementation Support Intervention

Problematic rates of alcohol, e-cigarette, and other drug use among US adolescents highlight the need for effective implementation of evidence-based programs (EBPs), yet schools and community organizations have great difficulty implementing and sustaining EBPs. Although a growing number of studies show that implementation support interventions can improve EBP implementation, the literature on how to improve sustainability through implementation support is limited. This randomized controlled trial advances the literature by testing the effects of one such implementation intervention-Getting To Outcomes (GTO)-on sustainability of CHOICE, an after-school EBP for preventing substance use among middle-school students. CHOICE implementation was tracked for 2 years after GTO support ended across 29 Boys and Girls Club sites in the greater Los Angeles area. Predictors of sustainability were identified for a set of key tasks targeted by the GTO approach (e.g., goal setting, evaluation, collectively called "GTO performance") and for CHOICE fidelity using a series of path models. One year after GTO support ended, we found no differences between GTO and control sites on CHOICE fidelity. GTO performance was also similar between groups; however, GTO sites were superior in conducting evaluation. Better GTO performance predicted better CHOICE fidelity. Two years after GTO support ended, GTO sites were significantly more likely to sustain CHOICE implementation when compared with control sites. This study suggests that using an implementation support intervention like GTO can help low-resource settings continue to sustain their EBP implementation to help them get the most out of their investment. ClinicalTrials.gov Identifier: NCT02135991.

Supporting systemic social and emotional learning with a schoolwide implementation model

A strong body of research indicates that evidence-based programs designed to promote social and emotional learning (SEL) can lead to positive developmental outcomes for children and youth. Although these evidence-based programs have demonstrated benefits for students, it is also well-established that programs must be implemented with quality and sustained to maximize positive outcomes. To support schools in implementing SEL that is integrated into all aspects of a school community, the CASEL School Guide implementation model was developed to guide school leadership teams in establishing a vision; selecting, implementing, and sustaining evidence-based programs; and integrating SEL schoolwide. The School Guide model is based on CASEL's School Theory of Action which includes planning, implementing, and monitoring schoolwide SEL. This paper describes findings about the feasibility and degree of implementation of this model as carried out by 14 school leadership teams who were supported by SEL coaches. The participating schools implemented an evidence-based SEL program as the foundation of their efforts to promote schoolwide SEL. Findings demonstrate the feasibility of implementing this model in urban schools that primarily serve students of color, as all teams successfully carried out the implementation model and demonstrated increased levels of capacity related to supporting schoolwide SEL.

A Cluster-Randomized Trial of Getting To Outcomes' Impact on Sexual Health Outcomes in Community-Based Settings

The USA has high teen pregnancy rates compared to other developed nations. Many community-based organizations need assistance conducting evidence-based teen pregnancy prevention programs (EBPs) appropriately. This study evaluated the impact of an implementation support intervention called Getting To Outcomes (GTO) designed to help such organizations. This cluster randomized controlled trial compared 16 Boys and Girls Clubs (BGCs) implementing a teen pregnancy prevention EBP called Making Proud Choices for two years, with 16 BGCs implementing MPC augmented with GTO training, tools, and technical assistance. Participating middle school youth were compared on proximal outcomes (knowledge, attitudes, and intentions about sex and condoms from baseline to post) and sexual behaviors (frequency of sex and condom use, from baseline to 6-month follow-up). In year 1, there were no significant effects of GTO for any proximal outcome. After GTO-stimulated quality improvement in year 2, the GTO group improved significantly more on condom attitudes and use intentions. Frequency of sex and condom use did not differ between the two groups in either year; however, base rates of these behaviors in the sample were very low. Findings suggest that in typical community-based settings, detailed manuals and training common to structured EBPs may be sufficient to yield some improvement in key proximal outcomes, but that more systematic implementation support is needed to achieve greater improvement in these outcomes. Using GTO with many communities, as currently supported by various federal agencies, could yield public health impact via improvements in condom attitudes and use intentions.

Preventing Adolescent Substance Use Through an Evidence-Based Program: Effects of the Italian Adaptation of Life Skills Training

Evidence-based preventive interventions for adolescent substance use, violence, and mental health issues are increasingly being adapted and disseminated internationally. In the present paper, we report the results of an effectiveness study that was part of a comprehensive initiative by a coalition of health promotion organizations in the Lombardy region of Italy to select, culturally adapt, implement, evaluate, and sustain an evidence-based drug abuse prevention program developed in the USA. Findings are presented from a large-scale effectiveness study of the Life Skills Training prevention program among over 3000 students attending 55 middle schools in Italy. The prevention program taught drug refusal skills, antidrug norms, personal self-management skills, and general social skills. Relative to comparison group students, students who received the prevention program were less likely to initiate smoking at the post-test and 2-year follow-up, and less likely to initiate weekly drunkenness at the 1-year follow-up. The program had direct positive effects on several cognitive, attitudinal, and skill variables believed to play a protective role in adolescent substance use. The findings from this study show that a drug abuse prevention program originally designed for adolescents in the USA is effective in a sample of Italian youth when a rigorous and systematic approach to cultural adaptation is followed that incorporates the input of multiple stakeholders.

A large-scale initiative to disseminate an evidence-based drug abuse prevention program in Italy: Lessons learned for practitioners and researchers

Across developed countries, experimentation with alcohol, tobacco, and other drugs often begins in the early adolescent years. Several evidence-based programs have been developed to prevent adolescent substance use. Many of the most rigorously tested and empirically supported prevention programs were initially developed and tested in the United States. Increasingly, these interventions are being adopted for use in Europe and throughout the world. This paper reports on a large-scale comprehensive initiative designed to select, adapt, implement, and sustain an evidence-based drug abuse prevention program in Italy. As part of a large-scale regionally funded collaboration in the Lombardy region of Italy, we report on processes through which a team of stakeholders selected, translated and culturally adapted, planned, implemented and evaluated the Life Skills Training (LST) school-based drug abuse prevention program, an evidence-based intervention developed in the United States. We discuss several challenges and lessons learned and implications for prevention practitioners and researchers attempting to undertake similar international dissemination projects. We review several published conceptual models designed to promote the replication and widespread dissemination of effective programs, and discuss their strengths and limitations in the context of planning and implementing a complex, large-scale real-world dissemination effort.

Knowledge, Attitudes, and Commitment Concerning Evidence-Based Prevention Programs: Differences between Family and Consumer Sciences and 4-H Youth Development Educators

In this article, we describe the results of a study designed to assess knowledge, perceptions, and attitudes towards evidence-based and other prevention programs among county Extension educators. We examined differences across educators from the Family and Consumer Sciences (FCS) and 4-H Youth Development program areas. Analyses based on a multi-state sample of educators revealed significant differences across program areas such that, compared to their 4-H counterparts, FCS educators were more knowledgeable of evidence-based programs and had more favorable perceptions and attitudes towards evidence-based and other pre-packaged prevention programs. These findings suggest that Cooperative Extension administrators should work to encourage the use of evidence-based and other prevention programs, particularly within the 4-H program area.