Understanding the Organizational Implementation Context of Schools: A Qualitative Study of School District Administrators, Principals, and Teachers

Harnessing Human-Centered Design for Evidence-Based Psychosocial Interventions and Implementation Strategies in Community Settings: Protocol for Redesign to Improve Usability, Engagement, and Appropriateness

BACKGROUND

Although substantial progress has been made in establishing evidence-based psychosocial clinical interventions and implementation strategies for mental health, translating research into practice-particularly in more accessible, community settings-has been slow.

OBJECTIVE

This protocol outlines the renewal of the National Institute of Mental Health-funded University of Washington Advanced Laboratories for Accelerating the Reach and Impact of Treatments for Youth and Adults with Mental Illness Center, which draws from human-centered design (HCD) and implementation science to improve clinical interventions and implementation strategies. The Center's second round of funding (2023-2028) focuses on using the Discover, Design and Build, and Test (DDBT) framework to address 3 priority clinical intervention and implementation strategy mechanisms (ie, usability, engagement, and appropriateness), which we identified as challenges to implementation and scalability during the first iteration of the center. Local redesign teams work collaboratively and share decision-making to carry out DDBT.

METHODS

All 4 core studies received institutional review board approval by June 2024, and each pilot project will pursue institutional review board approval when awarded. We will provide research infrastructure to 1 large effectiveness study and 3 exploratory pilot studies as part of the center grant. At least 4 additional small pilot studies will be solicited and funded by the center. All studies will explore the use of DDBT for clinical interventions and implementation strategies to identify modification targets to improve usability, engagement, and appropriateness in accessible nonspecialty settings (Discover phase); develop redesign solutions with local teams to address modification targets (Design and Build phase); and determine if redesign improves usability, engagement, and appropriateness (Test phase), as well as implementation outcomes. Center staff will collaborate with local redesign teams to develop and test clinical interventions and implementation strategies for community settings. We will collaborate with teams to use methods and centerwide measures that facilitate cross-project analysis of the effects of DDBT-driven redesign on outcomes of interest.

RESULTS

As of January 2025, three of the 4 core studies are underway. We will generate additional evidence on the robustness of DDBT and whether combining HCD and implementation science is an asset for improving clinical interventions and implementation strategies.

CONCLUSIONS

During the first round of the center, we established that DDBT is a useful approach to systematically identify and address chronic challenges of implementing clinical interventions and implementation strategies. In this subsequent grant, we expect to increase evidence of DDBT's impact on clinical interventions and implementation strategies by expanding a list of common challenges that could benefit from modification, a list of exemplary solutions to address these challenges, and guidance on using the DDBT framework. These resources will contribute to broader discourse on how to enhance implementation of clinical interventions and implementation strategies that integrate HCD and implementation science.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/65446.

Testing an Organizational Implementation Process Model Related to Teachers' Implementation-Related Attitudes and Behaviors: a Multilevel Mediation Analysis

The effectiveness of school-based universal prevention programs is frequently diminished due to low-quality implementation. Organizational factors support high-quality implementation because of their broad influence across implementers. Conceptually, implementation leadership (i.e., behaviors that prioritize, reward, and support evidence-based practice [EBP] implementation) works to embed a favorable implementation climate (i.e., implementers' collective perceptions that their organization prioritizes, rewards, and support EBP implementation) leading to improved implementation citizenship behavior and attitudes toward EBP. This organizational implementation process model has some empirical support but has not been tested in a multilevel framework or related to hypothesized attitudinal and behavioral outcomes. The sample included 319 teachers across 39 US public elementary schools; all were implementing Schoolwide Positive Behavior Interventions and Supports. Multilevel mediation (level 1 = teacher, level 2 = school) was used to test the indirect association of implementation leadership on implementation-related attitudes and behaviors via implementation climate across two time points (fall and spring). At the school level, the organizational implementation process model was validated related to implementation citizenship behavior, but not attitudes toward EBP. At the teacher level, the process model was validated related to both outcomes, and there was a significant direct effect of implementation leadership on attitudes toward EBP. Developing strong leaders for implementation seems key to achieving high-quality EBP implementation. Implications for schools, principal training, and research are discussed.

Partnering with Educators to Iteratively Co-create Tools to Support Teachers’ Use of Equity-Focused Positive Behavioral Supports

In the Maximize Project, we are engaging in a research-practice partnership to co-create implementation strategies to facilitate elementary school teachers’ use of equity-focused positive behavior supports (EF-PBS). In this paper, we describe the processes used to build an interactive, technology-based platform to enhance teachers’ use of EF-PBS via self-reflection, self-assessment, goal setting, and goal review. We describe how we established a multi-disciplinary, multi-state community advisory board to collaborate on Version 1 of platform (Phase 1). We explain how we obtained quantitative and qualitative feedback about the platform from educators in three partnering schools, and how we used those data to produce Version 2 of the platform (Phase 2). Platform use data suggested high utilization in Quarter 1 (August–October) of the school year, when there was protected time to complete activities. However, platform use was moderate in Quarter 2 (October–December) and low in Quarters 3 and 4 (January–May). Educator feedback revealed moderate acceptability, feasibility, and appropriateness of the platform and highlighted ways to improve the user experience (e.g., streamlining steps in goal setting, making resources about strategy implementation easier to find). We discuss lessons learned to inform school mental health co-creation endeavors, including strategies for supporting diverse perspectives, for enhancing advisory board members’ voices and confidence, and for creating practical and feasible methods for teachers to benefit from co-created technology-based implementation strategies. Our processes offer guidance for others engaging in research-practice partnerships, developing education technologies and/or supporting teachers’ use of equity-focused practices to improve daily school experiences for all students.

The perspective of school leaders on the implementation of evidence-based practices: A mixed methods study

Background

School leaders play an integral role in the use of implementation strategies, which in turn support special education teachers in the implementation of evidence-based practices (EBPs). In this convergent mixed methods study, we explored school leaders' perceptions of the facilitators and barriers to EBP implementation, particularly for students receiving special education, as well as the importance and feasibility of 15 implementation strategies.

Method

School leaders (N  =  22, principals, assistant principals, school psychologists, etc.) participated in a semistructured interview that included three parts-qualitative questions, quantitative ratings of strategies' importance and feasibility, and discussion of the top three implementation strategies. Data strands were analyzed independently and then integrated to generate meta-inferences.

Results

The qualitative data identified facilitators such as access to resources about the intervention (e.g., professional development) and collaboration and teamwork, while barriers centered on lack of school supports, culture/climate, and organizational factors (e.g., lack of communication). The quantitative data indicated that the implementation strategy provide ongoing consultation/coaching was rated as important and feasible. Monitor the progress of the implementation effort was rated as important but less feasible, while conduct educational meetings and change the environment were rated as feasible, but less important. Build partnerships to support implementation was rated as less important and feasible. There was convergence and divergence in mixed methods findings.

Conclusion

This study underscores the critical need to increase school leaders' knowledge and skills related to implementation science to better leverage implementation strategies that address the confluence of relevant implementation determinants.

Implementation leadership and implementation climate in context: A single organization intrinsic case study for implementation of digital measurement-based care

Background

Although studies have demonstrated that implementation leadership and climate are important constructs in predicting evidence-based practice (EBP) implementation, concrete descriptions of how they operate during organizational implementation efforts are lacking. This case study fills that gap through an in-depth description of an organization with effective implementation leadership that successfully built a strong implementation climate. This case study provides an illustration of implementation leadership and climate in tangible, replicable terms to assist managers, practitioners, and researchers in addressing the organizational context in their own implementation projects.

Method

A single organization, intrinsic case study was employed to paint a multifaceted picture of how one organization leveraged implementation leadership to strengthen a climate for the successful implementation of digital measurement-based care. The case was drawn from a cluster-randomized trial designed to test the effects of a leadership-focused implementation strategy on youth-level fidelity and clinical outcomes of digital measurement-based care. Following the completion of the trial, case study activities commenced. Descriptive summaries of multiple data sources (including quantitative data on implementation leadership and climate, coaching call and organizational alignment meeting recordings and notes, and development plans) were produced and revised iteratively until consensus was reached. Leadership actions were analyzed for corresponding dimensions of implementation leadership and climate.

Results

Specific actions organizational leaders took, as well as the timing specific strategies were enacted, to create a climate for implementation are presented, along with lessons learned from this experience.

Conclusion

This case study offers concrete steps organizational leaders took to create a consistent and aligned message that the implementation of a specific EBP was a top priority in the agency. The general approach taken to create an implementation climate provides several lessons for leaders, especially for EBPs that have broad implications across an organization.

Co-design and Development of Implementation Strategies: Enhancing the PAX Good Behaviour Game in Australian Schools

Effective implementation strategies are important for take-up of programs in schools. However, to date, few implementation strategies have been co-designed with teachers and support staff (including principals) in Australia. The aim of this study was to iteratively co-design multiple implementation strategies to enhance the delivery of mental health prevention program, PAX Good Behaviour Game, in New South Wales primary schools. The secondary aim was to evaluate the acceptability of the implementation strategies from the perspective of school staff. Twenty-nine educational staff (including principals) informed the co-design of the implementation strategies across three phases. Phase 1 involved a rapid review of the literature and stakeholder meetings to agree upon potential evidence-based strategies. Phase 2 involved focus group discussions with educational staff to co-design implementation strategies. Phase 3 involved semi-structured interviews with school staff to assess strategy acceptability after implementation at 6-months post-baseline. Data were analysed using deductive, framework analysis. The final co-designed intervention included nine implementation strategies accessible through a toolkit delivered to the school's leadership team. These strategies were deemed acceptable in school settings that experienced periods of both face-to-face and remote learning due to the changing COVID-19 situation in 2021. This paper contributes to the implementation literature by transparently reporting how educational staff-informed implementation strategies were iteratively co-designed. This will provide a roadmap for other researchers to co-design implementation strategies to further support the delivery of evidence-based prevention programs in schools.

Barriers and Facilitators to the Online Delivery of a School-Based Intervention to Reduce Racial Trauma

Students of color face disproportionate rates of adverse childhood experiences, including racial discrimination in school settings. Effective interventions strategies are needed to address school-based racial trauma. Link for Equity was designed to be a culturally-responsive trauma-informed intervention that includes universal cultural humility training for teachers. As a result of the COVID-19 pandemic, the in-person trauma-informed cultural humility training was adapted for online delivery. The purpose of this study was to assess barriers and facilitators that impacted online delivery of the training. We conducted semi-structured interviews with 25 high school teachers from three public school districts in the Midwestern United States who participated in the online training. Interview transcripts were coded by two team members, and thematic analysis was utilized. Barriers and facilitators to online delivery were identified across five domains: receptivity, logistics, engagement, comfort, and application. The implications of these barriers and facilitators are discussed and tailored recommendations for the virtual delivery of culturally-responsive trauma-informed interventions to reduce racial discrimination in schools are provided.

Exploring the acceptability, feasibility, and appropriateness of a communication-friendly classroom tool for use in Irish schools: A qualitative inquiry

BACKGROUND

Ten percent of the school-aged population have speech, language, and communication needs (SLCN) that impact access to the curriculum. Successful implementation of classroom-based SLCN interventions can reduce barriers to learning, thereby improving educational outcomes for this vulnerable population. The challenges of implementing innovations in educational settings are well-documented, yet limited studies have addressed such considerations when developing, and piloting universal level SLCN interventions for use in Irish schools.

METHODS

A qualitative exploratory study was undertaken to establish the acceptability, feasibility, and appropriateness of a universal level SLCN intervention. An advisory panel of teachers (n = 8) and children with SLCN (n = 2) were engaged as co-researchers in the study. The Communication Supporting Classrooms Observation Tool, developed as part of the Better Communication Project in the UK, was trialled across a diverse sample of school settings (n = 5). Semi-structured interviews were conducted with school practitioners and school leaders, and a deductive content analysis was undertaken using the domains of the Consolidation Framework for Implementation Research.

DISCUSSION

The observation tool was viewed as acceptable with suggested additions. Integrating use of the tool within existing data-informed, school self-evaluation processes aimed at supporting school improvement was noted as a potential means of supporting implementation. A knowledge gap in relation to school-based models of support for SLCN was identified which may negatively impact implementation. An implementation strategy targeting coherence, cognitive engagement and contextual integration is indicated if the tool is to be normalised into routine practice in Irish classrooms. Implementation needs appeared to vary at the school level.

CONCLUSIONS

The importance of early-stage exploration to guide implementation planning with regards to developing and testing universal level interventions for SLCN in schools is highlighted. Engaging an advisory panel provides important insights to guide implementation decisions. Findings suggest an adaptive design is required when planning implementation studies targeting classroom setting.

Sticking with Programs That Do Not Work: The Role of Escalation of Commitment in Schools

Schools are the most common site to implement evidence-based prevention programs and practices (EBPs) to improve behavioral and mental health outcomes among children and adolescents. Research has highlighted the critical role of school administrators in the adoption, implementation, and evaluation of such EBPs, focusing on the factors they should consider during the adoption decision and the behaviors needed for successful implementation. However, scholars have only recently begun to focus on the de-adoption or de-implementation of low-value programs and practices to make room for evidence-based alternatives. This study introduces escalation of commitment as a theoretical framework for understanding why school administrators may stick with ineffective programs and practices. Escalation of commitment is a robust decision-making bias in which people feel compelled to continue with a course of action even when performance indicators suggest it is not going well. Using grounded theory methodology, we conducted semi-structured interviews with 24 building- and district-level school administrators in the Midwestern United States. Results suggested that escalation of commitment occurs when administrators attribute the underlying causes of poor program performance not to the program itself but instead to issues related to implementation, leadership, or the limitations of the performance indicators themselves. We also identified a variety of psychological, organizational, and external determinants that accentuate administrators' continuance of ineffective prevention programs. Based on our findings, we highlight several contributions to theory and practice.

Teacher attitudes toward evidence-based practices: Exploratory and confirmatory analyses of the school-adapted evidence-based practice attitude scale

Background

The Evidence-Based Practice Attitudes Scale (EBPAS) is widely used in implementation research, but it has not been adapted and validated for use among general education teachers, who are most likely to deliver evidence-based prevention programs in schools, the most common setting where youth access social, emotional, and behavioral health services.

Method

School-based stakeholders and a research team comprised of experts in the implementation of evidence-based practices in schools adapted the EBPAS for teachers (the S-EBPAS). The adapted instrument was administered to a representative sample ( n = 441) of general education teachers (grades K—5) to assess the reliability and internal consistency via factor analyses. The S-EBPAS included two forms (i.e., EBP-agnostic and EBP-specific item referents), therefore, a multiple-group confirmatory factor analysis (CFA) was also performed to establish measurement invariance between the two forms.

Results

After adaptation and refinement, a 9-item, 3-factor structure was confirmed, with the final model supporting three first-order factors that load onto a second-order factor capturing attitudes toward adopting evidence-based practices. Multiple-group CFA analyses of measurement invariance indicated there were no significant differences between the two forms.

Conclusions

Overall, this study provides a brief, flexible instrument capturing attitudes toward adopting EBPs that has high reliability and internal consistency, which support its use among general education teachers in school settings implementing evidence-based practices.

Plain Language Summary

The Evidence-Based Practice Attitudes Scale (EBPAS) is a popular instrument for measuring attitudes toward evidence-based practices (EBPs). This instrument provides valuable information during implementation initiatives, such as whether providers or front-line implementers have favorable attitudes toward a given practice. The EBPAS has been used in many different settings, such as in community-based mental health clinics, medical hospitals, and in child welfare. However, it's use in schools has been limited, and it has not yet been tested with general education teachers, who are key implementers of evidence-based practices in schools. In order to trust that the scores from an instrument are accurate, it needs to be evaluated when scaling it out to new populations and settings. One popular method to determine this is to use factor analysis, which was employed in this study. This study fills the identified gap by assessing the reliability (i.e., accuracy) and internal consistency of the EBPAS among a representative sample of general education teachers. Findings from this study indicate that the school-adapted EBPAS (S-EBPAS) is a brief, nine-item instrument that provides a reliable estimate of teachers’ attitudes toward evidence-based practices. Our results also provide evidence that the S-EBPAS can be used to capture attitudes toward specific EBPs as well as attitudes toward EBP -agnostic. This study provides a flexible instrument that can be used by school-based implementation researchers, practitioners, and intermediaries at multiple phases of implementation projects, such as when exploring a new EBP to adopt.

Individual and organizational factors as predictors of early evidence-based practice adoption in Michigan high schools: Baseline data from an implementation trial

Background

Adolescents increasingly access mental health services in schools. School mental health professionals (SPs; school counselors, social workers, etc.) can offer evidence-based mental health practices (EBPs) in schools, which may address access gaps and improve clinical outcomes. Although some studies have assessed factors associated with EBP adoption in schools, additional research focusing on SP- and school-level factors is warranted to support EBP implementation as SPs' mental health delivery grows.

Methods

Baseline data were collected from SPs at Michigan high schools participating in a statewide trial to implement SP-delivered cognitive behavioral therapy (CBT) to students. Models examined factors associated with attitudes about EBPs, implementation climate, and implementation leadership, and their associations with CBT knowledge, training attendance, and pre-training CBT delivery.

Results

One hundred ninety-eight SPs at 107 schools (87%) completed a baseline survey. The mean Evidence-Based Practice Attitude Scale (EBPAS) total score was 2.9, and school-aggregated mean scores of the Implementation Climate Scale (ICS) and Implementation Leadership Scale (ILS) were 1.83 and 1.77, respectively, all on a scale ranging from 0 (low) to 4 (high). ICS and ILS scores were lower than typically reported in clinical settings, while EBPAS scores were higher. School characteristics were not significantly associated with EBPAS, ICS, or ILS scores, but scores did differ by SP role. Higher EBPAS scores were associated with more CBT knowledge (average marginal effect for 1 SD change [AME] = 0.15 points) and a higher probability of training completion (AME = 8 percentage points). Higher ICS scores were associated with a higher probability of pre-training CBT delivery (AME = 6 percentage points), and higher ILS scores were associated with higher probability of training completion (AME = 10 percentage points).

Conclusions

Our findings suggest that SPs' attitudes toward EBPs and organizational support were positively associated with early signs of implementation success. As schools increasingly fill the adolescent mental healthcare access gap, efforts to strengthen both provider attitudes toward EBP and strategic organizational factors supporting EBP delivery will be key to encouraging EBP uptake in schools.

Plain Language Summary

Schools are an important setting in which adolescents receive mental healthcare. We need to better understand how to implement evidence-based practices (EBPs) in this setting to improve student mental health. This study examined the attitudes and perceptions of school professionals (SPs) as key contributors to the implementation of a particular EBP, the delivery of cognitive behavioral therapy (CBT) in schools. The study found that implementation climate and leadership scores in participating schools were lower than scores typically reported in clinical settings, while scores for SP attitudes about EBP adoption were higher than typical scores in clinical settings. Results further suggest that SPs with more positive attitudes toward EBPs are more knowledgeable of CBT and more likely to complete a 1-day CBT training. We also found that higher implementation climate scores were associated with SPs reporting pre-training CBT delivery (although this association was not statistically significant), and more implementation leadership was associated with SPs completing the CBT training. These findings suggest that SP attitudes toward EBPs and organizational support in schools are positively associated with early signs of implementation success. Early, low-intensity efforts to (1) improve SP attitudes about mental health EBPs, and (2) increase schools' support for implementation may scaffold more intensive implementation efforts in schools down the road.

Predicting school readiness program implementation in community-based childcare centers

Introduction

Targeted curricular interventions can increase preschool program quality and boost children's academic and social-emotional readiness skills, but variable funding and weak organizational infrastructure in many community-based childcare centers may reduce the effective implementation of these programs.

Method

This study examined individual teacher and workplace predictors of the REDI program implementation, a targeted school readiness program that was adapted to support delivery in childcare centers. REDI was delivered by 63 teachers in 37 community-based childcare centers with center directors serving as local implementation coaches.

Results

Results showed that individual teacher factors (e.g., teaching skills and receptivity to intervention consultation) predicted the quality with which REDI activities and teaching strategies were delivered, and workplace factors were important predictors across multiple implementation indicators.

Discussion

Practice and policy implications for improving intervention implementation and corresponding program quality in childcare centers are highlighted.

Power and resistance in schools: Implementing institutional change to promote health equity for sexual and gender minority youth

Introduction

Schools in the United States are hierarchical institutions that actively (re)produce the power relations of the wider social world, including those associated with heteronormativity. Structural stigma, informed by heteronormativity and perpetuated through schools, contributes to the production of academic and health disparities among youth who are lesbian, gay, bisexual, transgender, queer, or of other gender and sexual identities (LGBTQ+). We draw upon 5 years of qualitative data from a cluster randomized controlled trial conducted in New Mexico that used implementation science frameworks to promote the uptake and sustainment of evidence-informed practices (EIPs) to examine how power operates to hinder or promote the ability of school staff to change school environments, disrupt structural stigma, and increase safety and support for LGBTQ+ youth.

Methods

Data sources included annual individual and small group qualitative interviews with school professionals (e.g., administrators, school nurses, teachers, and other staff), several of whom took part in Implementation Resource Teams (IRTs) charged with applying the EIPs. Other data sources included bi-weekly periodic reflections with implementation coaches and technical assistance experts. Data were recorded, transcribed, and analyzed using deductive and inductive coding techniques.

Results

The IRTs experienced variable success in implementing EIPs. Their efforts were influenced by: (1) constraining school characteristics, including staff turnover and resource scarcity; (2) community-based opposition to change and concerns about community backlash; (3) the presence or absence of supportive school leadership; and (4) variations in school, district, and state policies affecting LGBTQ+ students and attitudes about their importance. Findings illustrate how diverse power structures operated in and across outer and inner contexts to bound, shift, amplify, and otherwise shape how new practices were received and implemented.

Conclusion

Findings indicate that the efforts of IRTs were often a form of resistant power that operated within and against school hierarchies to leverage epistemic, discursive, and material power toward implementation. To improve health equity, implementation scientists must attend to the multiple real and perceived power structures that shape implementation environments and influence organizational readiness and individual motivation. Implementers must also work to leverage resistant power to counter the institutional structures and social norms that perpetuate inequities, like heteronormativity and structural stigma.

Testing a Theory of Implementation Leadership and Climate Across Autism Evidence-Based Interventions of Varying Complexity

Prominent theories within the field of implementation science contend that organizational leaders can improve providers' fidelity to evidence-based practices (EBPs) by using focused implementation leadership behaviors that create an organizational climate for EBP implementation. However, this work has been criticized for overreliance on nonspecific, self-report fidelity measures and poor articulation of the boundary conditions that may attenuate leadership and climate's influence. This study tests the predictions of EBP implementation leadership and climate theory on observed fidelity to three school-based EBPs for autism that vary in complexity: pivotal response training (PRT), discrete trial training (DTT), and visual schedules (VS). Educators in kindergarten to third-grade autism support classrooms in 65 schools assessed their principals' EBP implementation leadership and school EBP implementation climate prior to the school year. Mid-school year, trained observers rated educator fidelity to all three interventions. Expert raters confirmed PRT was significantly more complex than DTT or VS using the Intervention Complexity Assessment Tool for Systematic Reviews. Linear regression analyses at the school level indicated principals' increased frequency of EBP implementation leadership predicted a higher school EBP implementation climate, which in turn predicted higher educator fidelity to PRT-however, there was no evidence of a relationship between implementation climate and fidelity to DTT or VS. Comparing principals whose EBP implementation leadership was ±1 SD from the mean, there was a significant indirect association of EBP implementation leadership with PRT fidelity through EBP implementation climate (d = 0.49, 95% CI [0.04, 0.93]). Strategies that target EBP implementation leadership and climate may support fidelity to complex behavioral interventions.

Teacher Perceptions of Implementation Climate Related to Feasibility of Implementing Schoolwide Positive Behavior Supports and Interventions

For Schoolwide Positive Behavior Interventions and Supports (SWPBIS) to be effective, educators must use the practices as intended. Whether a teacher uses a practice as intended can depend, in part, on how feasible the practice is perceived to be. Identifying malleable factors associated with teachers’ perceptions of SWPBIS feasibility can help schools identify targeted supports to improve feasibility. Implementation climate, or the shared perception among implementers that their school supports implementation efforts, is known to promote high quality implementation. However, little is known about how individuals’ perceptions of their school’s implementation climate may influence their perceptions of feasibility. The lack of empirical evidence points to a need to explore whether educators’ shared and individual perceptions of implementation climate influence feasibility of implementing SWPBIS. Therefore, this study examines the link between teachers’ individual and shared perceptions of implementation climate related to the feasibility of implementing SWPBIS in a sample of 348 K-5 general education teachers across 39 elementary schools in the pacific northwestern United States. Results indicate that teachers who experience their schools’ implementation climate as positive are more likely to report SWPBIS as feasible, controlling for teachers’ shared perceptions of implementation climate. Implications for schools aiming to improve their implementation of SWPBIS, including the development of individualized implementation supports (e.g., tailoring implementation strategies to support each and every teacher), are discussed.

Implementation fidelity, attitudes, and influence: a novel approach to classifying implementer behavior

Background

The current study sought to (1) describe a new classification approach for types of implementer behavior and (2) explore the implementer behavior change in response to tailored implementation facilitation based on the classifications.

Methods

A small-scale, cluster-randomized hybrid type III implementation trial was conducted in 38 early care and education classrooms that were part of the Together, We Inspire Smart Eating (WISE) program. WISE focuses on 4 evidence-based practices (EBPs), which are implemented by teachers to promote nutrition. External facilitators (N = 3) used a modified Rapid Assessment Procedure Informed Clinical Ethnography (RAPICE) to complete immersion (i.e., observations) and thematic content analyses of interviews to identify the characteristics of teachers’ behavior at varying levels of implementation fidelity. Three key factors—attitudes toward the innovation, fidelity/adaptations, and influence—were identified that the research team used to classify teachers’ implementation behavior. This process resulted in a novel classification approach. To assess the reliability of applying the classification approach, we assessed the percent agreement between the facilitators. Based on the teachers’ classification, the research team developed a tailored facilitation response. To explore behavior change related to the tailored facilitation, change in fidelity and classification across the school year were evaluated.

Results

The classifications include (1) enthusiastic adopters (positive attitude, meeting fidelity targets, active influence), (2) over-adapting adopters (positive attitude, not meeting fidelity targets, active influence), (3) passive non-adopters (negative attitude, not meeting fidelity targets, passive influence), and (4) active non-adopters (negative attitudes, not meeting fidelity targets, active influence). The average percent agreement among the three facilitators for classification was 75%. Qualitative data support distinct patterns of perceptions across the classifications. A positive shift in classification was observed for 67% of cases between the mid-point and final classification. Finally, we generated an expanded classification approach to consider additional combinations of the three factors beyond those observed in this study.

Conclusions

Data from this study support the ability to apply the classification approach with moderate to high reliability and to use the approach to tailor facilitation toward improved implementation. Findings suggest the potential of our approach for wider application and potential to improve tailoring of implementation strategies such as facilitation.

Evaluation of a Large-Scale School Wellness Intervention Through the Consolidated Framework for Implementation Research (CFIR): Implications for Dissemination and Sustainability

Background

Numerous studies have tested school-based interventions promoting healthy behaviors in youth, but few have integrated dissemination and implementation (D&I) frameworks. Using D&I frameworks can inform if and how an evidence-based intervention is implemented and maintained and provide strategies to address contextual barriers. Such application is necessary to understand how and why interventions are sustained over time. We evaluated a school wellness initiative called SWITCH® (School Wellness Integration Targeting Child Health) to (1) assess implementation outcomes of adoption, fidelity, and penetration, (2) discern implementation determinants through the Consolidated Framework for Implementation Research (CFIR), and (3) examine differences among inexperienced and experienced schools and influential factors to sustainment.

Methods

A total of 52 schools from Iowa, United States enrolled in the 2019-2020 iteration of SWITCH (22 inexperienced; 30 experienced). The CFIR guided the adaptation of mixed methods data collection and analysis protocols for school settings. Specific attention was focused on (1) fidelity to core elements; (2) adoption of best practices; and (3) penetration of behavior change practices. Determinants were investigated through in-depth qualitative interviews and readiness surveys with implementation leaders. A systematic process was used to score CFIR domains (between -2 and +2) indicating positive or negative influence. Independent t-tests were conducted to capture differences between samples, followed by a cross-case analysis to compare determinants data. Inductive coding yielded themes related to sustainment of SWITCH beyond formal implementation support.

Results

Experienced schools had higher scores on fidelity/compliance (t = -1.86 p = 0.07) and adoption (t = -2.03 p = 0.04). CFIR determinants of innovation source, culture, relative priority, and leadership engagement were positive implementation determinants, whereas tension for change and networks and communications were negative determinants. Distinguishing factors between experienced and inexperienced schools were Readiness for Implementation and Self-efficacy (experienced significantly higher; p < 0.05). Strategies to enhance sustainability were increasing student awareness/advocacy, keeping it simple, and integrating into school culture.

Conclusions

Findings provide specific insights related to SWITCH implementation and sustainability but more generalized insights about the type of support needed to help schools implement and sustain school wellness programming. Tailoring implementation support to both inexperienced and experienced settings will ultimately enhance dissemination and sustainability of evidence-based interventions.

Construct validity of the school-implementation climate scale

Background

Implementation climate is an organizational construct theorized to facilitate the adoption and delivery of evidence-based practices. Within schools, teachers often are tasked with implementing universal prevention programs. Therefore, they are ideal informants when assessing school implementation climate for initial and continuous implementation improvement efforts. The purpose of this study was to examine the construct validity (i.e., factor structure and convergent/divergent validity) of a school-adapted measure of strategic implementation climate called the School Implementation Climate Scale (SICS).

Methods

Confirmatory factor analyses of SICS data, collected from 441 teachers in 52 schools, were used to compare uncorrelated and correlated first-order factor models and a second-order hierarchical model. Correlations with other school measures were examined to assess SICS convergent and divergent validities.

Results

Results demonstrated acceptable internal consistency for each SICS subscale ( αs > 0.80 for all subscales) and construct validity of the hypothesized factor structure of the SICS with three new scales. The hierarchical second-order factor structure with eight first-order factors was found to best model the SICS data. Correlations with other school measures were in the expected direction and magnitude.

Conclusions

Results from this study provide psychometric evidence that supports the use of the SICS to inform the implementation research and practice in schools.

Plain Language Summary

Schools are busy trying to implement various universal programs and systems to help support kids in their growth. Beginning and sustaining these efforts is quite challenging, and there is need for tools and ideas to help those implementation efforts. One concept is implementation climate, which is broadly the school staff’s perception of the implementation support for a given practice. However, no measure currently exists to help schools assess their implementation climate. The goal of our study was to adapt a measure of implementation climate used in other settings to the school environment. We used feedback from educational experts to make changes and used various analyses to determine if the newly adapted measure was psychometrically sound. Findings suggest the new measure is usable to guide implementation efforts in schools.

Implementation Leadership in School Nutrition: A Qualitative Study

OBJECTIVE

This paper identifies implementation leadership characteristics in the school nutrition setting and places findings in the context of implementation leadership literature.

METHODS

Fourteen interviews were conducted with school district leadership/staff in an urban school district. Modified grounded theory was employed.

RESULTS

Four themes emerged: (1) understanding of technical/operational intervention details; (2) ability to proactively develop and communicate plans; (3) supervisory oversight; and (4) intervention framing. Themes were consistent with 4 of the 5 dimensions comprising the Implementation Leadership Scale: knowledgeable, proactive, perseverant, and distributed leadership. The supportive domain was not a major finding. An additional domain, how leaders message the intervention to staff, was identified.

CONCLUSIONS AND IMPLICATIONS

Implementation leadership in school nutrition appears similar, but not identical, to leader behaviors present in the Implementation Leadership Scale. School nutrition leaders might consider involving staff early in implementation planning, incorporating technical expertise, and clearly communicating the intervention purpose to support successful implementation. Future research might explore the interplay between leadership and implementation outcomes.

Organizational Context in General and Special Education: An Exploratory Investigation to Describe the Perspective of School Leaders

Organizational context (e.g., climate, culture, resources) can impede or enhance implementation of evidence-based practices in general education settings or special education settings serving students with autism spectrum disorder. We examined the relations between organizational context and individual (i.e., implementation leadership, administrator- or service provider-role) or school (i.e., enrollment size, public/nonpublic school type) characteristics. Participants were administrative or service providing leaders (n = 34) from 11 schools in one state on the East Coast of the United States. School leaders' average ratings of the organizational context were generally more positive for special education than general education; however, greater culture stress was reported for special education. Correlation analyses indicated being an administrator and implementation leadership were positively associated with implementation climate in both education settings. Being an administrator was also positively associated with cultural effort (i.e., how hard people work towards achieving goals) in special education, but negatively associated with culture stress in general education. In special education, nonpublic schools had better climates (both learning and implementation), but more culture stress. Additionally, school enrollment size was negatively related to available resources and implementation climate in special education. Investigating the similarities and differences in organizational context across educational settings is needed in future research.

Exploring implementation of a nationwide requirement to increase physical activity in the curriculum in Danish public schools: a mixed methods study

BACKGROUND

In 2014, the Danish Government introduced a wide-ranging school reform that applied to all public schools in Denmark. A distinctive feature of the reform was that it became mandatory to implement an average of 45 min of daily physical activity within the curriculum. Using the RE-AIM framework as an evaluation tool, the objective of the current study was to evaluate the reach, effectiveness, adoption, implementation, and maintenance of mandatory physical activity within the curriculum at ten Danish schools.

METHODS

A complementary mixed-methods approach using accelerometers, questionnaires, and semi-structured interviews was conducted. A total of 10 schools were invited to participate, including 846 students, 76 teachers, and 10 school managers on various levels. Students were invited to wear an accelerometer for seven consecutive days. Teachers were invited to participate in a questionnaire, and school managers were encouraged to take part in a semi-structured interview.

RESULTS

Results showed that, on average, 45.2% of the students were active at least 45 min daily within the curriculum. Teacher and school management interest in physical activity, competencies development, and shared decision-making were identified as central factors for adoption of the requirement. Scheduling physical activity within scheduels and collaborations with external parties were found to influence implementation. Finally, internal coordination, motivated school staff, and school management priority were identified as central factors for maintenance.

CONCLUSIONS

This study provides an evaluation on a nationwide physical activity requirement in Danish public schools. When introducing a wide-ranging nation-wide requirement on physical activity within the curriculum, school managers need to prioritize and support the implementation process. Teachers need to be involved in the decision processes in order to ensure motivation and local ownership. The study also highlights the benefits of an internal coordinator as well as development of a shared strategy among schools, municipalities, and other stakeholders in order to succeed with the implementation.

Preschool teachers' selection of social communication interventions for children with autism: An application of the theory of planned behavior

LAY ABSTRACT

Preschool special educators' are more likely to choose an educational practice to teach a young child with autism a social communication skill if they have positive beliefs about it. We asked preschool special educators to read a description of an autistic student and their social communication goal and imagine they were the student's teacher. We then asked them to pick one of five practices to teach the student. We also asked them questions to understand their attitudes about, confidence in their ability to use, and their perception of their coworkers' support of each practice. There are many research-based practices that a teacher could use to help children learn, and preschool teachers often make these decisions for their students. Teachers' beliefs varied in how supportive they were of each practice, and research shows people are more likely to do something that their beliefs support. In this study, they had more supportive beliefs and were more likely to use some practices, like naturalistic intervention, than other practices, like discrete trial teaching. By knowing this, researchers can help teachers use practices that their beliefs support and help change teachers' beliefs to be supportive of a practice they may need to use.

A Qualitative Examination of a School-Based Implementation of Computer-Assisted Cognitive-Behavioral Therapy for Child Anxiety

Mental health treatment in schools has the potential to improve youth treatment access. However, school-specific barriers can make implementing evidence-based interventions difficult. Task-shifting (i.e., training lay staff to implement interventions) and computer-assisted interventions may mitigate these barriers. This paper reports on a qualitative examination of facilitators and barriers of a school-based implementation of a computer-assisted intervention for anxious youth (Camp Cope-A-Lot; CCAL). Participants (N = 45) included school staff in first through fourth grades. Providers attended a training in CCAL and received weekly, hour-long group consultation calls for three months. In the second year, the sustainability of CCAL use was assessed. Qualitative interviews were conducted after the first year (initial implementation) and second year (sustainability). Interviews were analyzed using the Consolidated Framework for Implementation Research domains to classify themes. Although participants reported that CCAL included useful skills, they expressed concerns about recommended session length (45 minutes) and frequency (weekly). Time burden of consultation calls was also a barrier. School staff facilitated implementation by enabling flexible scheduling for youth to be able to participate in the CCAL program. However, the sustainability of the program was limited due to competing school/time demands. Results suggest that even with computer assisted programs, there is a need to tailor interventions and implementation efforts to account for the time restrictions experienced by school-based service providers. Optimal fit between the intervention and specific school is important to maintain the potential benefits of computer-assisted treatments delivered by lay service providers in schools.

User testing to drive the iterative development of a strategy to improve implementation of evidence-based practices in school mental health

Research has produced a steady stream of evidence-based practices (EBP) that can promote youth behavioral health, but widespread implementation is often poor. To narrow the "science to practice gap," an implementation strategy was developed to enhance school-based mental health providers' intentions to implement EBP. The current study adopted a user-testing approach to inform the iterative development of this implementation strategy, which consisted of strategic education, social influence techniques, and motivational interviewing. Segments of the implementation strategy were demonstrated live for a representative sample of the intended audience. Participants rated each segment on acceptability, appropriateness, and likely impact on intentions to use EBP. Ratings were tallied in real time and presented to participants to spur discussion. Qualitative input was coded via conventional content analyses. Results indicated that that implementation strategies may need to be tailored to the specific EBP. In addition, implementation goal setting was well-received by some participants but not others, suggesting a difficulty of conducting motivational interviewing in group settings. Participants also perceived themselves as familiar with EBP and strong advocates of school mental health services. The paper concludes with a discussion of how this research has influenced the ongoing development of the strategy and implications for EBP implementation efforts.

Individual and organizational characteristics predicting intervention use for children with autism in schools

LAY ABSTRACT

Several interventions or treatment programs have been shown through research to improve social outcomes for children with autism, but they often are not used in schools. This study examined individual (school personnel) and organizational (school level) factors associated with the use of a research-informed social engagement intervention, Remaking Recess, for children with autism in elementary schools. A total of 28 school personnel from 12 schools in 5 districts in the northeastern United States participated. Schools were randomly assigned to (1) training in Remaking Recess only or (2) training in Remaking Recess with implementation support (tailored strategies to remove barriers to increase use of Remaking Recess). School personnel rated their attitudes about research-informed interventions, organizational readiness (school's readiness to use a research-informed intervention), and fidelity or the degree to which an intervention is used as it was originally designed. Observers rated school personnel's fidelity at baseline (pre-intervention training) and exit (post-intervention training). The results suggest that self-rated fidelity was lower when staff perceived the use of Remaking Recess was required; however, observer-rated fidelity was lower when staff rated Remaking Recess as appealing. In addition, self-rated fidelity was higher when there was a sufficient number of staff, positive individual growth, and organizational adaptability. The results also indicated higher observer-rated fidelity when staff perceived positive influence over their coworkers. The results suggest that both individual (attitudes) and organizational (influence, staffing, growth, adaptability) characteristics may affect implementation success in schools. A collaborative decision-making approach for evidence-based practice use is recommended.

Adoption, implementation and sustainability of school-based physical activity and sedentary behaviour interventions in real-world settings: a systematic review

BACKGROUND

Globally, many children fail to meet the World Health Organization's physical activity and sedentary behaviour guidelines. Schools are an ideal setting to intervene, yet despite many interventions in this setting, success when delivered under real-world conditions or at scale is limited. This systematic review aims to i) identify which implementation models are used in school-based physical activity effectiveness, dissemination, and/or implementation trials, and ii) identify factors associated with the adoption, implementation and sustainability of school-based physical activity interventions in real-world settings.

METHODS

The review followed PRISMA guidelines and included a systematic search of seven databases from January 1st, 2000 to July 31st, 2018: MEDLINE, EMBASE, CINAHL, SPORTDiscus, PsycINFO, CENTRAL, and ERIC. A forward citation search of included studies using Google Scholar was performed on the 21st of January 2019 including articles published until the end of 2018. Study inclusion criteria: (i) a primary outcome to increase physical activity and/or decrease sedentary behaviour among school-aged children and/or adolescents; (ii) intervention delivery within school settings, (iii) use of implementation models to plan or interpret study results; and (iv) interventions delivered under real-world conditions.

EXCLUSION CRITERIA

(i) efficacy trials; (ii) studies applying or testing school-based physical activity policies, and; (iii) studies targeting special schools or pre-school and/or kindergarten aged children.

RESULTS

27 papers comprising 17 unique interventions were included. Fourteen implementation models (e.g., RE-AIM, Rogers' Diffusion of Innovations, Precede Proceed model), were applied across 27 papers. Implementation models were mostly used to interpret results (n = 9), for planning evaluation and interpreting results (n = 8), for planning evaluation (n = 6), for intervention design (n = 4), or for a combination of designing the intervention and interpreting results (n = 3). We identified 269 factors related to barriers (n = 93) and facilitators (n = 176) for the adoption (n = 7 studies), implementation (n = 14 studies) and sustainability (n = 7 studies) of interventions.

CONCLUSIONS

Implementation model use was predominately centered on the interpretation of results and analyses, with few examples of use across all study phases as a planning tool and to understand results. This lack of implementation models applied may explain the limited success of interventions when delivered under real-world conditions or at scale.

TRIAL REGISTRATION

PROSPERO (CRD42018099836).

Individual and organizational factors that affect implementation of evidence-based practices for children with autism in public schools: a cross-sectional observational study

BACKGROUND

Children with autism receive most of their intervention services in public schools, but implementation of evidence-based practices (EBPs) for autism varies. Studies suggest that individual (attitudes) and organizational characteristics (implementation leadership and climate) may influence providers' use of EBPs, but research is relatively limited in this area. This study examined individual and organizational factors associated with implementation of three EBPs-discrete trial training, pivotal response training, and visual schedules-for children with autism in special education classrooms in public elementary schools.

METHODS

Participants included 67 autism support teachers and 85 other classroom staff from 52 public elementary schools in the northeastern United States. Participants reported their attitudes toward EBPs (e.g., intuitive appeal, willingness if required, openness, and divergence), implementation leadership and climate of their school, and the frequency with which they deliver each of three EBPs. Linear regression was used to estimate the association of attitudes about EBPs with organizational characteristics and intensity of EBP use. Demographic covariates with a bivariate association with EBP use significant at p < .20 were entered into the adjusted models.

RESULTS

There were significant findings for only one EBP, discrete trial training. Teachers who reported higher perceived divergence (perceived difference of usual practice with academically developed or research-based practices) between EBPs and current practices used less discrete trial training (f2 = .18), and teachers who reported higher appeal (willingness to adopt EBPs given their intuitive appeal) of EBPs used more discrete trial training (f2 = .22). No organizational factors were significantly associated with implementation with any of the three EBPs.

CONCLUSIONS

Attitudes toward EBPs may affect teachers' decisions to use EBPs; however, implementation leadership and climate did not predict EBP use. Future implementation efforts ought to consider the type of EBP and its fit within the context in terms of the EBP's similarities to and differences from existing practices and programs in the setting. Implementation strategies that target individual attitudes about EBPs may be warranted in public schools.

From evidence to impact: Joining our best school mental health practices with our best implementation strategies

There is substantial research evidence for the effectiveness of school mental health strategies across problem areas, developmental levels, and the prevention-intervention spectrum. At the same time, it is clear that the education and mental health fields continue to struggle to apply this evidence at any level of scale. This commentary reflects on ways in which education-specific applications of implementation science principles - and explicit consideration of determinants of implementation success - may guide more consistent use of evidence in school mental health. After reviewing implementation determinants and strategies across multiple levels of effect (i.e., the outer setting, inner setting, individual, and intervention levels), the commentary goes on to recommend specific areas of needed attention in school mental health implementation efforts and research. These include a need to adapt interventions to better fit the context of schools, streamlining school mental health programs and practices to make them more implementable, and recognizing the critical role of assessment and selection of evidence-based interventions by school leaders. The commentary concludes by reflecting on the substantial opportunity provided by the education sector to both apply and advance implementation science.