Illustrating the multiple facets and levels of fidelity of implementation to a teacher classroom management intervention

Global health reciprocal innovation to address mental health and well-being: strategies used and lessons learnt

Over the past two decades there have been major advances in the development of interventions promoting mental health and well-being in low- and middle-income countries (LMIC), including delivery of care by non-specialist providers, incorporation of mobile technologies and development of multilevel community-based interventions. Growing inequities in mental health have led to calls to adopt similar strategies in high-income countries (HIC), learning from LMIC. To overcome shared challenges, it is crucial for projects implementing these strategies in different global settings to learn from one another. Our objective was to examine cases in which mental health and well-being interventions originating in or conceived for LMIC were implemented in the USA. The cases included delivery of psychological interventions by non-specialists, HIV-related stigma reduction programmes, substance use mitigation strategies and interventions to promote parenting skills and family functioning. We summarise commonly used strategies, barriers, benefits and lessons learnt for the transfer of these innovative practices among LMIC and HIC. Common strategies included intervention delivery by non-specialists and use of digital modalities to facilitate training and increase reach. Common barriers included lack of reimbursement mechanisms for care delivered by non-specialists and resistance from professional societies. Despite US investigators' involvement in most of the original research in LMIC, only a few cases directly involved LMIC researchers in US implementation. In order to achieve greater equity in global mental health and well-being, more efforts and targeted funding are needed to develop best practices for global health reciprocal innovation and iterative learning in HIC and LMIC.

The role of teachers' commitment to implement in delivering evidence-based social-emotional learning programs

Although there are reasons to believe that teachers' commitment to learn and enact an evidence-based program (i.e., their commitment to implement) predicts their implementation fidelity, there is surprisingly little quantitative research testing this relationship. Using a national large-scale evaluation of three preschool social-emotional interventions, this study investigated how strongly teachers' commitment predicted implementation fidelity and whether commitment was a meaningful predictor of fidelity as compared to other individual factors (i.e., teacher stress at baseline) and contextual factors (i.e., collegial supports, classroom behavioral problems, and classroom quality at baseline). We surveyed 230 preschool teachers in their first year of implementing the interventions; data sources include surveys from teachers and 52 intervention coaches as well as classroom observational data. We found that teachers' baseline commitment consistently predicted implementation fidelity across time and that commitment predicted unique variation in fidelity after accounting for other individual and contextual factors. In addition, implementation fidelity had a moderate positive relationship with teachers' baseline classroom quality and a small negative association with baseline classroom behavior problems. Findings are discussed with respect to implementation science in education.

Leveraging Motivational Interviewing to Coach Teachers in the Implementation of Preventive Evidence-Based Practices: A Sequential Analysis of the Motivational Interviewing Process

Though emerging research supports the effectiveness of school-based coaching models utilizing motivational interviewing (MI), an examination of the specific drivers behind these effects is notably lacking in the prevention field. This study leveraged sequential analysis to examine how teachers' verbalization of change talk (i.e., language in support of change) and sustain talk (i.e., language in support of maintaining the status quo) was influenced by coaches' use of MI-consistent (i.e., collaborative language supportive of change) and MI-inconsistent (e.g., confrontational, directive) language, respectively. We also examined whether teacher and coach factors were related to coach-teacher language dynamics. Data were collected from 87 teachers in 16 elementary and middle schools randomized in a trial to the Double Check preventive intervention (see Bradshaw et al., 2018). Audio-recorded coaching feedback sessions were coded using an adapted version of the Motivational Interviewing Sequential Code for Observing Process Exchanges (MI-SCOPE). Sequential analyses indicated that MI-consistent and change talk were significantly more likely than chance to occur consecutively. Teachers' sustain talk was also more likely to occur sequentially with coach use of MI-consistent language and teacher change talk; the latter suggests teacher ambivalence. Coaches rarely used MI-inconsistent language, and its occurrence was only associated with more MI-inconsistent language. Regression analyses indicated that teacher age, efficacy, burnout, classroom organization, and some design features (i.e., cohort, coach, coach-teacher racial match) were associated with different coach-teacher language dynamics. This novel school-based study illustrates how coaching MI evoked teacher change talk related to use of evidence-based programs.

The Incredible Years Teacher Classroom Management Program: Outcomes from a Group Randomized Trial

This group randomized controlled trial (RCT) evaluated the efficacy of the Incredible Years Teacher Classroom Management Program (IY TCM) on student social behavioral and academic outcomes among a large diverse sample of students within an urban context. Participants included 105 teachers and 1817 students in kindergarten to third grade. Three-level hierarchical linear models (HLM) were conducted to examine the overall treatment effects on teacher-reported student behavior and academic outcomes. In addition, multi-level moderation analyses were conducted to examine whether the treatment effects on student outcomes differed by demographic variables and pretest measures of social emotional and disruptive behavior and academics. Findings indicate that IY TCM reduced student emotional dysregulation (d = - 0.14) and increased prosocial behavior (d = 0.13) and social competence (d = 0.13). In addition, students initially lower on measures of social and academic competence demonstrated significant improvements on the same measure at outcome in comparison to similar peers in control classrooms. Practical significance of the findings and implications for schools and policy makers are discussed.

Promoting Teachers' Implementation of Classroom-Based Prevention Programming Through Coaching: The Mediating Role of the Coach-Teacher Relationship

There is growing awareness of the importance of implementation fidelity and the supports, such as coaching, to optimize it. This study examined how coaching activities (i.e., check-ins, needs assessment, modeling, and technical assistance) related directly and indirectly to implementation dosage and quality of the PAX Good Behavior Game, via a mediating pathway through working relationship. Mediation analyses of 138 teachers revealed direct effects of modeling and working relationship on implementation dosage, whereas needs assessment was associated with greater dosage indirectly, by higher ratings of the working relationship. Understanding how coaching activities promote implementation fidelity elements has implications for improving program effectiveness.

Training teachers in classroom management to improve mental health in primary school children: the STARS cluster RCT

Background

Poor mental health in childhood is common, persistent and associated with a range of adverse outcomes that include persistent psychopathology, as well as risk-taking behaviour, criminality and educational failure, all of which may also compromise health. There is a growing policy focus on children’s mental health and the role of schools in particular in addressing this.

Objectives

To evaluate whether or not the Incredible Years®(IY) Teacher Classroom Management (TCM) training improved children’s mental health, behaviour, educational attainment and enjoyment of school, improved teachers’ mental health and relationship with work, and was cost-effective in relation to potential improvements.

Design

A two-arm, pragmatic, parallel-group, superiority, cluster randomised controlled trial.

Setting

A total of 80 UK schools (clusters) were recruited in three distinct cohorts between 2012 and 2014 and randomised to TCM (intervention) or teaching as usual [(TAU) control] with follow-ups at 9, 18 and 30 months. Schools and teachers were not masked to allocation.

Participants

Eighty schools (n = 2075 children) were randomised: 40 (n = 1037 children) to TCM and 40 (n = 1038 children) to TAU.

Interventions

TCM was delivered to teachers in six whole-day sessions, spread over 6 months. The explicit goals of TCM are to enhance classroom management skills and improve teacher–student relationships.

Main outcome measures

The primary planned outcome was the teacher-reported Strengths and Difficulties Questionnaire Total Difficulties (SDQ-TD) score. Random-effects linear regression and marginal logistic regression models using generalized estimating equations were used to analyse outcomes.

Results

The intervention reduced the SDQ-TD score at 9 months [adjusted mean difference (AMD) –1.0, 95% confidence interval (CI) –1.9 to –0.1;p = 0.03] but there was little evidence of effects at 18 months (AMD –0.1, 95% CI –1.5 to 1.2;p = 0.85) and 30 months (AMD –0.7, 95% CI –1.9 to 0.4;p = 0.23). Planned subgroup analyses suggested that TCM is more effective than TAU for children with poor mental health. Cost-effectiveness analysis using the SDQ-TD suggested that the probability of TCM being cost-effective compared with TAU was associated with some uncertainty (range of 40% to 80% depending on the willingness to pay for a unit improvement in SDQ-TD score). In terms of quality-adjusted life-years (QALYs), there was evidence to suggest that TCM was cost-effective compared with TAU at the National Institute for Health and Care Excellence thresholds of £20,000–30,000 per QALY at 9- and 18-month follow-up, but not at 30-month follow-up. There was evidence of reduced disruptive behaviour (p = 0.04) and reductions in inattention and overactivity (p = 0.02) at the 30-month follow-up. Despite no main effect on educational attainment, subgroup analysis indicated that the intervention’s effect differed between those who did and those who did not have poor mental health for both literacy (interactionp = 0.04) and numeracy (interactionp = 0.03). Independent blind observations and qualitative feedback from teachers suggested that teachers’ behaviour in the classroom changed as a result of attending TCM training.

Limitations

Teachers were not masked to allocation and attrition was marked for parent-reported data.

Conclusions

Our findings provide tentative evidence that TCM may be an effective universal child mental health intervention in the short term, particularly for primary school children who are identified as struggling, and it may be a cost-effective intervention in the short term.

Future work

Further research should explore TCM as a whole-school approach by training all school staff and should evaluate the impact of TCM on academic progress in a more thorough and systematic manner.

Trial registration

Current Controlled Trials ISRCTN84130388.

Funding

This project was funded by the National Institute for Health Research (NIHR) Public Health Research programme and will be published in full inPublic Health Research; Vol. 7, No. 6. See the NIHR Journals Library website for further project information. Funding was also provided by the NIHR Collaboration for Leadership in Applied Health Research and Care South West Peninsula (NIHR CLAHRC South West Peninsula).

Implementation fidelity and adolescent smoking: The X:IT study-A school randomized smoking prevention trial

BACKGROUND

Assessing the actual implementation of multi-component interventions can provide important knowledge for future interventions. Intervention components may be implemented differently, knowledge about this can provide an understanding of which components are essential and therefore must be included. The aim of this study was to examine the implementation of one, two, or all three main intervention components at the individual level and to assess the association to current smoking among 13 year-olds in the X:IT study.

METHODS

Data stems from a cluster-randomized controlled trial in 94 Danish elementary schools (51 intervention; 43 control schools). Implementation was measured by aspects of adherence, dose, quality of delivery, and participant responsiveness based on questionnaire data from 4161 pupils at baseline (mean-age: 12.5 years) and 3764 pupils at first follow-up eight months later. Coordinator responses from 49 intervention schools were also included. Associations between individual level implementation of the three main components and pupil smoking were examined through a 3-level logistic regression model.

RESULTS

Although implementation fidelity for the three main intervention components was good, only one third (38.8%) of pupils in intervention schools were exposed to full implementation of the intervention. Among these pupils odds ratio for smoking was 0.25 (95% CI: 0.15 - 0.42).

CONCLUSIONS

School-based programs can be very effective if carefully implemented. Future school-based smoking preventive initiatives should include multiple components, and seek to enhance implementation quality of all components.

An empirical study of multidimensional fidelity of COMPASS consultation

Consultation is essential to the daily practice of school psychologists (National Association of School Psychologist, 2010). Successful consultation requires fidelity at both the consultant (implementation) and consultee (intervention) levels. We applied a multidimensional, multilevel conception of fidelity (Dunst, Trivette, & Raab, 2013) to a consultative intervention called the Collaborative Model for Promoting Competence and Success (COMPASS) for students with autism. The study provided 3 main findings. First, multidimensional, multilevel fidelity is a stable construct and increases over time with consultation support. Second, mediation analyses revealed that implementation-level fidelity components had distant, indirect effects on student Individualized Education Program (IEP) outcomes. Third, 3 fidelity components correlated with IEP outcomes: teacher coaching responsiveness at the implementation level, and teacher quality of delivery and student responsiveness at the intervention levels. Implications and future directions are discussed. (PsycINFO Database Record

Transportability of an Evidence-Based Early Childhood Intervention in a Low-Income African Country: Results of a Cluster Randomized Controlled Study

Children in Sub-Saharan Africa (SSA) are burdened by significant unmet mental health needs. Despite the successes of numerous school-based interventions for promoting child mental health, most evidence-based interventions (EBIs) are not available in SSA. This study investigated the implementation quality and effectiveness of one component of an EBI from a developed country (USA) in a SSA country (Uganda). The EBI component, Professional Development, was provided by trained Ugandan mental health professionals to Ugandan primary school teachers. It included large-group experiential training and small-group coaching to introduce and support a range of evidence-based practices (EBPs) to create nurturing and predictable classroom experiences. The study was guided by the Consolidated Framework for Implementation Research, the Teacher Training Implementation Model, and the RE-AIM evaluation framework. Effectiveness outcomes were studied using a cluster randomized design, in which 10 schools were randomized to intervention and wait-list control conditions. A total of 79 early childhood teachers participated. Teacher knowledge and the use of EBPs were assessed at baseline and immediately post-intervention (4-5 months later). A sample of 154 parents was randomly selected to report on child behavior at baseline and post-intervention. Linear mixed effect modeling was applied to examine effectiveness outcomes. Findings support the feasibility of training Ugandan mental health professionals to provide Professional Development for Ugandan teachers. Professional Development was delivered with high levels of fidelity and resulted in improved teacher EBP knowledge and the use of EBPs in the classroom, and child social competence.

Association of School Characteristics and Implementation in the X:IT Study-A School-Randomized Smoking Prevention Program

BACKGROUND

Assessment of implementation is essential for the evaluation of school-based preventive activities. Interventions are more easily implemented in schools if detailed instructional manuals, lesson plans, and materials are provided; however, implementation may also be affected by other factors than the intervention itself-for example, school-level characteristics, such as principal support and organizational capacity. We examined school-level characteristics of schools in groups of high, medium, and low implementation of a smoking prevention intervention.

METHODS

The X:IT study is a school-randomized trial testing a multicomponent intervention to prevent smoking among adolescents. Our data came from electronic questionnaires completed by school coordinators at 96.1% of participating intervention schools (N = 49) at first follow -up.

RESULTS

Schools that implemented the X:IT intervention to a medium or high degree had higher levels of administrative leadership (77.3% and 83.3% vs 42.9%), school climate/organizational health (95.5% and 91.7% vs 66.7%), mission-policy alignment (90.9% and 100.0% vs 71.4%), personnel expertise (81.8% and 75.0% vs 46.7%), school culture (77.3% and 91.7% vs 53.3%), positive classroom climate (91.4% and 96.2% vs 82.9%) compared with low implementation schools.

CONCLUSION

Our findings highlight the importance of considering the school context in future health prevention initiatives.

High impact of implementation on school-based smoking prevention: the X:IT study-a cluster-randomized smoking prevention trial

BACKGROUND

Implementation fidelity describes how well an intervention is implemented in the real-world setting. Assessing implementation fidelity is essential in the understanding of intervention results. In most studies, implementation fidelity is measured insufficiently, though, not taking into account the complexity of the concept nor the intervention. The objective of the present study was to develop an overall quantitative measure of implementation fidelity, to examine the degree of implementation fidelity and the association of implementation and effect of a randomized school-based smoking prevention trial-the X:IT study.

METHODS

A cluster-randomized trial testing is a multi-component intervention to prevent smoking among adolescents in 94 Danish elementary schools (51 intervention, 43 control schools). Participants were grade 7 pupils (mean age 12.5 years). Data was collected by electronic questionnaires among pupils at baseline (n = 4161), the first follow-up (n = 3764), and the second follow-up (n = 3269) and among school coordinators at intervention schools at the first and second follow-up (50 and 39 coordinators).

INTERVENTION

The intervention included three components: (1) smoke-free school grounds, (2) smoke-free curriculum, and (3) parental involvement, contracts, and dialogues. Implementation fidelity was assessed by four domains: adherence, dose, quality of delivery, and participant responsiveness. These were combined into an overall school-wise implementation index. The association of implementation and smoking was examined by logistic regression analyses.

RESULTS

One fourth of the schools was characterized as high implementers of the program (all three components) at both first (12 schools, 24.0 %) and second follow-up (11 schools, 28.2 %). Implementation fidelity was strongly associated with smoking at the first and second follow-up, e.g., the odds for smoking at schools with high implementation both years were OR = 0.44 (95 % CI 0.32 to 0.68).

CONCLUSIONS

Using an overall measure based on several aspects of implementation fidelity, we showed a negative graded association between implementation and smoking. This study suggests that higher degrees of implementation will improve the effect of the X:IT intervention. Studying the association between implementation and effect is extremely important; only by doing so, we can distinguish the quality of the intervention from the success of the implementation.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN77415416.

Understanding and Measuring Coach-Teacher Alliance: A Glimpse Inside the 'Black Box'

Coaching models are increasingly used in schools to enhance fidelity and effectiveness of evidence-based interventions; yet, little is known about the relationship between the coach and teacher (i.e., coach-teacher alliance), which may indirectly enhance teacher and student outcomes through improved implementation quality. There is also limited research on measures of coach-teacher alliance, further hindering the field from understanding the active components for successful coaching. The current study examined the factor structure and psychometric characteristics of a measure of coach-teacher alliance as reported by both teachers and coaches and explored the extent to which teachers and coaches reliably rate their alliance. Data come from a sample of 147 teachers who received implementation support from one of four coaches; both the teacher and the coach completed an alliance questionnaire. Separate confirmatory factor analyses for each informant revealed four factors (relationship, process, investment, and perceived benefits) as well as an additional coach-rated factor (perceived teacher barriers). A series of analyses, including cross-rater correlations, intraclass correlation coefficients, and Kuder-Richardson reliability estimates suggested that teachers and coaches provide reliable, though not redundant, information about the alliance. Implications for future research and the utilization of the parallel coach-teacher alliance measures to increase the effectiveness of coaching are discussed.

Within-year fidelity growth of SWPBIS during installation and initial implementation

The purpose of this study was to document within-year fidelity growth during installation and initial implementation of School-Wide Positive Behavioral Interventions and Supports (SWPBIS). Participants included school teams from schools throughout the United States that were in years 1 to 4 of SWPBIS implementation and routinely evaluated their implementation fidelity. The fidelity outcome was assessed with the Team Implementation Checklist (TIC) and was completed multiple times per year by SWPBIS teams. Results from multilevel fidelity growth models documented within- and between-school variability and growth predictors. Years implementing, location, school type, and enrollment size were significant predictors of beginning-year fidelity scores (intercept), and years implementing and relative socioeconomic status were significant predictors of the average rate of fidelity change per month of school (slope). These results add to the growing field of implementation science, and practice recommendations are outlined. (PsycINFO Database Record

Implementing an early childhood school-based mental health promotion intervention in low-resource Ugandan schools: study protocol for a cluster randomized controlled trial

Background

Children in Sub-Saharan Africa (SSA) are burdened by significant unmet mental health needs, but this region has limited access to mental health workers and resources to address these needs. Despite the successes of numerous school-based interventions for promoting child mental health, most evidence-based interventions are not available in SSA. This study will investigate the transportability of an evidence-based program from a developed country (United States) to a SSA country (Uganda). The approach includes task-shifting to early childhood teachers and consists of professional development (five days) to introduce strategies for effective behavior management and positive teacher-student interactions, and group-based consultation (14 sessions) to support adoption of effective practices and tailoring to meet the needs of individual students.

Methods/Design

The design of this study is guided by two implementation frameworks, the Consolidated Framework for Implementation Research and the Teacher Training Implementation Model, that consider multidimensional aspects of intervention fidelity and contextual predictors that may influence implementation and teacher outcomes. Using a cluster randomized design, 10 schools in Uganda will be randomized to either the intervention group (five schools) or the waitlist control group (five schools). A total of 80 to 100 early childhood teachers will be enrolled in the study. Teacher utilization of evidence-based strategies and practices will be assessed at baseline, immediate post-intervention (six months after baseline), and at seven months post-intervention (during a new academic year). Fidelity measures will be assessed throughout the program implementation period (during professional development and consultation sessions). Individual teacher and contextual factors will be assessed at baseline. Data will be collected from multiple sources. Linear mixed-effect modeling, adjusting for school nesting, will be applied to address study questions.

Discussion

The study will produce important information regarding the value of an evidence-based early intervention, and a theory-guided implementation process and tools designed for use in implementing early childhood evidence-based programs in SSA countries or resource-constrained community settings.

Trial registration

This trial was registered with ClinicalTrials.gov (registration number: NCT097115) on 15 May 2013.

Consultation as an implementation strategy for evidence-based practices across multiple contexts: unpacking the black box

There is great interest in the dissemination and implementation of evidence-based treatments and practices for children across schools and community mental health settings. A growing body of literature suggests that the use of one-time workshops as a training tool is ineffective in influencing therapist behavior and patient outcomes and that ongoing expert consultation and coaching is critical to actual uptake and quality implementation. Yet, we have very limited understanding of how expert consultation fits into the larger implementation support system, or the most effective consultation strategies. This commentary reviews the literature on consultation in child mental health, and proposes a set of core consultation functions, processes, and outcomes that should be further studied in the implementation of evidence-based practices for children.