Measuring Fidelity and Adaptation: Reliability of a Instrument for School-Based Prevention Programs

Fidelity of D.A.R.E. Officers' Delivery of "keepin' it REAL" in Elementary & Middle School

The goal of the current study is to examine the degree to which measures of quality of implementation and student engagement moderate pretest-posttest changes in mediating variables that are targeted by DARE "keepin' it REAL." DARE officers (10 elementary school, five middle school) taught DARE "keepin' it REAL lessons to 1,017 elementary students (480 boys and 534 girls) and 435 middle school students (217 boys and 215 girls). We examined teachers' and students' ratings of elementary and middle schools in response to DARE officers' delivery of the program. HLM analyses revealed that students' engagement was a significant and meaningful predictor of changes in targeted mediators. Teachers' ratings of student responsiveness added little in terms of understanding these outcomes with main effects observed only for students' ability to respond to bulling and students' estimates of peer drug use. Teachers' ratings of the quality of officer implementation, on the other hand, did add to understanding students' outcomes. Effects were seen for three (peer norms about drug use, decision-making (DM) skills, intentions to avoid drug use) out of six outcome variables and suggest a stronger positive effect for elementary versus middle school students. At least for these three outcomes, understanding quality of implementation added to our ability to interpret results. Specifically, in addition to students' engagement, quality of implementation (which varied by grade) contributed to achieving positive changes in students' outcomes.

Fidelity Assessment of the Treatnet family (TF): A family-based intervention for adolescents with substance use disorders and their families

•

There was a high level of agreement between practitioners’ and supervisors’ ratings on the practitioner’s use of family-based therapeutic skills.

•

In working with adolescents who are affected by substance use disorders, the practitioners showed very good adherence to the Treatnet Family approach.

•

There was a good level of agreement between the adolescents and their family member in terms of how interactive they found the Treatnet Family session.

Introduction

In transporting family-based interventions to community settings, establishing and maintaining fidelity to intervention is important. This exploratory study was implemented in the framework of a United Nations Office on Drugs and Crime (UNODC) global programme on Drug Dependence Treatment and Care. It is the first to examine an evidence-informed family-based intervention (“Treatnet Family”; TF) adherence for the treatment of adolescents with substance use disorders (SUD) among practitioners in community settings in Jakarta, Indonesia.

Method

Twenty-three practitioners at five community-based counselling centres were trained in Treatnet Family and delivered it to 19 adolescents with SUD and their family members over a 6-week period. One of the five local Treatnet Family-trained supervisors randomly selected one session of the family-based intervention (TF) and observed the extent to which the practitioner’s adhered to the TF manual.

Results

According to the supervisors’ observation, all the practitioners used the Treatnet Family core skills such as positive reframing, positive relational reframing, perspective taking, relational questions, and going with resistance. There was a high level of agreement between practitioners’ and supervisors’ rating on the practitioners’ use of specific therapeutic skills as measured using the Inventory of Therapy Techniques (ITT).

Conclusion

Results suggest that Treatnet Family can be delivered with adherence by practitioners in community-based settings.

Iterative Decision-making for Evaluation of Adaptations (IDEA): A decision tree for balancing adaptation, fidelity, and intervention impact

BACKGROUND

Evidence-based practices (EBPs) are frequently adapted to maximize outcomes while maintaining fidelity to core EBP elements. Many step-by-step frameworks for adapting EBPs have been developed, but these models may not account for common complexities in the adaptation process. In this paper, we introduce the Iterative Decision-making for Evaluation of Adaptations (IDEA), a tool to guide adaptations that addresses these issues.

FRAMEWORK DESIGN AND USE

Adapting EBPs requires attending to key contingencies incorporated into the IDEA, including: the need for adaptations; fidelity to core EBP elements; the timeframe in which to make adaptations; the potential to collect pilot data; key clinical and implementation outcomes; and stakeholder viewpoints. We use two examples to illustrate application of the IDEA.

CONCLUSIONS

The IDEA is a practical tool to guide EBP adaptation that incorporates important decision points and the dynamism of ongoing adaptation. Its use may help implementation scientists, clinicians, and administrators maximize EBP impact.

The Unified Protocol for Transdiagnostic Treatment of Emotional Disorders in Adolescents (UP-A) Adapted as a School-Based Anxiety and Depression Prevention Program: An Initial Cluster Randomized Wait-List-Controlled Trial

Anxiety and depression are common debilitating conditions that show high comorbidity rates in adolescence. The Unified Protocol for Transdiagnostic Treatment of Emotional Disorders in Adolescents (UP-A; Ehrenreich-May et al., 2018) is one of the few existing resources aimed at applying transdiagnostic treatment principles across the core dysfunctions implicated in the development of both anxiety and depression using a single protocol. This is the first known controlled study to examine the efficacy of the UP-A adapted as a nine-session universal preventive intervention program delivered in a school setting. A total of 151 students (mean age: 15.05) participated in this randomized wait-list-controlled trial conducted in Madrid, Spain. An unexpected decline in anxiety and depression levels from pre- to posttreatment and follow-up was found in both groups (p = .009, d = -0.22), and overall differences between conditions did not reach significance. Exploratory analyses of baseline emotional symptom severity as a potential predictor trended toward a significantly greater decrease in symptoms of depression for those with greater baseline emotional symptoms in the UP-A group compared to the wait-list-control group. Future trials with larger samples are justified to estimate the effect of the UP-A adapted as a selective prevention program for anxiety and depression.

Self-Reported Engagement in a Drug Prevention Program: Individual and Classroom Effects on Proximal and Behavioral Outcomes

Numerous studies emphasize the role of student engagement in academic learning and performance. Less known is whether engagement plays a role in drug prevention program outcomes. We examined a self-report measure of engagement as part of the All Stars Core drug prevention program evaluation, assessing its impact on target risk mechanisms and behavioral outcomes. Students completed pretests just prior to and posttests just after completing the intervention. Surveys assessed demographics, proximal intervening measures (i.e., commitments to avoid substance use and antisocial behavior, perceived lifestyle incongruence with substance use and antisocial behavior, normative beliefs about substance use and antisocial behavior, and parental attentiveness), and distal outcome measures of alcohol, cigarette use, and antisocial behaviors. A brief 6-item posttest measure including items tapping the students' perspective on the quality of teaching the program material and their level of engagement with the program was internally consistent (α = .79). Multi-level analyses positing engagement effects at both the classroom- and individual-level indicated that classroom average engagement was significantly associated with all the targeted risk mechanisms, and outcomes of antisocial behavior and alcohol use, controlling for pretest measures and classroom size. Individual student engagement relative to classroom peers was significantly associated with all posttest target risk mechanisms and behavioral outcomes. The current findings suggest that students should routinely provide assessments of engagement and perceived quality of teaching, which would improve our understanding of how prevention programs work. Teachers can improve engagement by paying attention to students when they speak in class, making the program enjoyable to participants, encouraging students to share opinions, stimulating attentiveness, being well prepared to deliver the intervention, and helping students think broadly about implications of drug prevention as it affects their lives. This type of support will ultimately engage students in ways that will enhance the likelihood that these programs will have their desired effects.

Cluster randomised controlled trial of an online intervention to prevent ecstasy and new psychoactive substance use among adolescents: final results and implications for implementation

OBJECTIVES

To evaluate the effectiveness of the online Climate Schools: Ecstasy and Emerging Drugs module over 2 years, and examine the impact of intervention dose on outcomes.

DESIGN

Cluster randomised controlled trial.

SETTING

Secondary schools in Australia.

PARTICIPANTS

1126 students (aged 14.9 years) from 11 schools.

INTERVENTION

Five schools were randomly allocated to the four-lesson internet-based Climate Schools: Ecstasy and Emerging Drugs module. This universal intervention uses cartoon storylines to deliver harm-minimisation information about ecstasy and new psychoactive substances (NPS). It was delivered during health education classes over 4 weeks. Six schools were randomised to the control group (health education as usual). Participants were not blinded to intervention allocation.

OUTCOMES MEASURES

Students completed self-report surveys at baseline, post-test, 6, 12 and 24 months post-baseline. Intentions to use ecstasy and NPS (including synthetic cannabis and synthetic stimulants), knowledge about ecstasy and NPS and lifetime use of ecstasy and NPS were assessed. This paper reports the results at 24 months post-baseline.

ANALYSIS

Mixed effects regressions were conducted to analyse intervention effects from baseline to 24 months. Post hoc analyses using Inverse Probability of Treatment Weighting compared controls with students who: i) completed all four lessons ('full dose') and ii) partially completed the intervention (≤three lessons, 'incomplete dose').

RESULTS

Primary analyses found that controls were significantly more likely to intend on using synthetic cannabis compared with intervention group students (OR=3.56, p=0.01). Results from the weighted analyses indicated that controls reported significantly lower knowledge about ecstasy (p=0.001) and NPS (p=0.04) compared with the full-dose group. No significant differences were observed between the incomplete dose and control groups.

CONCLUSIONS

The online intervention was effective in modifying students' intentions to use synthetic cannabis up to 24 months; however, this study highlights the importance of delivering prevention programmes in full to maximise student outcomes.

TRIAL REGISTRATION NUMBER

ACTRN12613000708752.

The effectiveness of Rock and Water in improving students' socio-emotional adjustment and social safety: study protocol for a randomized controlled trial

Background

Students following a low education track have an increased risk for developing problem behaviors. Rock and Water is a widespread, but still poorly evaluated, intervention that aims to improve students’ socio-emotional adjustment and social safety. The aims of this study are to evaluate (1) the effectiveness of Rock and Water on socio-emotional adjustment (i.e., psychosocial wellbeing, sexual autonomy, and resilience) and social safety (i.e., perceived social security in the classroom, aggression, and bullying) and to examine (2) moderators and (3) mediators of its effects.

Methods

Schools are randomly assigned into four conditions: ‘Light’ (a core team of teachers is trained), ‘Standard’ (a core team of teachers and the whole school team is trained), ‘Plus’ (a core team of teachers, the whole school team is trained, and parents are involved), or ‘Control condition’ (Care As Usual). We aim to include 180 7th Grade students in each condition (N = 720) across all waves. A multi-informant (i.e., students, parents, and teachers) approach is used to assess the outcomes (socio-emotional adjustment and social safety), moderators (student, trainer, and parent characteristics) and mediators (self-control, self-reflection, self-esteem, and emotion regulation). Video-observations will be analyzed in a subsample to study the possible mediating effect of changes in deviant and prosocial communication among students on the effect on social safety.

Discussion

This project will provide information on the effectiveness of (different levels of school and parental involvement in) Rock and Water, which can be used by schools to decide upon the most efficient way to improve the care for the students. We will be able to shed more light on what works for whom and the working mechanisms of Rock and Water.

Trial registration

Dutch Trial Registration number 6554, registered on the 3rd of July 2017. The design of this study was approved by the Ethical Committee of the Faculty of Social and Behavioral Sciences of Utrecht University (FETC17–015). This study is financially supported by a grant from The Netherlands Organization for Health Research and Development, grant number 531001106.

Electronic supplementary material

The online version of this article (10.1186/s40359-018-0247-y) contains supplementary material, which is available to authorized users.

Interaction of Theory and Practice to Assess External Validity

BACKGROUND

Variations in local context bedevil the assessment of external validity: the ability to generalize about effects of treatments. For evaluation, the challenges of assessing external validity are intimately tied to the translation and spread of evidence-based interventions. This makes external validity a question for decision makers, who need to determine whether to endorse, fund, or adopt interventions that were found to be effective and how to ensure high quality once they spread.

OBJECTIVE

To present the rationale for using theory to assess external validity and the value of more systematic interaction of theory and practice.

METHODS

We review advances in external validity, program theory, practitioner expertise, and local adaptation. Examples are provided for program theory, its adaptation to diverse contexts, and generalizing to contexts that have not yet been studied. The often critical role of practitioner experience is illustrated in these examples. Work is described that the Robert Wood Johnson Foundation is supporting to study treatment variation and context more systematically.

RESULTS

Researchers and developers generally see a limited range of contexts in which the intervention is implemented. Individual practitioners see a different and often a wider range of contexts, albeit not a systematic sample. Organized and taken together, however, practitioner experiences can inform external validity by challenging the developers and researchers to consider a wider range of contexts. Researchers have developed a variety of ways to adapt interventions in light of such challenges.

CONCLUSIONS

In systematic programs of inquiry, as opposed to individual studies, the problems of context can be better addressed. Evaluators have advocated an interaction of theory and practice for many years, but the process can be made more systematic and useful. Systematic interaction can set priorities for assessment of external validity by examining the prevalence and importance of context features and treatment variations. Practitioner interaction with researchers and developers can assist in sharpening program theory, reducing uncertainty about treatment variations that are consistent or inconsistent with the theory, inductively ruling out the ones that are harmful or irrelevant, and helping set priorities for more rigorous study of context and treatment variation.

Évaluation de la qualité de l’implantation du programme Pare-Chocs à l’école secondaire

Malgré l’importante augmentation de la prévalence de la dépression durant l’adolescence, seulement une faible proportion des adolescents qui présentent des symptômes dépressifs élevés bénéficie des services de santé mentale en milieu scolaire. De plus, les programmes offerts à ces élèves souffrent souvent d’une implantation lacunaire, limitant ainsi leur efficacité potentielle. L’identification des facteurs qui influencent la fidélité de l’implantation est donc essentielle pour améliorer la qualité des services offerts. Guidés par le modèle d’évaluation basé sur la théorie des programmes (theory-driven evaluation), nous évaluons la qualité de l’implantation du programme Pare-Chocs et les facteurs qui l’ont influencée à l’aide des composantes du modèle d’action de Chen (2005). Une approche multi-méthodes a été utilisée pour obtenir des données quantitatives sur la fidélité d’implantation dans les six groupes et des données qualitatives sur les composantes du modèle d’action. Les résultats suggèrent que l’adhésion, le dosage et la participation étaient élevés. Les contraintes de temps et le peu de formation antérieure en lien avec la théorie du programme ont limité la fidélité de l’implantation tandis que la formation, la supervision et le manuel y ont contribué. Ces résultats confirment que les programmes de prévention disséminés dans les milieux scolaires peuvent être implantés avec un niveau de fidélité élevé, même si certains défis doivent être pris en considération lors de la planification de l’intervention pour contribuer à des effets plus importants du programme. De plus, la fidélité devrait être évaluée de manière systématique dans ce contexte.

Generalizing about Public Health Interventions: A Mixed-Methods Approach to External Validity

Public health researchers and practitioners are calling for greater focus on external validity, the ability to generalize findings of evidence-based interventions (EBIs) beyond the limited number of studies testing effectiveness. For public health, the goal is applicability: to translate, disseminate, and implement EBIs for an impact on population health. This article is a review of methods and how they might be combined to better assess external validity. The methods include (a) better description of EBIs and their contexts; (b) combining of statistical tools and logic to draw inferences about study samples; (c) sharper definition of the theory behind the intervention and core intervention components; and (d) more systematic consultation of practitioners. For population impact, studies should focus on context features that are likely to be both important (based on program theory) and frequently encountered by practitioners. Mixed-method programs of research will allow public health to expand causal generalizations.

London Education and Inclusion Project (LEIP): Results from a Cluster-Randomized Controlled Trial of an Intervention to Reduce School Exclusion and Antisocial Behavior

School exclusion as a disciplinary measure remains a controversial issue. In spite of numerous attempts to reduce this practice, no solutions with documented effectiveness exist. This article reports results of a cluster-randomized controlled field trial carried out in 36 schools across London. The trial is an independent evaluation of a 12-week-long intervention, Engage in Education-London (EiE-L), delivered by Catch22. The intervention was aimed at students in secondary school who are most at risk of school exclusion. It targeted their social communication and broader social skills with the aim of reducing school exclusions and problem behaviors. The study employed a multi-informant design that included students and teacher reports as well as official records for exclusions and arrests. Data were analyzed through intent-to-treat analyses based on self-reports from 644 students and 685 teacher reports for students who were nominated for the study and for whom data was available at baseline or post-intervention. At baseline data collection the students ranged in age from 12.85 to 15.03, with M = 14.03; 71 % were male and included a number of ethnic minorities, the largest of which was black African/black Caribbean comprising 40 % of the sample. The results suggested a small but statistically significant negative effect on the primary outcome of exclusion and null effects for the secondary outcomes that measured behavioral and socio-emotional outcomes. The study's findings are discussed in terms of the possible reasons for the null effects and negative (iatrogenic) effect.

Implementers' Perspectives on Fidelity of Implementation

Fidelity of implementation is an essential factor in realizing outcomes of evidence-based interventions. Yet perspectives vary on the degree to which fidelity is necessary or desirable, implementers may receive limited guidance on fidelity, and research on implementers' understanding and operationalization of fidelity is lacking. We conducted key informant interviews with 20 individuals who implemented an evidence-based teen dating violence prevention curriculum in seven sites. Interviews addressed how implementers defined fidelity and the adaptations they made in implementing the curriculum. Although all implementers reported delivering the curriculum with fidelity, their definitions of fidelity varied. Most defined fidelity in terms of adherence to the published curriculum, although definitions sometimes allowed modifications of pedagogy or adding or subtracting materials. A smaller group of implementers defined fidelity in terms of their perceptions of the curriculum's core messages. All implementers described variations from the published curriculum, which were frequently inconsistent with their own definitions of fidelity. Implementers committed to the value of implementation with fidelity may need support in its definition and application. Developers can support implementers by communicating program theory in terms that implementers will understand, clarifying expectations for fidelity and allowable adaptations, and codifying lessons learned from previous implementation.

Implementer-initiated adaptation of evidence-based interventions: kids remember the blue wig

Adaptation of evidence-based interventions by implementers is widespread. Although frequently viewed as departures from fidelity, adaptations may be positive in impact and consistent with fidelity. Research typically catalogs adaptations but rarely includes the implementers' perspectives on adaptation. We report data on individuals implementing an evidence-based teen dating violence prevention curriculum. Key informant interviews (n = 20) and an online focus group (n = 10) addressed reasons for adaptations, adaptation processes and kinds of adaptations. All implementers described making adaptations, which they considered necessary to achieving intended outcomes. Adaptations were tailored to needs of individual students or learning opportunities presented by current events, fine-tuned over repeated applications and shared with colleagues. Adaptations modified both content and delivery and included both planned and in-the-moment changes. Implementers made adaptations to increase student engagement, and to fit students' learning needs, learning style, social maturity and culture. Student engagement served as an indicator that adaptation might be needed and provided feedback about the immediate effects of the adaptation. These findings underscore the value of fidelity assessments that measure participant response, intervention-specific guidance to implementers and evaluation of the impact of adaptations on participant response and intervention outcomes.

London Education and Inclusion Project (LEIP): Exploring Negative and Null Effects of a Cluster-Randomised School-Intervention to Reduce School Exclusion--Findings from Protocol-Based Subgroup Analyses

This paper presents subgroup analyses from the London Education and Inclusion Project (LEIP). LEIP was a cluster-randomised controlled trial of an intervention called Engage in Education-London (EiE-L) which aimed to reduce school exclusions in those at greatest risk of exclusion. Pupils in the control schools attended an hour-long employability seminar. Minimisation was used to randomly assign schools to treatment and control following baseline data collection. The study involved 36 schools (17 in treatment—373 pupils; 19 in control—369 pupils) with >28% free school meal eligibility across London and utilised on pupil self-reports, teacher reports as well as official records to assess the effectiveness of EiE-L. Due to multiple data sources, sample sizes varied according to analysis. Analyses of pre-specified subgroups revealed null and negative effects on school exclusion following the intervention. Our findings suggest that the design and implementation of EiE-L may have contributed to the negative outcomes for pupils in the treatment schools when compared to those in the control schools. These findings call into question the effectiveness of bolt-on short-term interventions with pupils, particularly those at the highest risk of school exclusion and when they are faced with multiple problems. This is especially pertinent given the possibility of negative outcomes.

Trial Registration: Controlled Trials: ISRCTN23244695

Effects of a school-based prevention programme on smoking in early adolescence: a 6-month follow-up of the 'Eigenstandig werden' cluster randomised trial

OBJECTIVES

To test the effects of a school-based prevention programme on students' smoking-related behaviour, attitudes and knowledge 6 months after implementation over 2 school-years has ended.

DESIGN

Two-arm prospective cluster randomised controlled trial with a follow-up survey 6 months after end of programme implementation, that is, 26 months after baseline.

SETTING

45 public secondary schools from four federal states in Germany (Bremen, Hesse, North Rhine-Westphalia, Schleswig-Holstein).

PARTICIPANTS

A total of 3444 students from 172 classes with a mean age of 10.37 years (SD=0.59) and 47.9% girls at baseline. Analysis sample with follow-up up data merged to baseline data comprises 2513 datasets (73%).

INTERVENTION

'Eigenständig werden 5+6', a school-based prevention programme for grades 5 and 6 to enhance substance-specific and general life skills, consisting of 14 units (à 90 min) and two workshops (4-6 h) being taught over a time period of 2 school-years by trained teachers.

OUTCOME MEASURES

Lifetime and current smoking, incidence of smoking in baseline never smokers, smoking-related knowledge, attitudes, perceived norms of smoking and self-efficacy to refuse cigarette offers were assessed in students.

RESULTS

6 months after the end of programme implementation, students of intervention classes showed significantly lower rates for lifetime smoking (adjusted OR=0.63; 95% CI 0.41 to 0.96; p=0.026) and incidence of smoking (adjusted OR=0.66; 95% CI 0.43 to 1.00; p=0.047), a higher increase of smoking-related knowledge (adjusted β=9.38; 95% CI 6.73 to 12.04; p<0.001) and a greater change in attitudes towards a more critical perception of risks and disadvantages of smoking (adjusted β=0.10; 95% CI 0.03 to 0.16; p=0.002). No group differences were found for current smoking, perceived norms of smoking and self-efficacy to refuse cigarette offers.

CONCLUSIONS

Participation in the school-based prevention programme 'Eigenständig werden 5+6' may have small effects on smoking behaviour and attitudes and a moderate effect on smoking-related knowledge.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN99442407.

Development of a framework and coding system for modifications and adaptations of evidence-based interventions

BACKGROUND

Evidence-based interventions are frequently modified or adapted during the implementation process. Changes may be made to protocols to meet the needs of the target population or address differences between the context in which the intervention was originally designed and the one into which it is implemented [Addict Behav 2011, 36(6):630-635]. However, whether modification compromises or enhances the desired benefits of the intervention is not well understood. A challenge to understanding the impact of specific types of modifications is a lack of attention to characterizing the different types of changes that may occur. A system for classifying the types of modifications that are made when interventions and programs are implemented can facilitate efforts to understand the nature of modifications that are made in particular contexts as well as the impact of these modifications on outcomes of interest.

METHODS

We developed a system for classifying modifications made to interventions and programs across a variety of fields and settings. We then coded 258 modifications identified in 32 published articles that described interventions implemented in routine care or community settings.

RESULTS

We identified modifications made to the content of interventions, as well as to the context in which interventions are delivered. We identified 12 different types of content modifications, and our coding scheme also included ratings for the level at which these modifications were made (ranging from the individual patient level up to a hospital network or community). We identified five types of contextual modifications (changes to the format, setting, or patient population that do not in and of themselves alter the actual content of the intervention). We also developed codes to indicate who made the modifications and identified a smaller subset of modifications made to the ways that training or evaluations occur when evidence-based interventions are implemented. Rater agreement analyses indicated that the coding scheme can be used to reliably classify modifications described in research articles without overly burdensome training.

CONCLUSIONS

This coding system can complement research on fidelity and may advance research with the goal of understanding the impact of modifications made when evidence-based interventions are implemented. Such findings can further inform efforts to implement such interventions while preserving desired levels of program or intervention effectiveness.