Examining implementer fidelity: Conceptualizing and measuring adherence and competence

Brief modular anxiety intervention for primary care: Hybrid I pilot randomized controlled trial of feasibility, acceptability, effectiveness, and implementation potential

BACKGROUND

Anxiety is highly prevalent, but undertreated, in primary care. Brief, non-pharmacological interventions are needed. Modular Anxiety Skills Training (MAST), a cognitive-behavioral anxiety intervention, was developed for primary care and tailored for a Veteran sample (MAST-V). The purpose of this mixed methods pilot study was to evaluate MAST-V's feasibility, acceptability, and implementation potential, and preliminarily examine its effectiveness compared to Primary Care Behavioral Health (PCBH) usual care.

METHODS

This hybrid I randomized controlled trial (conducted 2019-2021) assigned 35 primary care patients (Mage = 47, 17 % female, 27 % racial/ethnic minority) with clinically significant anxiety symptoms to receive MAST-V or PCBH usual care. Participants completed validated measures of anxiety symptoms and functional impairment at 0, 4, 8, 12, and 16 weeks.

RESULTS

Participants attended more sessions in MAST-V than usual care. After necessary adjustments to reduce session duration, MAST-V will likely fit within PCBH practice parameters. Participants in both conditions valued treatment, but treatment satisfaction, credibility, and therapeutic alliance were higher for MAST-V. Study therapists achieved high treatment fidelity and rated MAST-V as highly feasible, acceptable, and appropriate for PCBH. They identified ways to address potential barriers to implementation. MAST-V was more effective than usual care in reducing anxiety symptoms and impairment.

LIMITATIONS

This was a small pilot study at a single site using study therapists. Results should be considered preliminary until replicated in a full-scale clinical trial.

CONCLUSIONS

This brief modular anxiety intervention, which was designed with implementation in mind, may help to address the anxiety treatment gap in primary care.

Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions

As evidence-based interventions are scaled, fidelity of implementation, and thus effectiveness, often wanes. Validated fidelity measures can improve researchers' ability to attribute outcomes to the intervention and help practitioners feel more confident in implementing the intervention as intended. We aim to provide a model for the validation of fidelity observation protocols to guide future research studying evidence-based interventions scaled-up under real-world conditions. We describe a process to build evidence of validity for items within the Session Review Form, an observational tool measuring fidelity to interactive drug prevention programs such as the Botvin LifeSkills Training program. Following Kane's (2006) assumptions framework requiring that validity evidence be built across four areas (scoring, generalizability, extrapolation, and decision), confirmatory factor analysis supported the hypothesized two-factor structure measuring quality of delivery (seven items assessing how well the material is implemented) and participant responsiveness (three items evaluating how well the intervention is received), and measurement invariance tests suggested the structure held across grade level and schools serving different student populations. These findings provide some evidence supporting the extrapolation assumption, though additional research is warranted since a more complete overall depiction of the validity argument is needed to evaluate fidelity measures.

Positive Attitude Upper Middle School social and emotional learning program: influences of implementation quality on program outcome

Introduction

There is an increased call for studies analyzing how implementation quality influences Social and Emotional Learning (SEL) program effectiveness.

Methods

The current dissemination study analyzed the effectiveness of the Positive Attitude Upper Middle School SEL program on a Portuguese nationwide sample composed of 813 middle school students (7th and 8th grade; 51.7% boys; M_age = 12.41, S.D. = 1.06), from 36 classrooms (M_classroom = 22.58; S.D. = 2.86), distributed between the control group (179 students), and three intervention groups (643 students) that reflected low, middle, and high implementer experience (respectively, Gulbenkian Academies of Knowledge, Positive Attitude Cadaval and Positive Attitude Torres Vedras). Dosage and fidelity (as implementation quality dimensions), gender, and classroom size (as individual and classroom-level variables) were also analyzed. Self-report questionnaires were administered pre- and post-intervention and at a 6-month follow-up.

Results

Multilevel models were employed, and results showed that participating in the PAUMS SEL program led to more positive trajectories in self-control, social awareness, relationship skills, and responsible decision-making when compared with control groups. Regarding implementation quality, only the implementer's experience impacted the effectiveness of the PAUMS SEL program; students in the Gulbenkian Academies of Knowledge intervention group displayed a less positive trajectory in self-control than students in the Positive Attitude Torres Vedras intervention group.

Discussion

Altogether, results showed that the PAUMS SEL program is ready for dissemination in Portugal, although a higher level of implementer experience is needed to achieve the best effectiveness, and they support the importance of analyzing implementer experience in SEL programs' effectiveness studies.

Fidelity to the ACT SMART Toolkit: an instrumental case study of implementation strategy fidelity

BACKGROUND

Evidence-based practices (EBPs) are shown to improve a variety of outcomes for autistic children. However, EBPs often are mis-implemented or not implemented in community-based settings where many autistic children receive usual care services. A blended implementation process and capacity-building implementation strategy, developed to facilitate the adoption and implementation of EBPs for autism spectrum disorder (ASD) in community-based settings, is the Autism Community Toolkit: Systems to Measure and Adopt Research-based Treatments (ACT SMART Toolkit). Based on an adapted Exploration, Adoption decision, Preparation, Implementation, Sustainment (EPIS) Framework, the multi-phased ACT SMART Toolkit is comprised of (a) implementation facilitation, (b) agency-based implementation teams, and (c) a web-based interface. In this instrumental case study, we developed and utilized a method to evaluate fidelity to the ACT SMART Toolkit. This study responds to the need for implementation strategy fidelity evaluation methods and may provide evidence supporting the use of the ACT SMART Toolkit.

METHODS

We used an instrumental case study approach to assess fidelity to the ACT SMART Toolkit during its pilot study with six ASD community agencies located in southern California. We assessed adherence, dose, and implementation team responsiveness for each phase and activity of the toolkit at both an aggregate and individual agency level.

RESULTS

Overall, we found that adherence, dose, and implementation team responsiveness to the ACT SMART Toolkit were high, with some variability by EPIS phase and specific activity as well as by ASD community agency. At the aggregate level, adherence and dose were rated notably lowest during the preparation phase of the toolkit, which is a more activity-intensive phase of the toolkit.

CONCLUSIONS

This evaluation of fidelity to the ACT SMART Toolkit, utilizing an instrumental case study design, demonstrated the potential for the strategy to be used with fidelity in ASD community-based agencies. Findings related to the variability of implementation strategy fidelity in the present study may also inform future adaptations to the toolkit and point to broader trends of how implementation strategy fidelity may vary by content and context.

Development and evaluation of a fidelity tool in a post-discharge responsive parenting intervention program for very preterm born children

The TOP program is a fully implemented responsive parenting intervention for very preterm born infants. Fidelity monitoring of interventions is important for preserving program adherence, impact outcomes and to make evidence-based adaptations. The aim of this study was to develop a fidelity tool for the TOP program following an iterative and co-creative process and subsequently evaluate the reliability of the tool. Three consecutive phases were carried out. Phase I: Initial development and pilot testing two methods namely self-report and video based observation. Phase II: Adaptations and refinements. Phase III: Evaluation of the psychometric properties of the tool based on 20 intervention videos rated by three experts.The interrater reliability of the adherence and competence subscales was good (ICC.81 to .84) and varied from moderate to excellent for specific items (ICC between .51 and .98). The FITT displayed a high correlation (Spearman's rho.79 to.82) between the subscales and total impression item. The co-creative and iterative process resulted in a clinical useful and reliable tool for evaluating fidelity in the TOP program. This study offers insights in the practical steps in the development of a fidelity assessment tool which can be used by other intervention developers.

Supporting young adults with intellectual/developmental disabilities to deliver a peer mentoring intervention: Evaluating fidelity and resources required

BACKGROUND

Peer-delivered interventions are a best practice in mental health, yet no such intervention exists for young adults (YA) with intellectual/developmental disabilities with co-occurring mental health conditions. We evaluated YA with intellectual/developmental disabilities' ability to deliver a novel peer mentoring intervention with fidelity and the supports received.

METHODS

We coded audio-recorded mentoring sessions to evaluate if 4 mentors adhered to 'content' (e.g., psychoeducation) and 'quality' (e.g., validation) fidelity criteria (codes: yes/no). We conducted content analysis of mentor support logs and interviews with mentors', mentors' parents and teachers to describe the supports mentors received.

RESULTS

Average fidelity for content criteria (M = 73.3%) was higher than quality criteria (M = 60.0%). Weekly support addressed logistics, delivering content, interpersonal interactions, emotional support, professionalism, and organisation. Family members and teachers rarely provided additional support.

CONCLUSIONS

With support, YA can deliver a peer mentoring intervention addressing mental health. Additional training activities will be developed to improve fidelity.

Physiotherapy informed by Acceptance and Commitment Therapy for chronic low back pain: A mixed-methods treatment fidelity evaluation

OBJECTIVES

A randomized controlled trial of a new type of Physiotherapy informed by Acceptance and Commitment Therapy (PACT), found that it improved functioning in people with chronic low back pain compared to usual physiotherapy care. Fidelity evaluation is necessary to understand trial processes and outcomes. This study evaluated PACT treatment fidelity including delivery, receipt, and enactment.

DESIGN

A mixed-methods study nested within a randomized controlled trial was conducted.

METHODS

A total of 72 (20% of total) PACT treatment audio files were independently assessed by two raters, according to a novel framework developed to measure PACT treatment content adherence, therapeutic alliance, ACT competence, and treatment enactment. Interview transcripts from 19 trial participants randomized to PACT were analysed thematically for evidence of treatment receipt and enactment.

RESULTS

PACT physiotherapists delivered treatment as intended with high content adherence and satisfactory therapeutic alliance, but ACT competence was low. Qualitative findings indicated participant receipt of 11/17 and enactment of 3/17 components; 89% (n = 17) and 47% (n = 9) of participants reported treatment receipt and enactment of at least one component, respectively.

CONCLUSIONS

This mixed-methods study of PACT treatment demonstrated high fidelity reflecting treatment content delivery and receipt, and therapeutic alliance. There was some evidence of treatment enactment in participants with chronic low back pain. Low ACT competence could be addressed through additional support and adaptations to therapeutic processes for delivery by physiotherapists.

Distilling Check-in/Check-Out into Its Core Practice Elements Through an Expert Consensus Process

Check-in/Check-out (CICO) is a widely implemented evidence-based program for supporting students with at-risk levels of social and emotional behavior concerns. It is comprised of several core features described in the previous literature, including practice elements, which are the specific actions that are delivered directly to students, and implementation components, which are actions that support the implementation by adults. Practice elements and implementation components are both important to implementation but have been combined and conflated in descriptions of CICO implementation. Well-defined and differentiated practice elements could provide improved clarity in communicating implementation expectations to front-line implementers as well as support future research into essential active ingredients and measures of front-line intervention fidelity. The purpose of the present study was to distill, differentiate, and operationally define the student-facing practice elements of CICO. A panel of research experts and practice experts participated in a three-round modified e-Delphi process that led to the identification and operational definition of 19 discreet practice elements organized into five domains. Results are discussed in terms in implications for future development of measures of commitment and intervention fidelity, future research into active ingredients of CICO, and in terms of how well-defined practice elements can improve communication of implementation expectations for front-line implementers of CICO such as teachers.

A Novel Peer-to-Peer Coaching Program to Support Digital Mental Health: Design and Implementation

Many individuals in need of mental health services do not currently receive care. Scalable programs are needed to reduce the burden of mental illness among those without access to existing providers. Digital interventions present an avenue for increasing the reach of mental health services. These interventions often rely on paraprofessionals, or coaches, to support the treatment. Although existing programs hold immense promise, providers must ensure that treatments are delivered with high fidelity and adherence to the treatment model. In this paper, we first highlight the tension between the scalability and fidelity of mental health services. We then describe the design and implementation of a peer-to-peer coach training program to support a digital mental health intervention for undergraduate students within a university setting. We specifically note strategies for emphasizing fidelity within our scalable framework, including principles of learning theory and competency-based supervision. Finally, we discuss future applications of this work, including the potential adaptability of our model for use within other contexts.

Which factors explain information exchange and user referral among program implementers?

The Program for Psychosocial Care and Comprehensive Health for Victims serves, on a yearly basis, an average of 25,000 users in northern Colombia alone. The program is implemented by multidisciplinary teams comprised of psychologists, social workers, and community facilitators, who step in at the individual, family, and community levels. An attempt has been made to determine the effect generated by the timeframe through which professionals have been engaged with the program-filling positions of centrality and betweenness within the networks of information exchange and user referral, including the potential mediating effect from the structure of the egocentric network of implementers in the two aforementioned networks and the moderating effect of the sense of belonging to a team of professionals. Both centrality and betweenness are positional measures describing the location actors occupied within the network structure. Centrality reflects the nominations made and receipt by an actor in a network and is considered an individual indicator of prominence and power. Betweenness shows the times that an actor act as a bridge among two actors in a network and it is considered an indicator of strategic positioning in social networks. An egocentric network is the local structure of relationships that each implementer maintains with his or her direct contacts. In this study, 112 active implementers were included, mostly women (n = 97, 88.2%), who had been working on the program for 16.9 months on average (SD = 14.7). Through conditional process analysis, it has been shown that the time that the implementers have been working on the program and the sense of belonging to the task team are relevant factors that interact with each other toward explaining the level of centrality and betweenness of professionals in the information exchange and user referral networks.

Acceptability, Appropriateness, and Preliminary Effects of the PrEP Diffusion Training for Lay HIV Workers: Increased PrEP Knowledge, Decreased Stigma, and Diffusion of Innovation

Training lay health workers is a critical intervention strategy to support HIV pre-exposure prophylaxis (PrEP) implementation. However, few evaluations of such trainings have been published. We conducted multi-time-point surveys to evaluate the effect of a training intervention on knowledge, PrEP stigma, and implementation behavior among lay HIV workers in China. Results indicated high acceptability and appropriateness of the training. PrEP knowledge score increased by 65% from pre- to post-training, and remained high one-month post-training. We observed a significant decrease in PrEP stigma and a positive influence on determinants of implementation behaviors. All lay HIV workers surveyed one-month after the training reported having disseminated PrEP information in social networks; 43% reported integrating PrEP education into routine work. The training is an effective strategy to build lay HIV workers' capacity in PrEP implementation. Longer-term post-training follow up would be ideal to further assess actual PrEP uptake and sustained effects on PrEP implementation.

Implementation strategy fidelity evaluation for a multidisciplinary Chest Injury Protocol (ChIP)

Background

Blunt chest wall injuries can lead to complications, especially without early intervention. A blunt Chest Injury Protocol (ChIP) was developed to help improve the consistency of evidence-based care following admission to the emergency department. Implementation strategy fidelity is the extent to which the strategies of implementation are delivered in line with the intended plan. The aim of this study was to assess fidelity to the strategies of the implementation plan developed for ChIP.

Methods

A retrospective evaluation of strategies used for implementation was performed, specifically the behaviour change techniques (BCTs). BCTs were used as part of an implementation plan derived based on the Behaviour Change Wheel from results from a staff survey at two hospitals.

Levels of implementation or adaptation for BCTs were scored by implementers as follows: ‘Were the behaviour change interventions implemented?’ (0 = ‘not implemented’, 1 = partially implemented, and 2 = fully implemented); ‘Were adaptations made to the implementation plan?’, scored 1 (many changes from plan) to 4 (just as planned). Free text explanation to their responses was also collected with supporting evidence and documentation (such as emails, implementation checklists, audit reports, and incident reports).

Results

There was high overall fidelity of 97.6% for BCTs partially or fully implemented. More than three quarters (32/42, 76.2%) of the BCTs were fully implemented with an additional 9/42 (21.4%) partially implemented. BCTs that were not fully implemented were social support, feedback on behaviour, feedback on outcomes of behaviour, adding objects to the environment, and restructuring the environment. The modes of delivery with poorer implementation or increased adaptations were clinical champions and audit/feedback.

Conclusions

This study describes the evaluation of implementation strategy fidelity in the acute care context. The systematic use and application of the behaviour change wheel was used to develop an implementation plan and was associated with high implementation strategy fidelity. A fidelity checklist developed during the implementation process may help implementers assess fidelity.

Trial registration

Trial registered on ANZCTR. Registration number ACTRN12618001548224, date approved 17/09/2018

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-021-00189-8.

Evaluating Delivery of a CBT-Based Group Intervention for Schoolchildren With Emotional Problems: Examining the Reliability and Applicability of a Video-Based Adherence and Competence Measure

Adherence and competence are essential parts of program fidelity and having adequate measures to assess these constructs is important. The Competence and Adherence Scale for Cognitive Behavioral Therapy (CAS CBT) was developed to evaluate the delivery of cognitive therapies for children with clinical anxiety. The present study is an assessment of the slightly adapted version of the CAS CBT evaluating the delivery of a Cognitive Behavioral Therapy (CBT)-based preventive group intervention: EMOTION: Kids Coping with Anxiety and Depression. This study was part of a Norwegian cluster randomized controlled trial (cRCT) investigating the effectiveness of a transdiagnostic intervention, the EMOTION program—an indicated prevention program targeting anxious and depressive symptoms. The applicability and psychometric properties of the CAS CBT were explored. Results are based on six raters evaluating 239 video-recorded sessions of the EMOTION program being delivered by 68 trained group leaders from different municipal services. Interrater reliability (intraclass correlation coefficients, ICC [3, 1]) indicated fair to good agreement between raters. Internal consistency of the instrument's key domains was calculated using the Omega coefficient which ranged between 0.70 to 0.94. There was a strong association between the two scales Adherence and Competence, and inter-item correlations were high across the items, except for the items rating the adherence to the session goals. Competence and Adherence Scale for Cognitive Behavioral Therapy is a brief measure for use in first-line services, with some promising features for easily assessing program fidelity, but some of the results indicated that the instrument should be improved. Future attention should also be made to adapt the instrument to fit better within a group setting, especially regarding evaluation of session goals. More research on how to adequately evaluate fidelity measures are also warranted.

Clinical Trial Registration:www.ClinicalTrials.gov, identifier: NCT02340637.

Fidelity and factors influencing implementation of Healthy Me, Healthy: process evaluation of a social marketing campaign for diet and physical activity behaviors of children in childcare

The effectiveness of interventions targeting children's eating and physical activity behaviors through childcare settings is inconsistent. To enhance public health impact, it is imperative to evaluate fidelity of implementing complex interventions in real-world settings. This study evaluated fidelity and contextual factors influencing implementation of Healthy Me, Healthy We (HMHW). HMHW was an 8-month social marketing campaign delivered through childcare to support children's healthy eating and physical activity. HMHW required two levels of implementation support (research team and childcare providers) and two levels of campaign delivery (childcare providers and parents). Process evaluation was conducted among childcare centers in the intervention group (n=48) of the cluster-randomized control trial. Measures included attendance logs, self-report surveys, observation checklists, field notes, and semi-structured interviews. A 35-item fidelity index was created to assess fidelity of implementation support and campaign delivery. The fidelity with which HMHW was implemented by childcare providers and parents was low (mean 17.4 out of 35) and decreased between childcare providers and parents. Childcare providers had high acceptability of the program and individual components (80 - 93%). Only half of parents felt intervention components were acceptable. Frequently cited barriers to implementation by childcare providers included time constraints, parent engagement, staff turnover, and restrictive policies. The lack of observable effect of HMHW on children's dietary or physical activity behaviors may be due to inadequate implementation at multiple levels. Different or additional strategies are necessary to support implementation of multilevel interventions, particularly when individuals are expected to deliver intervention components and support others in doing so.

Analyzing the role of fidelity of RTI Tier 2 reading intervention in Spanish kindergarten and first grade students

This study examined the role of fidelity in the reading outcomes within a Tier 2 intervention implemented by Spanish kindergarten and first grade school teachers. For this purpose, differences in reading performance growth were analyzed among at-risk students who received a Tier 2 intervention with high fidelity, at-risk students who received the same intervention with medium fidelity, and an at-risk control group that did not receive a Tier 2 intervention. Implementation fidelity was analyzed using direct observations and self-reports. All students were assessed three times during the academic year. Hierarchical linear modeling analyses were conducted to explore differences in growth rate. The at-risk children in the intervention condition showed a higher growth compared to at-risk children in the control condition, and specifically when the intervention was delivered with a high degree of fidelity.

Process evaluation of TeamUp: a movement-based psychosocial intervention for refugee children in the Netherlands

BACKGROUND

Nearly 60,000 people applied for asylum in the Netherland in 2015, confronting the governmental structures and services with great administrative, logistical and service provision challenges. Refugee children's psychosocial needs and wellbeing are often overlooked, and post-migration support is of pivotal importance.

METHODS

An easy accessible movement-based psychosocial intervention, called TeamUp, was developed for children aged 6-17 living in refugee reception centres. A mixed-method process evaluation was conducted of (1) implementation process, assessing attendance (n = 2183 children, and n = 209 children); (2) implementation quality, using structured observations at two time points to evaluate facilitator's (2a) individual-level fidelity (n = 81 facilitators); (2b) team-level fidelity (n = 22 teams); (2c) facilitators' competencies (n = 81); (2d) trainee perceived self-efficacy pre-post training (n = 73); and (3) perceptions on implementation and outcomes, employing a survey (n = 99), focus group discussions and key informant interviews with children (n = 94), facilitators (n = 24) and reception centre staff (n = 10).

RESULTS

Attendance lists showed a mean of 8.5 children per session, and children attending 31.3% of sessions. Structured observations demonstrated 49.2% and 58.2% individual-level fidelity, 72.5% and 73.0% team-level fidelity, and 82.9% and 88.4% adequacy in competencies, each at T1 and T2 respectively. The main reported challenges included managing children's energy regulation (e.g. offering settling moments) and challenging behaviour. Training participation significantly improved perceived self-efficacy for trainees. The facilitator survey demonstrated on average, high satisfaction and self-efficacy, low experienced burden, and high perceived capacity-building support. Qualitatively, TeamUp was positively perceived by all stakeholders and was regarded as contributing to children's psychosocial outcomes.

CONCLUSION

(1) Attendance and group size were lower than expected. (2) The intervention's facilitator fidelity ranged from moderate to adequate-exhibiting a need for specific fidelity and capacity strengthening-while facilitator competencies were high. Trainee's perceived self-efficacy improved significantly following a 2-day training. (3) Facilitators expressed high levels of satisfaction, self-efficacy and support, and low burden. The intervention was positively perceived by all stakeholders and to have a positive impact on children's psychosocial learning and wellbeing.

Fidelity evaluation of the compared procedures for conducting the PVS-PREDIAPS implementation strategy to optimize diabetes prevention in primary care

Background

Assessing the fidelity of an implementation strategy is important to understand why and how the strategy influences the uptake of evidence-based interventions. The present study aims to assess the fidelity of the two procedures for engaging primary care (PC) professionals and for the deployment of an implementation strategy for optimizing type 2 diabetes prevention in routine PC within the PREDIAPS randomized cluster implementation trial.

Method

We conducted a mixed-method fidelity evaluation study of the PVS-PREDIAPS implementation strategy. Nine PC centers from the Basque Health Service (Osakidetza) were allocated to two different procedures to engage physicians and nurses and deploy a implementation strategy to model and adapt the clinical intervention and its implementation to their specific contexts: a Global procedure, promoting the cooperation of all healthcare professionals from the beginning; or a Sequential procedure, centered first on nurses who then pursued the pragmatic cooperation of physicians. Process indicators of the delivery and receipt of implementation strategy actions, documented modifications to the planned implementation strategy, and a structured group interview with centers’ leaders were all used to assess the following components of fidelity: adherence, dose, quality of delivery, professionals’ responsiveness and program differentiation.

Results

Generally, the procedures compared for professionals engagement and deployment of the implementation strategy were carried out with the planned differentiation. Nonetheless, some unexpected between-group differences were observed, the initial rate of collaboration of nurses being higher in the Sequential (93%) than in the Global (67%) groups. Exposure rate to the programed implementation actions (% of hours received out of those delivered) were similar in both groups by professional category, with nurses (86%) having a higher rate of exposure than physicians (75%). Professionals identified half of the planned discrete strategies and their rating of strategies’ perceived usefulness was overwhelmingly positive, with few differences between Sequential and Global centers.

Conclusions

The PVS-PREDIAPS implementation strategy has been implemented with high fidelity and minor unplanned reactive modifications. Professionals’ exposure to the implementation strategy was high in both groups. The centers’ organizational context (i.e., work overload) led to small mismatches between groups in participation and exposure of professionals to implementation actions.

Trial registration

Clinicaltrials.gov identifier: NCT03254979. Registered 16 August 2017.

Different Approaches to Address Bullying in KiVa Schools: Adherence to Guidelines, Strategies Implemented, and Outcomes Obtained

We examined the extent to which school personnel implementing the KiVa® antibullying program in Finland during 2009–2015 systematically employed the program-recommended approaches (confronting or non-confronting), used one or the other depending on the bullying case (case-specific approach), or used their own adaptation when talking to perpetrators of bullying, and whether they organized follow-up meetings after such discussions. In addition to investigating adherence to program guidelines, we tested how effective these different approaches were in stopping bullying. Finally, we tested the contribution of follow-up meetings and the number of years KiVa had been implemented in a school to the effectiveness of the interventions, using reports from both school personnel and victimized students. The data were collected annually across 6 years via online questionnaires and included responses from 1221 primary and secondary schools. The school personnel were more likely to use the confronting approach than the non-confronting approach. Over time, rather than sticking to the two program-recommended approaches, they made adaptations (e.g., combining the two; using their own approach). Two-level regression analyses indicated that the discussions were equally effective, according to both personnel and victimized students, when the confronting, non-confronting, or a case-specific approach had been used. The discussions were less effective when the personnel used their own adaptation or could not specify the method used. Perceived effectiveness was higher in primary school and when follow-up meetings were organized systematically after each intervention, but unrelated to the number of years KiVa had been implemented.

Parenting support to prevent overweight during regular well-child visits in 0-3 year old children (BBOFT+ program), a cluster randomized trial on the effectiveness on child BMI and health behaviors and parenting

Background

Prevention of overweight during early childhood seems promising.

Objective

To evaluate the effectiveness of the parenting-based BBOFT+ overweight prevention program on child BMI, child health behavior and parenting behavior among 0–36 month old children. BBOFT+ is an acronym for the key healthy lifestyle behaviors that are targeted in the BBOFT+ intervention: breastfeeding (B), daily breakfast (B), daily going outdoors (O), limiting sweet beverages (in Dutch, F) and minimal TV or computer time (T), complemented with healthy sleep behavior and improvement of parenting skills (+).

Methods

A cluster randomized controlled trial in newborn children visiting well-baby clinics, comparing the BBOFT+ intervention (N = 901) with care as usual (CAU) (N = 1094). In both groups, parents received regular well-child visits (±11 visits in the first 3 years). In the intervention group, care was supplemented with the BBOFT+ program, which focuses on improving parenting skills from birth onwards to increase healthy behavior. Questionnaires were filled in at child's age 2–4 weeks, 6, 14 and 36 months. In multivariate analyses we corrected for child’s birthweight, age, ethnic background, mother’s educational level and BMI.

Results

No differences were found in weight status at 36 months between intervention and control group children. At 6 months, BBOFT+ parents reported their child drinking less sweet beverages than control parents (48% vs 54%;p = .027), and going outdoors daily with their child less often (57% vs 62%;p = .03). At 14 months, more BBOFT+ parents than control parents reported to have breastfed for six months or longer (32% vs 29%;p = .022). At 36 months, more BBOFT+ parents than control parents reported their child going outside daily (78% vs 72%;p = .011) and having less TV/computer time on week- (38% vs 46%;p = .001) and weekend days (48% vs 56%;p = .002). Also, BBOFT+ parents reported having more parental control than control parents (3.92 vs 3.89;p = .02). No significant differences were found for daily breakfast, sleep duration and parenting practices in adjusted analyses.

Conclusion

The BBOFT+ overweight prevention program showed small improvements in parent-reported child health behaviors, compared to care as usual; no effect was observed on child BMI. The identified modifiable elements are potentially relevant for interventions that aim to prevent overweight.

Addressing the Adherence-Adaptation Debate: Lessons from the Replication of an Evidence-Based Sexual Health Program in School Settings

Whether high adherence to programs is necessary to achieve program outcomes is an area of great debate. The objectives of this study were to determine the frequency, type, and rationale of adaptations made in the implementation of an evidence-based program and to determine program outcomes for intervention program participants, as compared to comparison participants, by the level of adaptations. A total of 1608 participants in 45 classrooms participated. Percent adaptations was calculated by classroom. Thematic qualitative analysis was used to categorize types and rationales for adaptations. Program outcomes by level of adaptations were determined using logistic regression analyses and mean differences. Propensity score matching methods were used to create comparability between adaptation subgroup participants and comparison participants. Adaptations ranged from 2 to 97% across classrooms, with mean adaptations of 63%. Thematic analysis revealed that the adaptations made were related to delivery of content, rather than to the content itself and in response to participant needs and setting constraints. Program outcomes did not appear to be reduced for the high-adaptation subgroup. Understanding both rationale (intent) and type of adaptation made is crucial to understanding the complexity of adaptations. These finding support the argument for allowing facilitators some flexibility and autonomy to adapt the delivery of prescribed content to participant needs and setting constraints.

Psychometric Properties of a Fidelity Scale for Illness Management and Recovery

This study examined the psychometric properties and feasibility of the Illness Management and Recovery (IMR) Fidelity scale. Despite widespread use of the scale, the psychometric properties have received limited attention. Trained fidelity assessors conducted assessments four times over 18 months at 11 sites implementing IMR. The IMR Fidelity scale showed excellent interrater reliability (.99), interrater item agreement (94%), internal consistency (.91–.95 at three time points), and sensitivity to change. Frequency distributions generally showed that item ratings included the entire range. The IMR Fidelity scale has excellent psychometric properties and should be used to evaluate and guide the implementation of IMR.

Trial registration: ClinicalTrials.gov Identifier: NCT03271242.

Psychological interventions for epilepsy: How good are trialists at assessing their implementation fidelity, what are the barriers, and what are journals doing to encourage it? A mixed methods study

INTRODUCTION

Psychological interventions hold promise for the epilepsy population and continue to be trialed to determine their efficacy. Such interventions present opportunities for variance in delivery. Therefore, to accurately interpret a trial's estimate of effect, information on implementation fidelity (IF) is required. We present a novel 3-part study. Part 1 systematically rated trials for the extent to which they reported assessing whether the intervention was delivered as intended (adherence) and with what sort of skill (competence). Part 2 identified barriers to reporting and assessing on fidelity perceived by trialists. Part 3 determined what journals publishing epilepsy trials are doing to support IFs reporting.

METHODS

Articles for 50 randomized controlled trials (RCTs)/quasi-RCTs of psychological interventions identified by Cochrane searches were rated using the Psychotherapy Outcome Study Methodology Rating Form's fidelity items. The 45 corresponding authors for the 50 trials were invited to complete the 'Barriers to Treatment Integrity Implementation Survey'. 'Instructions to Authors' for the 17 journals publishing the trials were reviewed for endorsement of popular reporting guidelines which refer to fidelity (Consolidated Standards of Reporting Trials (CONSORT) statement or Journal Article Reporting Standards [JARS]) and asked how they enforced compliance.

RESULTS

Part 1: 15 (30%) trials reported assessing for adherence, but only 2 (4.3%) gave the result. Four (8.5%) reported assessing for competence, 1 (2.1%) gave the result. Part 2: 22 trialists - mostly chief investigators - responded. They identified 'lack of theory and specific guidelines on treatment integrity procedures', 'time, cost, and labor demands', and 'lack of editorial requirement' as "strong barriers". Part 3: Most (15, 88.2%) journals endorsed CONSORT or JARS, but only 5 enforced compliance.

CONCLUSIONS

Most trials of psychological interventions for epilepsy are not reported in a transparent way when it comes to IF. The barriers' trialists identify for this do not appear insurmountable. Addressing them could ultimately help the field to better understand how best to support the population with epilepsy.

Testing a Motivational Interviewing Implementation Intervention in Adolescent HIV Clinics: Protocol for a Type 3, Hybrid Implementation-Effectiveness Trial

Background

Motivational interviewing (MI) has been shown to effectively improve self-management for youth living with HIV (YLH) and has demonstrated success across the youth HIV care cascade—currently, the only behavioral intervention to do so. Substantial barriers prevent the effective implementation of MI in real-world settings. Thus, there is a critical need to understand how to implement evidence-based practices (EBPs), such as MI, and promote behavior change in youth HIV treatment settings as risk-taking behaviors peak during adolescence and young adulthood.

Objective

This study aims to describe the Adolescent Medicine Trials Network for HIV/AIDS Interventions (ATN) protocol of a tailored MI (TMI) implementation-effectiveness trial (ATN 146 TMI) to scale up an EBP in multidisciplinary adolescent HIV settings while balancing flexibility and fidelity. This protocol is part of the Scale It Up program described in this issue.

Methods

This study is a type 3, hybrid implementation-effectiveness trial that tests the effect of TMI on fidelity (MI competency and adherence to program requirements) while integrating findings from two other ATN protocols described in this issue—ATN 153 Exploration, Preparations, Implementation, Sustainment and ATN 154 Cascade Monitoring. ATN 153 guides the mixed methods investigation of barriers and facilitators of implementation, while ATN 154 provides effectiveness outcomes. The TMI study population consists of providers at 10 adolescent HIV care sites around the United States. These 10 clinics are randomly assigned to 5 blocks to receive the TMI implementation intervention (workshop and trigger-based coaching guided by local implementation teams) utilizing the dynamic wait-listed controlled design. After 12 months of implementation, a second randomization compares a combination of internal facilitator coaching with the encouragement of communities of practice (CoPs) to CoPs alone. Participants receive MI competency assessments on a quarterly basis during preimplementation, during the 12 months of implementation and during the sustainment period for a total of 36 months. We hypothesize that MI competency ratings will be higher among providers during the TMI implementation phase compared with the standard care phase, and successful implementation will be associated with improved cascade-related outcomes, namely undetectable viral load and a greater number of clinic visits among YLH.

Results

Participant recruitment began in August 2017 and is ongoing. As of mid-May 2018, TMI has 150 active participants.

Conclusions

This protocol describes the underlying theoretical framework, study design, measures, and lessons learned for TMI, a type 3, hybrid implementation-effectiveness trial, which has the potential to scale up MI and improve patient outcomes in adolescent HIV settings.

Trial Registration

ClinicalTrials.gov NCT03681912; https://clinicaltrials.gov/ct2/show/NCT03681912 (Archived by WebCite at http://www.webcitation.org/754oT7Khx)

International Registered Report Identifier (IRRID)

DERR1-10.2196/11200

Self-Management education for adults with poorly controlled epILEpsy [SMILE (UK)]: a randomised controlled trial

BACKGROUND

Epilepsy is a common neurological condition resulting in recurrent seizures. Research evidence in long-term conditions suggests that patients benefit from self-management education and that this may improve quality of life (QoL). Epilepsy self-management education has yet to be tested in a UK setting.

OBJECTIVES

To determine the effectiveness and cost-effectiveness of Self-Management education for people with poorly controlled epILEpsy [SMILE (UK)].

DESIGN

A parallel pragmatic randomised controlled trial.

SETTING

Participants were recruited from eight hospitals in London and south-east England.

PARTICIPANTS

Adults aged ≥ 16 years with epilepsy and two or more epileptic seizures in the past year, who were currently being prescribed antiepileptic drugs.

INTERVENTION

A 2-day group self-management course alongside treatment as usual (TAU). The control group received TAU.

MAIN OUTCOME MEASURES

The primary outcome is QoL in people with epilepsy at 12-month follow-up using the Quality Of Life In Epilepsy 31-P (QOLIE-31-P) scale. Other outcomes were seizure control, impact of epilepsy, medication adverse effects, psychological distress, perceived stigma, self-mastery and medication adherence. Cost-effectiveness analyses and a process evaluation were undertaken.

RANDOMISATION

A 1 : 1 ratio between trial arms using fixed block sizes of two.

BLINDING

Participants were not blinded to their group allocation because of the nature of the study. Researchers involved in data collection and analysis remained blinded throughout.

RESULTS

The trial completed successfully. A total of 404 participants were enrolled in the study [SMILE (UK), n = 205; TAU, n = 199] with 331 completing the final follow-up at 12 months [SMILE (UK), n = 163; TAU, n = 168]. In the intervention group, 61.5% completed all sessions of the course. No adverse events were found to be related to the intervention. At baseline, participants had a mean age of 41.7 years [standard deviation (SD) 14.1 years], and had epilepsy for a median of 18 years. The mean QOLIE-31-P score for the whole group at baseline was 66.0 out of 100.0 (SD 14.2). Clinically relevant levels of anxiety symptoms were reported in 53.6% of the group and depression symptoms in 28.0%. The results following an intention-to-treat analysis showed no change in any measures at the 12-month follow-up [QOLIE-31-P: SMILE (UK) mean: 67.4, SD 13.5; TAU mean: 69.5, SD 14.8]. The cost-effectiveness study showed that SMILE (UK) was possibly cost-effective but was also associated with lower QoL. The process evaluation with 20 participants revealed that a group course increased confidence by sharing with others and improved self-management behaviours.

CONCLUSIONS

For people with epilepsy and persistent seizures, a 2-day self-management education course is cost-saving, but does not improve QoL after 12-months or reduce anxiety or depression symptoms. A psychological intervention may help with anxiety and depression. Interviewed participants reported attending a group course increased their confidence and helped them improve their self-management.

FUTURE WORK

More research is needed on self-management courses, with psychological components and integration with routine monitoring.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN57937389.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 22, No. 21. See the NIHR Journals Library website for further project information.

Measuring clinical competencies in facilitating group-based rehabilitation interventions: development of a new competency checklist

OBJECTIVE

Group-based intervention formats are common in rehabilitation, but no tool for objectively measuring clinical competencies in group facilitation currently exists. We aimed to develop a psychometrically sound group facilitation competency checklist for use in clinical, training, and research settings.

METHOD

The Delphi method of expert consensus was used to establish checklist items that clearly describe competencies considered important for effective group facilitation. Inter-rater reliability was determined with two experienced psychologists who used the checklist to rate the competencies of psychology trainees facilitating a memory skills group.

RESULTS

After two Delphi rounds, consensus was reached on 17 items, defined as at least 80% agreement among the panel of 15 experts. The four checklist item categories were (a) Facilitating focused group discussion, (b) Communication skills, (c) Interpersonal style, and (d) Session structure. One item was removed after piloting. Inter-rater reliability was excellent (88% agreement) using a simple coding method (competent/incompetent). When using a detailed coding method that discriminated between 'done adequately' and 'done well', inter-rater reliability was weaker (κ = 0.481, 55% agreement); however, it improved to almost perfect after the raters calibrated their standards.

CONCLUSION

The new group facilitation competency checklist is fit for purpose for measuring clinical competencies in delivering group-based rehabilitation interventions and can be used in the training of effective group facilitators.

Assessing Treatment Fidelity within an Epilepsy Randomized Controlled Trial: Seizure First Aid Training for People with Epilepsy Who Visit Emergency Departments

Purpose

To measure fidelity with which a group seizure first aid training intervention was delivered within a pilot randomized controlled trial underway in the UK for adults with epilepsy who visit emergency departments (ED) and informal carers. Estimates of its effects, including on ED use, will be produced by the trial. Whilst hardly ever reported for trials of epilepsy interventions—only one publication on this topic exists—this study provides the information on treatment fidelity necessary to allow the trial's estimates to be accurately interpreted. This rare worked example of how fidelity can be assessed could also provide guidance sought by neurology trialists on how to assess fidelity.

Methods

53 patients who had visited ED on ≥2 occasions in prior year were recruited for the trial; 26 were randomized to the intervention. 7 intervention courses were delivered for them by one facilitator. Using audio recordings, treatment “adherence” and “competence” were assessed. Adherence was assessed by a checklist of the items comprising the intervention. Using computer software, competence was measured by calculating facilitator speech during the intervention (didacticism). Interrater reliability was evaluated by two independent raters assessing each course using the measures and their ratings being compared.

Results

The fidelity measures were found to be reliable. For the adherence instrument, raters agreed 96% of the time, PABAK-OS kappa 0.91. For didacticism, raters' scores had an intraclass coefficient of 0.96. In terms of treatment fidelity, not only were courses found to have been delivered with excellent adherence (88% of its items were fully delivered) but also as intended they were highly interactive, with the facilitator speaking for, on average, 55% of course time.

Conclusions

The fidelity measures used were reliable and showed that the intervention was delivered as attended. Therefore, any estimates of intervention effect will not be influenced by poor implementation fidelity.

Achieving optimal technology and behavioral uptake of single and combined interventions of water, sanitation hygiene and nutrition, in an efficacy trial (WASH benefits) in rural Bangladesh

BACKGROUND

Uptake matters for evaluating the health impact of water, sanitation and hygiene (WASH) interventions. Many large-scale WASH interventions have been plagued by low uptake. For the WASH Benefits Bangladesh efficacy trial, high uptake was a prerequisite. We assessed the degree of technology and behavioral uptake among participants in the trial, as part of a three-paper series on WASH Benefits Intervention Delivery and Performance.

METHODS

This study is a cluster randomized trial comprised of geographically matched clusters among four districts in rural Bangladesh. We randomly allocated 720 clusters of 5551 pregnant women to individual or combined water, sanitation, handwashing, and nutrition interventions, or a control group. Behavioral objectives included; drinking chlorine-treated, safely stored water; use of a hygienic latrine and safe feces disposal at the compound level; handwashing with soap at key times; and age-appropriate nutrition behaviors (pregnancy to 24 months) including a lipid-based nutrition supplement (LNS). Enabling technologies and behavior change were promoted by trained local community health workers through periodic household visits. To monitor technology and behavioral uptake, we conducted surveys and spot checks in 30-35 households per intervention arm per month, over a 20-month period, and structured observations in 324 intervention and 108 control households, approximately 15 months after interventions commenced.

RESULTS

In the sanitation arms, observed adult use of a hygienic latrine was high (94-97% of events) while child sanitation practices were moderate (37-54%). In the handwashing arms, handwashing with soap was more common after toilet use (67-74%) than nonintervention arms (18-40%), and after cleaning a child's anus (61-72%), but was still low before food handling. In the water intervention arms, more than 65% of mothers and index children were observed drinking chlorine-treated water from a safe container. Reported LNS feeding was > 80% in nutrition arms. There was little difference in uptake between single and combined intervention arms.

CONCLUSIONS

Rigorous implementation of interventions deployed at large scale in the context of an efficacy trial achieved high levels of technology and behavioral uptake in individual and combined WASH and nutrition intervention households. Further work should assess how to achieve similar uptake levels under programmatic conditions.

TRIAL REGISTRATION

WASH Benefits Bangladesh: ClinicalTrials.gov, identifier: NCT01590095 . Registered on April 30, 2012.

WASH Benefits Bangladesh trial: system for monitoring coverage and quality in an efficacy trial

BACKGROUND

Researchers typically report more on the impact of public health interventions and less on the degree to which interventions were followed implementation fidelity. We developed and measured fidelity indicators for the WASH Benefits Bangladesh study, a large-scale efficacy trial, in order to identify gaps between intended and actual implementation.

METHODS

Community health workers (CHWs) delivered individual and combined water, sanitation, handwashing (WSH) and child nutrition interventions to 4169 enrolled households in geographically matched clusters. Households received free enabling technologies (insulated water storage container; sani-scoop, potty, double-pit, pour-flush latrine; handwashing station, soapy-water storage bottle), and supplies (chlorine tablets, lipid-based nutrient supplements, laundry detergent sachets) integrated with parallel behavior-change promotion. Behavioral objectives were drinking treated, safely stored water, safe feces disposal, handwashing with soap at key times, and age-appropriate nutrition behaviors. We administered monthly surveys and spot-checks to households from randomly selected clusters for 6 months early in the trial. If any fidelity measures fell below set benchmarks, a rapid response mechanism was triggered.

RESULTS

In the first 3 months, functional water seals were detected in 33% (14/42) of latrines in the sanitation only arm; 35% (14/40) for the combined WSH arm; and 60% (34/57) for the combined WSH and Nutrition arm, all falling below the pre-set benchmark of 80%. Other fidelity indicators met the 65 to 80% uptake benchmarks. Rapid qualitative investigations determined that households concurrently used their own latrines with broken water seals in parallel with those provided by the trial. In consultation with the households, we closed pre-existing latrines without water seals, increased the CHWs' visit frequency to encourage correct maintenance of latrines with water seals, and discouraged water-seal removal or breakage. At the sixth assessment, 86% (51/59) of households were in sanitation only; 92% (72/78) in the combined WSH; and 93% (71/76) in the combined WSH and Nutrition arms had latrines with functional water seals.

CONCLUSIONS

An intensive implementation fidelity monitoring and rapid response system proved beneficial for this efficacy trial. To implement a routine program at scale requires further research into an adaptation of fidelity monitoring that supports program effectiveness.

TRIAL REGISTRATION

WASH Benefits Bangladesh: ClinicalTrials.gov, ID: NCT01590095 . Registered on 30 April 2012.

Implementation fidelity of a voluntary sector-led diabetes education programme

Purpose

The quality of voluntary sector-led community health programmes is an important concern for service users, providers and commissioners. Research on the fidelity of programme implementation offers a basis for assessing and further enhancing practice. The purpose of this paper is to report on the fidelity assessment of Living Well Taking Control (LWTC) – a voluntary sector-led, community-based education programme in England focussing on the prevention and management of type 2 diabetes.

Design/methodology/approach

This fidelity of implementation (FoI) study was conducted with the Devon-based LWTC programme. A fidelity checklist was developed to analyse audio records of group-based lifestyle education sessions – implementation was rated in terms of adherence to protocol and competence in delivery; the influence of wider contextual factors was also assessed. Kappa statistics (κ) were used to test for inter-rater agreement. Course satisfaction data were used as a supplementary indicator of facilitator competence.

Findings

Analysis of 28 sessions, from five diabetes prevention and two diabetes management groups (total participants, n=49), yielded an overall implementation fidelity score of 77.3 per cent for adherence (moderate inter-rater agreement, κ=0.60) and 95.1 per cent for competence (good inter-rater agreement, κ=0.71). The diabetes prevention groups consistently achieved higher adherence scores than the diabetes management groups. Facilitator competence was supported by high participant satisfaction ratings.

Originality/value

An appropriate level of implementation fidelity was delivered for the LWTC group-based education programme, which provides some confidence that outcomes from the programme reflected intervention effectiveness. This study demonstrates the viability of assessing the FoI in a voluntary sector-led public health initiative and the potential of this method for assuring quality and informing service development.

Do mobile clinics provide high-quality antenatal care? A comparison of care delivery, knowledge outcomes and perception of quality of care between fixed and mobile clinics in central Haiti

Background

Antenatal care (ANC) is an important health service for women in developing countries, with numerous proven benefits. Global coverage of ANC has steadily increased over the past 30 years, in part due to increased community-based outreach. However, commensurate improvements in health outcomes such as reductions in the prevalence of maternal anemia and infants born small-for-gestational age have not been achieved, even with increased coverage, indicating that quality of care may be inadequate. Mobile clinics are one community-based strategy used to further improve coverage of ANC, but their quality of care delivery has rarely been evaluated.

Methods

To determine the quality of care of ANC in central Haiti, we compared adherence to national guidelines between fixed and mobile clinics by performing direct observations of antenatal care consultations and exit interviews with recipients of care using a multi-stage random sampling procedure. Outcome variables were eight components of care, and women’s knowledge and perception of care quality.

Results

There were significant differences in the predicted proportion or probability of recommended services for four of eight care components, including intake, laboratory examinations, infection control, and supplies, iron folic acid supplements and Tetanus Toxoid vaccine provided to women. These care components were more likely performed in fixed clinics, except for distribution of supplies, iron-folic acid supplements, and Tetanus Toxoid vaccine, more likely provided in mobile clinics. There were no differences between clinic type for the proportion of total physical exam procedures performed, health and communication messages delivered, provider communication or documentation. Women’s knowledge about educational topics was poor, but women perceived extremely high quality of care in both clinic models.

Conclusions

Although adherence to guidelines differed by clinic type for half of the care components, both clinics had a low percentage of overall services delivered. Efforts to improve provider performance and quality are therefore needed in both models. Mobile clinics must deliver high-quality ANC to improve health and nutrition outcomes.

Electronic supplementary material

The online version of this article (10.1186/s12884-017-1546-7) contains supplementary material, which is available to authorized users.

Implementation fidelity of a self-management course for epilepsy: method and assessment

BACKGROUND

Complex interventions such as self-management courses are difficult to evaluate due to the many interacting components. The way complex interventions are delivered can influence the effect they have for patients, and can impact the interpretation of outcomes of clinical trials. Implementation fidelity evaluates whether complex interventions are delivered according to protocol. Such assessments have been used for one-to-one psychological interventions; however, the science is still developing for group interventions.

METHODS

We developed and tested an instrument to measure implementation fidelity of a two-day self-management course for people with epilepsy, SMILE(UK). Using audio recordings, we looked at adherence and competence of course facilitators. Adherence was assessed by checklists. Competence was measured by scoring group interaction, an overall impression score and facilitator "didacticism". To measure "didacticism", we developed a novel way to calculate facilitator speech using computer software. Using this new instrument, implementation fidelity of SMILE(UK) was assessed on three modules of the course, for 28% of all courses delivered.

RESULTS

Using the instrument for adherence, scores from two independent raters showed substantial agreement with weighted Kappa of 0.67 and high percent agreement of 81.2%. For didacticism, the results from both raters were highly correlated with an intraclass coefficient of 0.97 (p < 0.0001). We found that the courses were delivered with a good level of adherence (> 50% of scored items received the maximum of 2 points) and high competence. Groups were interactive (mean score: 1.9-2.0 out of 2) and the overall impression was on average assessed as "good". Didacticism varied from 42% to 93% of total module time and was not associated with the other competence scores.

CONCLUSION

The instrument devised to measure implementation fidelity was reproducible and easy to use. The courses for the SMILE(UK) study were delivered with a good level of adherence to protocol while not compromising facilitator competence.

TRIAL REGISTRATION

ISRCTN57937389 .

Évaluation de la qualité de l’implantation du programme Pare-Chocs à l’école secondaire

Malgré l’importante augmentation de la prévalence de la dépression durant l’adolescence, seulement une faible proportion des adolescents qui présentent des symptômes dépressifs élevés bénéficie des services de santé mentale en milieu scolaire. De plus, les programmes offerts à ces élèves souffrent souvent d’une implantation lacunaire, limitant ainsi leur efficacité potentielle. L’identification des facteurs qui influencent la fidélité de l’implantation est donc essentielle pour améliorer la qualité des services offerts. Guidés par le modèle d’évaluation basé sur la théorie des programmes (theory-driven evaluation), nous évaluons la qualité de l’implantation du programme Pare-Chocs et les facteurs qui l’ont influencée à l’aide des composantes du modèle d’action de Chen (2005). Une approche multi-méthodes a été utilisée pour obtenir des données quantitatives sur la fidélité d’implantation dans les six groupes et des données qualitatives sur les composantes du modèle d’action. Les résultats suggèrent que l’adhésion, le dosage et la participation étaient élevés. Les contraintes de temps et le peu de formation antérieure en lien avec la théorie du programme ont limité la fidélité de l’implantation tandis que la formation, la supervision et le manuel y ont contribué. Ces résultats confirment que les programmes de prévention disséminés dans les milieux scolaires peuvent être implantés avec un niveau de fidélité élevé, même si certains défis doivent être pris en considération lors de la planification de l’intervention pour contribuer à des effets plus importants du programme. De plus, la fidélité devrait être évaluée de manière systématique dans ce contexte.

Measuring trainer fidelity in the transfer of suicide prevention training

BACKGROUND

Finding effective and efficient models to train large numbers of suicide prevention interventionists, including 'hotline' crisis counselors, is a high priority. Train-the-trainer (TTT) models are widely used but understudied.

AIMS

To assess the extent to which trainers following TTT delivered the Applied Suicide Intervention Skills Training (ASIST) program with fidelity, and to examine fidelity across two trainings and seven training segments.

METHOD

We recorded and reliably rated trainer fidelity, defined as adherence to program content and competence of program delivery, for 34 newly trained ASIST trainers delivering the program to crisis center staff on two separate occasions. A total of 324 observations were coded. Trainer demographics were also collected.

RESULTS

On average, trainers delivered two-thirds of the program. Previous training was associated with lower levels of trainer adherence to the program. In all, 18% of trainers' observations were rated as solidly competent. Trainers did not improve fidelity from their first to second training. Significantly higher fidelity was found for lectures and lower fidelity was found for interactive training activities including asking about suicide and creating a safe plan.

CONCLUSIONS

We found wide variability in trainer fidelity to the ASIST program following TTT and few trainers had high levels of both adherence and competence. More research is needed to examine the cost-effectiveness of TTT models.

Assessing Fidelity of a Community-Based Opioid Overdose Prevention Program: Modification of the Fidelity Checklist

BACKGROUND

The evaluation and maintenance of treatment fidelity is an important methodological consideration in intervention research. Treatment fidelity is the degree to which interventions are delivered as outlined by the program developers. A lack of fidelity to the intervention model has the potential to be a large source of error, including type I and type II error. Monitoring and assuring fidelity is critical to assuring the validity of the interventions. The Fidelity Checklist is a reliable and valid tool designed to measure fidelity in a group-based parenting program. The Fidelity Checklist assesses group leaders' maintenance of the intervention protocol (adherence) and their group facilitation and process skills (competence).

OBJECTIVES

The purpose of this article is to describe the systematic process of modifying the Fidelity Checklist for use in a community-based opioid overdose prevention group educational intervention.

METHODS

A biphasic approach was used to modify the Fidelity Checklist. Phase 1 included engaging key stakeholders during a full-day meeting to determine adherence subscale components. Phase 2 included: (a) the modification of the competence subscale of the Fidelity Checklist, (b) adaptation of the coding manual defining the components of the Fidelity Checklist, and (c) creating a fidelity checklist to guide interventionists.

RESULTS

The biphasic systematic modification approach resulted in a revised Fidelity Checklist that was successfully used to measure treatment fidelity in a community-based opioid overdose prevention program.

CLINICAL RELEVANCE

The findings of this study provide information for nurse researchers and nurse educators for modifying a fidelity checklist that can be used to enhance community-based educational interventions.

An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention

The valid and reliable assessment of fidelity is critical at all stages of intervention research and is particularly germane to interpreting the results of efficacy and implementation trials. Ratings of protocol adherence typically are reliable, but ratings of therapist competence are plagued by low reliability. Because family context and case conceptualization guide the therapist's delivery of interventions, the reliability of fidelity ratings might be improved if the coder is privy to client context in the form of an ecological assessment. We conducted a randomized experiment to test this hypothesis. A subsample of 46 families with 5-year-old children from a multisite randomized trial who participated in the feedback session of the Family Check-Up (FCU) intervention were selected. We randomly assigned FCU feedback sessions to be rated for fidelity to the protocol using the COACH rating system either after the coder reviewed the results of a recent ecological assessment or had not. Inter-rater reliability estimates of fidelity ratings were meaningfully higher for the assessment information condition compared to the no-information condition. Importantly, the reliability of the COACH mean score was found to be statistically significantly higher in the information condition. These findings suggest that the reliability of observational ratings of fidelity, particularly when the competence or quality of delivery is considered, could be improved by providing assessment data to the coders. Our findings might be most applicable to assessment-driven interventions, where assessment data explicitly guides therapist's selection of intervention strategies tailored to the family's context and needs, but they could also apply to other intervention programs and observational coding of context-dependent therapy processes, such as the working alliance.

Development and Pilot Evaluation of a Tablet-Based Application to Improve Quality of Care in Child Mental Health Treatment

Background

Children need access to high quality mental health care. Effective treatments now exist for a wide range of mental health conditions. However, these interventions are delivered with variable effectiveness in traditional mental health service settings. Innovative solutions are needed to improve treatment delivery quality and effectiveness.

Objective

The aim of this study was to develop a scalable, sustainable technology-based approach to improve the quality of care in child mental health treatment.

Methods

A tablet-based resource was developed with input from mental health training experts, mental health providers, and patients. A series of qualitative data collection phases (ie, expert interviews, patient and provider focus groups, usability testing) guided the initial concept and design of the resource, and then its refinement. The result was an iPad-based “e-workbook” designed to improve child engagement and provider fidelity in implementation of a best-practice treatment. We are currently conducting a small scale randomized controlled trial to evaluate the feasibility of e-workbook facilitated child mental health treatment with 10 providers and 20 families recruited from 4 local community-based mental health clinics.

Results

Usability and focus group testing yielded a number of strong, favorable reactions from providers and families. Recommendations for refining the e-workbook also were provided, and these guided several improvements to the resource prior to initiating the feasibility trial, which is currently underway.

Conclusions

This study aimed to develop and preliminarily evaluate a tablet-based application to improve provider fidelity and child engagement in child mental health treatment. If successful, this approach may serve as a key step toward making best-practice treatment more accessible to children and families. As various technologies continue to increase in popularity worldwide and within the health care field more specifically, it is essential to rigorously test the usability, feasibility, acceptability, and effectiveness of novel health technology solutions. It is also essential to ensure that patients and providers drive decision making that supports the development of these resources to ensure that they can be seamlessly integrated into practice.

Trial Registration

Clinicaltrials.gov NCT01915160; https://clinicaltrials.gov/ct2/show/NCT01915160 (Archived by WebCite at http://www.webcitation.org/6cPIiQDpu)

A framework for testing and promoting expanded dissemination of promising preventive interventions that are being implemented in community settings

Many evidence-based preventive interventions have been developed in recent years, but few are widely used. With the current focus on efficacy trials, widespread dissemination and implementation of evidence-based interventions are often afterthoughts. One potential strategy for reversing this trend is to find a promising program with a strong delivery vehicle in place and improve and test the program's efficacy through rigorous evaluation. If the program is supported by evidence, the dissemination vehicle is already in place and potentially can be expanded. This strategy has been used infrequently and has met with limited success to date, in part, because the field lacks a framework for guiding such research. To address this gap, we outline a framework for moving promising preventive interventions that are currently being implemented in community settings through a process of rigorous testing and, if needed, program modification in order to promote expanded dissemination. The framework is guided by RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance) (Glasgow et al., Am J Publ Health 89:1322-1327, 1999), which focuses attention on external as well as internal validity in program tests, and is illustrated with examples. Challenges, such as responding to negative and null results, and opportunities inherent in the framework are discussed.

What is the extent and quality of documentation and reporting of fidelity to implementation strategies: a scoping review

BACKGROUND

Implementation fidelity is critical to the internal and external validity of implementation research. Much of what is written about implementation fidelity addresses fidelity of evidence-informed interventions rather than fidelity of implementation strategies. The documentation and reporting of fidelity to implementation strategies requires attention. Therefore, in this scoping review, we identify the extent and quality of documentation and reporting of fidelity of implementation strategies that were used to implement evidence-informed interventions.

METHODS

A six-stage methodological framework for scoping studies guided our work. Studies were identified from the outputs of the Effective Practice and Organization of Care (EPOC) review group within the Cochrane Database of Systematic Reviews. EPOC's primary focus, implementation strategies influencing provider behavior change, optimized our ability to identify articles for inclusion. We organized the retrieved articles from the systematic reviews by journal and selected the three journals with the largest number of retrieved articles. Using a data extraction tool, we organized retrieved article data from these three journals. In addition, we summarized implementation strategies using the EPOC categories. Data extraction pertaining to the quality of reporting the fidelity of implementation strategies was facilitated with an "Implementation Strategy Fidelity Checklist" based on definitions adapted from Dusenbury et al. We conducted inter-rater reliability checks for all of the independently scored articles. Using linear regression, we assessed the fidelity scores in relation to the publication year.

RESULTS

Seventy-two implementation articles were included in the final analysis. Researchers reported neither fidelity definitions nor conceptual frameworks for fidelity in any articles. The most frequently employed implementation strategies included distribution of education materials (n = 35), audit and feedback (n = 32), and educational meetings (n = 25). Fidelity of implementation strategies was documented in 51 (71 %) articles. Inter-rater reliability coefficients of the independent reviews for each component of fidelity were as follows: adherence = 0.85, dose = 0.89, and participant responsiveness = 0.96. The mean fidelity score was 2.6 (SD = 2.25). We noted a statistically significant decline in fidelity scores over time.

CONCLUSIONS

In addition to identifying the under-reporting of fidelity of implementation strategies in the health literature, we developed and tested a simple checklist to assess the reporting of fidelity of implementation strategies. More research is indicated to assess the definitions and scoring schema of this checklist. Careful reporting of details about fidelity of implementation strategies will make an important contribution to implementation science.

Development, inter-rater reliability and feasibility of a checklist to assess implementation (Ch-IMP) in systematic reviews: the case of provider-based prevention and treatment programs targeting children and youth

BACKGROUND

Several papers report deficiencies in the reporting of information about the implementation of interventions in clinical trials. Information about implementation is also required in systematic reviews of complex interventions to facilitate the translation and uptake of evidence of provider-based prevention and treatment programs. To capture whether and how implementation is assessed within systematic effectiveness reviews, we developed a checklist for implementation (Ch-IMP) and piloted it in a cohort of reviews on provider-based prevention and treatment interventions for children and young people. This paper reports on the inter-rater reliability, feasibility and reasons for discrepant ratings.

METHODS

Checklist domains were informed by a framework for program theory; items within domains were generated from a literature review. The checklist was pilot-tested on a cohort of 27 effectiveness reviews targeting children and youth. Two raters independently extracted information on 47 items. Inter-rater reliability was evaluated using percentage agreement and unweighted kappa coefficients. Reasons for discrepant ratings were content analysed.

RESULTS

Kappa coefficients ranged from 0.37 to 1.00 and were not influenced by one-sided bias. Most kappa values were classified as excellent (n = 20) or good (n = 17) with a few items categorised as fair (n = 7) or poor (n = 1). Prevalence-adjusted kappa coefficients indicate good or excellent agreement for all but one item. Four areas contributed to scoring discrepancies: 1) clarity or sufficiency of information provided in the review; 2) information missed in the review; 3) issues encountered with the tool; and 4) issues encountered at the review level. Use of the tool demands time investment and it requires adjustment to improve its feasibility for wider use.

CONCLUSIONS

The case of provider-based prevention and treatment interventions showed relevancy in developing and piloting the Ch-IMP as a useful tool for assessing the extent to which systematic reviews assess the quality of implementation. The checklist could be used by authors and editors to improve the quality of systematic reviews, and shows promise as a pedagogical tool to facilitate the extraction and reporting of implementation characteristics.

Observational measures of implementer fidelity for a school-based preventive intervention: development, reliability, and validity

Current measures of implementer fidelity often fail to adequately measure core constructs of adherence and competence, and their relationship to outcomes can be mixed. To address these limitations, we used observational methods to assess these constructs and their relationships to proximal outcomes in a randomized trial of a school-based preventive intervention (Rochester Resilience Project) designed to strengthen emotion self-regulation skills in first-third graders with elevated aggressive-disruptive behaviors. Within the intervention group (n = 203), a subsample (n = 76) of students was selected to reflect the overall sample. Implementers were 10 paraprofessionals. Videotaped observations of three lessons from year 1 of the intervention (14 lessons) were coded for each implementer-child dyad on adherence (content) and competence (quality). Using multilevel modeling, we examined how much of the variance in the fidelity measures was attributed to implementer and to the child within implementer. Both measures had large and significant variance accounted for by implementer (competence, 68 %; adherence, 41 %); child within implementer did not account for significant variance indicating that ratings reflected stable qualities of the implementer rather than the child. Raw adherence and competence scores shared 46 % of variance (r = .68). Controlling for baseline differences and age, the amount (adherence) and quality (competence) of program delivered predicted children's enhanced response to the intervention on both child and parent reports after 6 months, but not on teacher report of externalizing behavior. Our findings support the use of multiple observations for measuring fidelity and that adherence and competence are important components of fidelity which could be assessed by many programs using these methods.

Implementation assessment of widely used but understudied prevention programs: an illustration from the Common Sense Parenting trial

Common Sense Parenting is a parent-training program that is widely disseminated, has promising preliminary support, and is being tested in a randomized controlled trial that targets lower-income, urban 8th-grade students and their families (recruited in two annual cohorts) to improve the transition to high school. The workshop-based program is being tested in both standard 6-session (CSP) and modified 8-session (CSP Plus) formats; CSP Plus adds adolescent-skills training activities. To offer a comprehensive picture of implementation outcomes in the CSP trial, we describe the tools used to assess program adherence, quality of delivery, program dosage, and participant satisfaction, and report the implementation data collected during the trial. Results indicated that workshop leaders had high adherence to the program content and manual-stated goal times of the CSP/CSP Plus curriculum and delivered the intervention with high quality. The majority of intervention families attended some or all of the sessions. Participant satisfaction ratings for the workshops were high. There were no significant cohort differences for adherence, quality and dosage; however, there were significant cohort improvements for participant satisfaction. Positive fidelity results may be due to the availability of detailed workshop leader guides, in addition to ongoing training and supervision, which included performance-based feedback.

Tailored Implementation for Chronic Diseases (TICD): a protocol for process evaluation in cluster randomized controlled trials in five European countries

Background

In the ‘Tailored Implementation for Chronic Diseases (TICD)’ project, five tailored implementation programs to improve healthcare delivery in different chronic conditions have been developed. These programs will be evaluated in distinct cluster-randomized controlled trials. This protocol describes the process evaluation across these trials, which aims to identify determinants of change in chronic illness care, to examine the validity of the tailoring methods that were applied, and to analyze the association of implementation activities and the effectiveness of the program.

Methods

A multilevel approach was used to develop five tailored implementation interventions. In order to guide the process evaluation in five distinct trials, the study protocols for the cluster randomized trials and the related process evaluations were developed simultaneously and iteratively.

Results

The process evaluation comprises three main components: a structured survey with health professionals in the trials, semi-structured interviews with a purposeful sample of this study population, and standardized documentation of organizational practice characteristics. Norway will only conduct the qualitative part of the analysis because the survey and documentation of practice characteristics are considered to be not feasible. The evaluation is guided by ‘logic models’ of the implementation programs: frameworks that specify the linkages between the strategies used, the determinants addressed by tailoring, and the anticipated outcomes. Standardization of measures across trials is sought to facilitate analysis of aggregated data from the trials.

Conclusions

This process evaluation will need to find a balance between standardization of methods across trials and the tailoring of measures to the specificities of each trial.

Fidelity in complex behaviour change interventions: a standardised approach to evaluate intervention integrity

OBJECTIVES

The aim of this study was to (1) demonstrate the development and testing of tools and procedures designed to monitor and assess the integrity of a complex intervention for chronic pain (COping with persistent Pain, Effectiveness Research into Self-management (COPERS) course); and (2) make recommendations based on our experiences.

DESIGN

Fidelity assessment of a two-arm randomised controlled trial intervention, assessing the adherence and competence of the facilitators delivering the intervention.

SETTING

The intervention was delivered in the community in two centres in the UK: one inner city and one a mix of rural and urban locations.

PARTICIPANTS

403 people with chronic musculoskeletal pain were enrolled in the intervention arm and 300 attended the self-management course. Thirty lay and healthcare professionals were trained and 24 delivered the courses (2 per course). We ran 31 courses for up to 16 people per course and all were audio recorded.

INTERVENTIONS

The course was run over three and a half days; facilitators delivered a semistructured manualised course.

OUTCOMES

We designed three measures to evaluate fidelity assessing adherence to the manual, competence and overall impression.

RESULTS

We evaluated a random sample of four components from each course (n=122). The evaluation forms were reliable and had good face validity. There were high levels of adherence in the delivery: overall adherence was two (maximum 2, IQR 1.67-2.00), facilitator competence exhibited more variability, and overall competence was 1.5 (maximum 2, IQR 1.25-2.00). Overall impression was three (maximum 4, IQR 2.00-3.00).

CONCLUSIONS

Monitoring and assessing adherence and competence at the point of intervention delivery can be realised most efficiently by embedding the principles of fidelity measurement within the design stage of complex interventions and the training and assessment of those delivering the intervention. More work is necessary to ensure that more robust systems of fidelity evaluation accompany the growth of complex interventions.

TRIAL REGISTRATION ISRCTN NO

ISRCTN24426731.

Intermediate outcomes of a chronic disease self-management program for Spanish-speaking older adults in South Florida, 2008-2010

Introduction

The prevalence and negative health effects of chronic diseases are disproportionately high among Hispanics, the largest minority group in the United States. Self-management of chronic conditions by older adults is a public health priority. The objective of this study was to examine 6-week differences in self-efficacy, time spent performing physical activity, and perceived social and role activities limitations for participants in a chronic disease self-management program for Spanish-speaking older adults, Tomando Control de su Salud (TCDS).

Methods

Through the Healthy Aging Regional Collaborative, 8 area agencies delivered 82 workshops in 62 locations throughout South Florida. Spanish-speaking participants who attended workshops from October 1, 2008, through December 31, 2010, were aged 55 years or older, had at least 1 chronic condition, and completed baseline and post-test surveys were included in analysis (N = 682). Workshops consisted of six, 2.5-hour sessions offered once per week for 6 weeks. A self-report survey was administered at baseline and again at the end of program instruction. To assess differences in outcomes, a repeated measures general linear model was used, controlling for agency and baseline general health.

Results

All outcomes showed improvement at 6 weeks. Outcomes that improved significantly were self-efficacy to manage disease, perceived social and role activities limitations, time spent walking, and time spent performing other aerobic activities.

Conclusion

Implementation of TCDS significantly improved 4 of 8 health promotion skills and behaviors of Spanish-speaking older adults in South Florida. A community-based implementation of TCDS has the potential to improve health outcomes for a diverse, Spanish-speaking, older adult population.

Affect regulation training (ART) for alcohol use disorders: development of a novel intervention for negative affect drinkers

Although negative affect is a common precipitant of alcohol relapse, there are few interventions for alcohol dependence that specifically target negative affect. In this stage 1a/1b treatment development study, several affect regulation strategies (e.g., mindfulness, prolonged exposure, distress tolerance) were combined to create a new treatment supplement called affect regulation training (ART), which could be added to enhance cognitive-behavioral therapy (CBT) for alcohol dependence. A draft therapy manual was given to therapists and treatment experts before being administered to several patients who also provided input. After two rounds of manual development (stage 1a), a pilot randomized clinical trial (N=77) of alcohol-dependent outpatients who reported drinking often in negative affect situations was conducted (stage 1b). Participants received 12-weekly, 90-minute sessions of either CBT for alcohol dependence plus ART (CBT+ART) or CBT plus a healthy lifestyles control condition (CBT+HLS). Baseline, end-of-treatment, and 3- and 6-month posttreatment interviews were conducted. For both treatment conditions, participant ratings of treatment satisfaction were high, with CBT+ART rated significantly higher. Drinking outcome results indicated greater reductions in alcohol use for CBT+ART when compared to CBT+HLS, with moderate effect sizes for percent days abstinent, drinks per day, drinks per drinking day, and percent heavy drinking days. Overall, findings support further research on affect regulation interventions for negative affect drinkers.

Promoting nurse interventionist fidelity to motivational interviewing in a diabetes self-care intervention

We describe interventionist training procedures for a pilot intervention study that tested the effects of a nurse-delivered Motivational Interviewing (MI) intervention on diabetes self-care among adults diagnosed with type 2 diabetes mellitus. It reports on findings from MI fidelity assessments. Training consisted of didactic workshops plus practicum. Fidelity of MI was assessed using the MI Treatment Integrity (MITI) Scale. Fidelity assessments were conducted on 18 (25%) audiotaped MI sessions, which were randomly selected from a total of 72 sessions with 26 participants. Scores of the MITI were in the proficient and competent range. Results suggest that training strategies were sufficient to promote satisfactory interventionist fidelity to MI.