The case for prioritizing implementation strategy fidelity measurement: benefits and challenges

Implementation of a crisis resolution team service improvement programme: a qualitative study of the critical ingredients for success

BACKGROUND

Crisis Resolution Teams (CRTs) offer home-based care for people in mental health crisis, as an alternative to hospital admission. The success of CRTs in England has been variable. In response to this, the CRT Optimization and RElapse prevention (CORE) study developed and trialled a 12-month Service Improvement Programme (SIP) based on a fidelity model. This paper describes a qualitative evaluation of the perspectives of CRT staff, managers, and programme facilitators. We identify barriers and facilitators to implementation, and mechanisms by which service improvements took place.

METHODS

Managers and staff from six purposively sampled CRTs were interviewed, as well as six facilitators who were employed to support the implementation of service improvement plans. Semi-structured focus groups and individual interviews were conducted and analysed using thematic analysis.

FINDINGS

A majority of participants viewed all components of the SIP as helpful in improving practice, although online resources were under-used. Perceived barriers to implementation centred principally around lack of staff time and ownership. Support from both senior staff and facilitators was essential in enabling teams to undertake the work associated with the SIP. All participating stakeholder groups reported that using the fidelity model to benchmark their CRT work to best practice and feel part of a 'bigger whole' was valuable.

CONCLUSION

CRT staff, managers and programme facilitators thought that a structured service improvement programme helped to increase fidelity to a best practice model. Flexibility (from all stakeholders) was key to enable service improvement actions to be manageable within time- and resource-poor teams.

An implementation strategy postmortem method developed in the VA rural Transitions Nurse Program to inform spread and scale-up

BACKGROUND

High-quality implementation evaluations report on intervention fidelity and adaptations made, but a practical process for evaluating implementation strategies is needed. A retrospective method for evaluating implementation strategies is also required as prospective methods can be resource intensive. This study aimed to establish an implementation strategy postmortem method to identify the implementation strategies used, when, and their perceived importance. We used the rural Transitions Nurse Program (TNP) as a case study, a national care coordination intervention implemented at 11 hospitals over three years.

METHODS

The postmortem used a retrospective, mixed method, phased approach. Implementation team and front-line staff characterized the implementation strategies used, their timing, frequency, ease of use, and their importance to implementation success. The Expert Recommendations for Implementing Change (ERIC) compilation, the Quality Enhancement Research Initiative phases, and Proctor and colleagues' guidance were used to operationalize the strategies. Survey data were analyzed descriptively, and qualitative data were analyzed using matrix content analysis.

RESULTS

The postmortem method identified 45 of 73 ERIC strategies introduced, including 41 during pre-implementation, 37 during implementation, and 27 during sustainment. External facilitation, centralized technical assistance, and clinical supervision were ranked as the most important and frequently used strategies. Implementation strategies were more intensively applied in the beginning of the study and tapered over time.

CONCLUSIONS

The postmortem method identified that more strategies were used in TNP than planned and identified the most important strategies from the perspective of the implementation team and front-line staff. The findings can inform other implementation studies as well as dissemination of the TNP intervention.

Developing the Workforce of the Digital Future: mHealth Competency and Fidelity Measurement in Community-Based Care

Integrating mobile health (mHealth) interventions into settings that serve diverse patient populations requires that prerequisite professional competencies are delineated and that standards for clinical quality assurance can be pragmatically assessed. Heretofore, proposed mHealth competencies have been broad and have lacked a framework to support specific applications. We outline the meta-competencies identified in the literature relevant to mHealth interventions and demonstrate how these meta-competencies can be integrated with population- and intervention-related competencies to help guide a pragmatic approach to competency assessment. We present a use case based on FOCUS-an evidence-based mHealth intervention designed for individuals with serious mental illness and currently being implemented in geographically and demographically diverse community behavioral health settings. Subsequent to identifying the cross-cutting competencies relevant to the target population (outpatients experiencing psychotic symptoms), substratal intervention (Cognitive Behavioral Therapy for psychosis), and treatment modality (mHealth), we detail the development process of an mHealth fidelity monitoring system (mHealth-FMS). We adhered to a published sequential 5-step process to design a fidelity monitoring system that aligns with our integrated mHealth competency framework and that was guided by best practices prescribed by the Treatment Fidelity Workgroup of the National Institutes of Health Behavior Change Consortium. The mHealth-FMS is intended to enhance both clinical and implementation outcomes by grounding the mHealth interventionist and the system of care in which they operate in the core functions, tasks, knowledge, and competencies associated with system-integrated mHealth delivery. Future research will explore acceptability and feasibility of the mHealth-FMS.

Implementing Dementia Caregiver Programs in Real-World Settings: Fidelity Considerations

Testing interventions in real-world settings requires fidelity monitoring to ensure implementation integrity. However, strategies to enhance, monitor, and measure fidelity deployed in efficacy trials may not be feasible in pragmatic trials or sustainable in practice. This paper reviews published translational or pragmatic studies of dementia caregiver support interventions to understand how fidelity was previously treated in order to derive recommendations for future pragmatic-like trials. A search using SCOPUS, EMBASE, and Google Scholar identified 31 translational caregiver intervention studies of which 20 (64.5%) referenced fidelity. Of these 20, 11 (55.0%) reported fidelity measurement, whereas 9 (45.0%) only recognized its importance. Of the 11 studies, fidelity was assessed using investigator-developed scoring forms, audio/video recordings, evaluations from caregivers and interventionists, and by comparing outcomes with the original efficacy trial. Additionally, 7 (63.6%) of 11 studies reported fidelity results, representing only 22.5% of 31 studies reporting outcomes demonstrating the inconsistency in the field concerning the reporting outcomes of fidelity. We conclude that fidelity methods used in translational studies to date are not practical nor sustainable for ongoing monitoring of evidence-based programs in real-world settings and that only 2 aspects of fidelity, intervention and adherence, are considered. New approaches are needed to ensure fidelity integrity in pragmatic trials and which can be sustained thereafter.

Application of the Expert Recommendations for Implementing Change (ERIC) compilation of strategies to health intervention implementation in low- and middle-income countries: a systematic review

BACKGROUND

The Expert Recommendations for Implementing Change (ERIC) project developed a compilation of implementation strategies that are intended to standardize reporting and evaluation. Little is known about the application of ERIC in low- and middle-income countries (LMICs). We systematically reviewed the literature on the use and specification of ERIC strategies for health intervention implementation in LMICs to identify gaps and inform future research.

METHODS

We searched peer-reviewed articles published through March 2023 in any language that (1) were conducted in an LMIC and (2) cited seminal ERIC articles or (3) mentioned ERIC in the title or abstract. Two co-authors independently screened all titles, abstracts, and full-text articles, then abstracted study, intervention, and implementation strategy characteristics of included studies.

RESULTS

The final sample included 60 studies describing research from all world regions, with over 30% published in the final year of our review period. Most studies took place in healthcare settings (n = 52, 86.7%), while 11 (18.2%) took place in community settings and four (6.7%) at the policy level. Across studies, 548 distinct implementation strategies were identified with a median of six strategies (range 1-46 strategies) included in each study. Most studies (n = 32, 53.3%) explicitly matched implementation strategies used for the ERIC compilation. Among those that did, 64 (87.3%) of the 73 ERIC strategies were represented. Many of the strategies not cited included those that target systems- or policy-level barriers. Nearly 85% of strategies included some component of strategy specification, though most only included specification of their action (75.2%), actor (57.3%), and action target (60.8%). A minority of studies employed randomized trials or high-quality quasi-experimental designs; only one study evaluated implementation strategy effectiveness.

CONCLUSIONS

While ERIC use in LMICs is rapidly growing, its application has not been consistent nor commonly used to test strategy effectiveness. Research in LMICs must better specify strategies and evaluate their impact on outcomes. Moreover, strategies that are tested need to be better specified, so they may be compared across contexts. Finally, strategies targeting policy-, systems-, and community-level determinants should be further explored.

TRIAL REGISTRATION

PROSPERO, CRD42021268374.

Fidelity to the ACT SMART Toolkit: an instrumental case study of implementation strategy fidelity

BACKGROUND

Evidence-based practices (EBPs) are shown to improve a variety of outcomes for autistic children. However, EBPs often are mis-implemented or not implemented in community-based settings where many autistic children receive usual care services. A blended implementation process and capacity-building implementation strategy, developed to facilitate the adoption and implementation of EBPs for autism spectrum disorder (ASD) in community-based settings, is the Autism Community Toolkit: Systems to Measure and Adopt Research-based Treatments (ACT SMART Toolkit). Based on an adapted Exploration, Adoption decision, Preparation, Implementation, Sustainment (EPIS) Framework, the multi-phased ACT SMART Toolkit is comprised of (a) implementation facilitation, (b) agency-based implementation teams, and (c) a web-based interface. In this instrumental case study, we developed and utilized a method to evaluate fidelity to the ACT SMART Toolkit. This study responds to the need for implementation strategy fidelity evaluation methods and may provide evidence supporting the use of the ACT SMART Toolkit.

METHODS

We used an instrumental case study approach to assess fidelity to the ACT SMART Toolkit during its pilot study with six ASD community agencies located in southern California. We assessed adherence, dose, and implementation team responsiveness for each phase and activity of the toolkit at both an aggregate and individual agency level.

RESULTS

Overall, we found that adherence, dose, and implementation team responsiveness to the ACT SMART Toolkit were high, with some variability by EPIS phase and specific activity as well as by ASD community agency. At the aggregate level, adherence and dose were rated notably lowest during the preparation phase of the toolkit, which is a more activity-intensive phase of the toolkit.

CONCLUSIONS

This evaluation of fidelity to the ACT SMART Toolkit, utilizing an instrumental case study design, demonstrated the potential for the strategy to be used with fidelity in ASD community-based agencies. Findings related to the variability of implementation strategy fidelity in the present study may also inform future adaptations to the toolkit and point to broader trends of how implementation strategy fidelity may vary by content and context.

"We start where we are": a qualitative study of barriers and pragmatic solutions to the assessment and reporting of implementation strategy fidelity

BACKGROUND

Fidelity measurement of implementation strategies is underdeveloped and underreported, and the level of reporting is decreasing over time. Failing to properly measure the factors that affect the delivery of an implementation strategy may obscure the link between a strategy and its outcomes. Barriers to assessing and reporting implementation strategy fidelity among researchers are not well understood. The aims of this qualitative study were to identify barriers to fidelity measurement and pragmatic pathways towards improvement.

METHODS

We conducted in-depth interviews among researchers conducting implementation trials. We utilized a theory-informed interview approach to elicit the barriers and possible solutions to implementation strategy fidelity assessment and reporting. Reflexive-thematic analysis guided coding and memo-writing to determine key themes regarding barriers and solutions.

RESULTS

Twenty-two implementation researchers were interviewed. Participants agreed that implementation strategy fidelity was an essential element of implementation trials and that its assessment and reporting should improve. Key thematic barriers focused on (1) a current lack of validated fidelity tools with the need to assess fidelity in the short term, (2) the complex nature of some implementation strategies, (3) conceptual complications when assessing fidelity within mechanisms-focused implementation research, and (4) structural issues related to funding and publishing. Researchers also suggested pragmatic solutions to overcome each barrier. Respondents reported using specification and tracking data in the short term until validated tools become available. Participants suggested that researchers with strategy-specific content expertise lead the way in identifying core components and setting fidelity requirements for them. Addressing the third barrier, participants provided examples of what pragmatic prospective and retrospective fidelity assessments might look like along a mechanistic pathway. Finally, researchers described approaches to minimize costs of data collection, as well as more structural accountability like adopting and enforcing reporting guidelines or changing the structure of funding opportunities.

DISCUSSION

We propose short- and long-term priorities for improving the assessment and reporting of implementation strategy fidelity and the quality of implementation research.

CONCLUSIONS

A better understanding of the barriers to implementation strategy fidelity assessment may pave the way towards pragmatic solutions.

Examining the complementarity between the ERIC compilation of implementation strategies and the behaviour change technique taxonomy: a qualitative analysis

BACKGROUND

Efforts to generate evidence for implementation strategies are frustrated by insufficient description. The Expert Recommendations for Implementing Change (ERIC) compilation names and defines implementation strategies; however, further work is needed to describe the actions involved. One potentially complementary taxonomy is the behaviour change techniques (BCT) taxonomy. We aimed to examine the extent and nature of the overlap between these taxonomies.

METHODS

Definitions and descriptions of 73 strategies in the ERIC compilation were analysed. First, each description was deductively coded using the BCT taxonomy. Second, a typology was developed to categorise the extent of overlap between ERIC strategies and BCTs. Third, three implementation scientists independently rated their level of agreement with the categorisation and BCT coding. Finally, discrepancies were settled through online consensus discussions. Additional patterns of complementarity between ERIC strategies and BCTs were labelled thematically. Descriptive statistics summarise the frequency of coded BCTs and the number of strategies mapped to each of the categories of the typology.

RESULTS

Across the 73 strategies, 41/93 BCTs (44%) were coded, with 'restructuring the social environment' as the most frequently coded (n=18 strategies, 25%). There was direct overlap between one strategy (change physical structure and equipment) and one BCT ('restructuring physical environment'). Most strategy descriptions (n=64) had BCTs that were clearly indicated (n=18), and others where BCTs were probable but not explicitly described (n=31) or indicated multiple types of overlap (n=15). For some strategies, the presence of additional BCTs was dependent on the form of delivery. Some strategies served as examples of broad BCTs operationalised for implementation. For eight strategies, there were no BCTs indicated, or they did not appear to focus on changing behaviour. These strategies reflected preparatory stages and targeted collective cognition at the system level rather than behaviour change at the service delivery level.

CONCLUSIONS

This study demonstrates how the ERIC compilation and BCT taxonomy can be integrated to specify active ingredients, providing an opportunity to better understand mechanisms of action. Our results highlight complementarity rather than redundancy. More efforts to integrate these or other taxonomies will aid strategy developers and build links between existing silos in implementation science.