The case for prioritizing implementation strategy fidelity measurement: benefits and challenges

A systems analysis and improvement approach to optimizing syringe services programs' delivery of HIV testing and referrals: Study protocol for a parallel-group randomized controlled trial (SAIA-SSP-HIV)

With changing drug supplies and associated drug consumption behaviors, HIV transmission has increased among people who inject drugs in the United States. HIV testing and referrals to effective prevention and treatment services are critical for individual and population health, yet multilevel barriers limit access to HIV testing for this population, even within syringe services programs (SSPs). In this organizational-level interrupted time series randomized controlled trial, we will assess the effectiveness and cost-effectiveness of an implementation strategy, the Systems Analysis and Improvement Approach (SAIA), in optimizing HIV testing and referrals to appropriate clinical services among U.S. SSPs. From 01/12/2023 to 01/07/2025, we will recruit a diverse sample of 32 SSPs nationally that directly provide HIV testing to participants. SSPs will be randomized to the active implementation arm (i.e., SAIA-SSP-HIV) or an implementation-as-usual arm (n = 16 organizations per arm). SAIA-SSP-HIV is a flexible, data-driven implementation strategy designed to help optimize SSPs' delivery of HIV testing and referrals to appropriate clinical services for HIV prevention (e.g., pre-exposure prophylaxis) and treatment. In the active implementation arm, trained SAIA specialists will guide SSPs through three cyclical steps over 12 months: (1) process mapping to identify organization-specific needs, (2) cascade analysis and prioritization of areas for improvement, and (3) testing solutions through continuous quality improvement. In both arms, we will collect outcome data over 21 months (3-month lead-in period, 12-month implementation period, 6-month sustainment period). We will assess the initial and sustained effectiveness of SAIA and calculate its cost and cost-effectiveness. This trial presents a novel opportunity to test the effectiveness of an organization-level implementation strategy for optimizing the delivery of HIV screening and referrals in community settings that are frequented by an at-risk population. If successful, SAIA-SSP-HIV could be adapted for other infectious or chronic disease care cascades within SSPs. Trial registration: ClinicalTrials.gov: NCT06025435.

Firm, yet flexible: a fidelity debate paper with two case examples

BACKGROUND

In healthcare research and practice, intervention and implementation fidelity represent the steadfast adherence to core components of research-supported interventions and the strategies employed for their implementation. Evaluating fidelity involves determining whether these core components were delivered as intended. Without fidelity data, the results of complex interventions cannot be meaningfully interpreted. Increasingly, the necessity for firmness and strict adherence by implementers and their organizations has been questioned, with calls for flexibility to accommodate contextual conditions. This shift makes contemporary fidelity a balancing act, requiring researchers to navigate various tensions. This debate paper explores these tensions, drawing on experiences from developing fidelity assessments in two ongoing effectiveness-implementation hybrid trials.

MAIN BODY

First, given often scarce knowledge about the core components of complex interventions and implementation strategies, decisions about fidelity requirements involve a degree of subjective reasoning. Researchers should make these decisions transparent using theory or logic models. Second, because fidelity is context-dependent and applies to both interventions and implementation strategies, researchers must rethink fidelity concepts with every study while balancing firmness and flexibility. This is particularly crucial for hybrid studies, with their differing emphasis on intervention and implementation fidelity. Third, fidelity concepts typically focus on individual behaviors. However, since organizational and system factors also influence fidelity, there is a growing need to define fidelity criteria at these levels. Finally, as contemporary fidelity concepts prioritize flexible over firm adherence, building, evaluating, and maintaining fidelity in healthcare research has become more complex. This complexity calls for intensified efforts to expand the knowledge base for pragmatic and adaptive fidelity measurement in trial and routine healthcare settings.

CONCLUSION

Contemporary conceptualizations of fidelity place greater demands on how fidelity is examined, necessitating the expansion of fidelity frameworks to include organizational and system levels, the service- and study-specific conceptualizations of intervention and implementation fidelity, and the development of pragmatic approaches for assessing fidelity in research and practice. Continuing to build knowledge on how to balance requirements for firmness and flexibility remains a crucial task within the field of implementation science.

A research agenda to advance the study of implementation mechanisms

BACKGROUND

Implementation science scholars have made significant progress identifying factors that enable or obstruct the implementation of evidence-based interventions, and testing strategies that may modify those factors. However, little research sheds light on how or why strategies work, in what contexts, and for whom. Studying implementation mechanisms-the processes responsible for change-is crucial for advancing the field of implementation science and enhancing its value in facilitating equitable policy and practice change. The Agency for Healthcare Research and Quality funded a conference series to achieve two aims: (1) develop a research agenda on implementation mechanisms, and (2) actively disseminate the research agenda to research, policy, and practice audiences. This article presents the resulting research agenda, including priorities and actions to encourage its execution.

METHOD

Building on prior concept mapping work, in a semi-structured, 3-day, in-person working meeting, 23 US-based researchers used a modified nominal group process to generate priorities and actions for addressing challenges to studying implementation mechanisms. During each of the three 120-min sessions, small groups responded to the prompt: "What actions need to be taken to move this research forward?" The groups brainstormed actions, which were then shared with the full group and discussed with the support of facilitators trained in structured group processes. Facilitators grouped critical and novel ideas into themes. Attendees voted on six themes they prioritized to discuss in a fourth, 120-min session, during which small groups operationalized prioritized actions. Subsequently, all ideas were collated, combined, and revised for clarity by a subset of the authorship team.

RESULTS

From this multistep process, 150 actions emerged across 10 priority areas, which together constitute the research agenda. Actions included discrete activities, projects, or products, and ways to shift how research is conducted to strengthen the study of implementation mechanisms.

CONCLUSIONS

This research agenda elevates actions to guide the selection, design, and evaluation of implementation mechanisms. By delineating recommended actions to address the challenges of studying implementation mechanisms, this research agenda facilitates expanding the field of implementation science, beyond studying what works to how and why strategies work, in what contexts, for whom, and with which interventions.

A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022

BACKGROUND

Studies of implementation strategies range in rigor, design, and evaluated outcomes, presenting interpretation challenges for practitioners and researchers. This systematic review aimed to describe the body of research evidence testing implementation strategies across diverse settings and domains, using the Expert Recommendations for Implementing Change (ERIC) taxonomy to classify strategies and the Reach Effectiveness Adoption Implementation and Maintenance (RE-AIM) framework to classify outcomes.

METHODS

We conducted a systematic review of studies examining implementation strategies from 2010-2022 and registered with PROSPERO (CRD42021235592). We searched databases using terms "implementation strategy", "intervention", "bundle", "support", and their variants. We also solicited study recommendations from implementation science experts and mined existing systematic reviews. We included studies that quantitatively assessed the impact of at least one implementation strategy to improve health or health care using an outcome that could be mapped to the five evaluation dimensions of RE-AIM. Only studies meeting prespecified methodologic standards were included. We described the characteristics of studies and frequency of implementation strategy use across study arms. We also examined common strategy pairings and cooccurrence with significant outcomes.

FINDINGS

Our search resulted in 16,605 studies; 129 met inclusion criteria. Studies tested an average of 6.73 strategies (0-20 range). The most assessed outcomes were Effectiveness (n=82; 64%) and Implementation (n=73; 56%). The implementation strategies most frequently occurring in the experimental arm were Distribute Educational Materials (n=99), Conduct Educational Meetings (n=96), Audit and Provide Feedback (n=76), and External Facilitation (n=59). These strategies were often used in combination. Nineteen implementation strategies were frequently tested and associated with significantly improved outcomes. However, many strategies were not tested sufficiently to draw conclusions.

CONCLUSION

This review of 129 methodologically rigorous studies built upon prior implementation science data syntheses to identify implementation strategies that had been experimentally tested and summarized their impact on outcomes across diverse outcomes and clinical settings. We present recommendations for improving future similar efforts.

Implementation of a crisis resolution team service improvement programme: a qualitative study of the critical ingredients for success

BACKGROUND

Crisis Resolution Teams (CRTs) offer home-based care for people in mental health crisis, as an alternative to hospital admission. The success of CRTs in England has been variable. In response to this, the CRT Optimization and RElapse prevention (CORE) study developed and trialled a 12-month Service Improvement Programme (SIP) based on a fidelity model. This paper describes a qualitative evaluation of the perspectives of CRT staff, managers, and programme facilitators. We identify barriers and facilitators to implementation, and mechanisms by which service improvements took place.

METHODS

Managers and staff from six purposively sampled CRTs were interviewed, as well as six facilitators who were employed to support the implementation of service improvement plans. Semi-structured focus groups and individual interviews were conducted and analysed using thematic analysis.

FINDINGS

A majority of participants viewed all components of the SIP as helpful in improving practice, although online resources were under-used. Perceived barriers to implementation centred principally around lack of staff time and ownership. Support from both senior staff and facilitators was essential in enabling teams to undertake the work associated with the SIP. All participating stakeholder groups reported that using the fidelity model to benchmark their CRT work to best practice and feel part of a 'bigger whole' was valuable.

CONCLUSION

CRT staff, managers and programme facilitators thought that a structured service improvement programme helped to increase fidelity to a best practice model. Flexibility (from all stakeholders) was key to enable service improvement actions to be manageable within time- and resource-poor teams.

Matters of Fidelity: School Provider Adherence and Competence in a Clustered Study of Adaptive Implementation Strategies

Schools are a promising access point for youth with mental health concerns, but school-based mental health professionals (SPs) often need ongoing support to provide high-fidelity cognitive behavioral therapy (CBT). Adherence and competence, two critical elements of fidelity, were examined in a cluster-randomized implementation trial. We evaluated CBT adherence and then triangulated CBT adherence with end-of-study competence. We then evaluated the effects of two implementation supports, Coaching and (for slower-responding schools) Facilitation, on adherence and competence. By the end of the 43-week study period, 27.8% of SPs met adherence criteria. Adherent SPs scored higher on the competence measure, the CBT Competence Scale (t (116.2) = 3.71, p < .001). No significant difference in adherence was found among SPs at schools assigned to Coaching vs. not (Δ = 6.0%, p = .385), however SPs at schools randomized to Coaching scored significantly higher on two of the four competence subscales (Non-Behavioral and Behavioral skills). Among slower-responder schools, SPs at schools assigned to Facilitation were more likely to demonstrate adherence (Δ = 16.3%, p = .022), but there was no effect of Facilitation on competence. Approximately one quarter of SPs met adherence criteria in the trial; adequate delivery of exposure was a primary obstacle to reaching adherence. Facilitation may be especially suited to help SPs overcome barriers to delivery, whereas Coaching may be especially suited to help SPs improve CBT competence. Both are likely needed to build a mental health work force with the competence and ability to deliver EBPs in schools.

An implementation strategy postmortem method developed in the VA rural Transitions Nurse Program to inform spread and scale-up

BACKGROUND

High-quality implementation evaluations report on intervention fidelity and adaptations made, but a practical process for evaluating implementation strategies is needed. A retrospective method for evaluating implementation strategies is also required as prospective methods can be resource intensive. This study aimed to establish an implementation strategy postmortem method to identify the implementation strategies used, when, and their perceived importance. We used the rural Transitions Nurse Program (TNP) as a case study, a national care coordination intervention implemented at 11 hospitals over three years.

METHODS

The postmortem used a retrospective, mixed method, phased approach. Implementation team and front-line staff characterized the implementation strategies used, their timing, frequency, ease of use, and their importance to implementation success. The Expert Recommendations for Implementing Change (ERIC) compilation, the Quality Enhancement Research Initiative phases, and Proctor and colleagues' guidance were used to operationalize the strategies. Survey data were analyzed descriptively, and qualitative data were analyzed using matrix content analysis.

RESULTS

The postmortem method identified 45 of 73 ERIC strategies introduced, including 41 during pre-implementation, 37 during implementation, and 27 during sustainment. External facilitation, centralized technical assistance, and clinical supervision were ranked as the most important and frequently used strategies. Implementation strategies were more intensively applied in the beginning of the study and tapered over time.

CONCLUSIONS

The postmortem method identified that more strategies were used in TNP than planned and identified the most important strategies from the perspective of the implementation team and front-line staff. The findings can inform other implementation studies as well as dissemination of the TNP intervention.

Developing the Workforce of the Digital Future: mHealth Competency and Fidelity Measurement in Community-Based Care

Integrating mobile health (mHealth) interventions into settings that serve diverse patient populations requires that prerequisite professional competencies are delineated and that standards for clinical quality assurance can be pragmatically assessed. Heretofore, proposed mHealth competencies have been broad and have lacked a framework to support specific applications. We outline the meta-competencies identified in the literature relevant to mHealth interventions and demonstrate how these meta-competencies can be integrated with population- and intervention-related competencies to help guide a pragmatic approach to competency assessment. We present a use case based on FOCUS-an evidence-based mHealth intervention designed for individuals with serious mental illness and currently being implemented in geographically and demographically diverse community behavioral health settings. Subsequent to identifying the cross-cutting competencies relevant to the target population (outpatients experiencing psychotic symptoms), substratal intervention (Cognitive Behavioral Therapy for psychosis), and treatment modality (mHealth), we detail the development process of an mHealth fidelity monitoring system (mHealth-FMS). We adhered to a published sequential 5-step process to design a fidelity monitoring system that aligns with our integrated mHealth competency framework and that was guided by best practices prescribed by the Treatment Fidelity Workgroup of the National Institutes of Health Behavior Change Consortium. The mHealth-FMS is intended to enhance both clinical and implementation outcomes by grounding the mHealth interventionist and the system of care in which they operate in the core functions, tasks, knowledge, and competencies associated with system-integrated mHealth delivery. Future research will explore acceptability and feasibility of the mHealth-FMS.

Implementing Dementia Caregiver Programs in Real-World Settings: Fidelity Considerations

Testing interventions in real-world settings requires fidelity monitoring to ensure implementation integrity. However, strategies to enhance, monitor, and measure fidelity deployed in efficacy trials may not be feasible in pragmatic trials or sustainable in practice. This paper reviews published translational or pragmatic studies of dementia caregiver support interventions to understand how fidelity was previously treated in order to derive recommendations for future pragmatic-like trials. A search using SCOPUS, EMBASE, and Google Scholar identified 31 translational caregiver intervention studies of which 20 (64.5%) referenced fidelity. Of these 20, 11 (55.0%) reported fidelity measurement, whereas 9 (45.0%) only recognized its importance. Of the 11 studies, fidelity was assessed using investigator-developed scoring forms, audio/video recordings, evaluations from caregivers and interventionists, and by comparing outcomes with the original efficacy trial. Additionally, 7 (63.6%) of 11 studies reported fidelity results, representing only 22.5% of 31 studies reporting outcomes demonstrating the inconsistency in the field concerning the reporting outcomes of fidelity. We conclude that fidelity methods used in translational studies to date are not practical nor sustainable for ongoing monitoring of evidence-based programs in real-world settings and that only 2 aspects of fidelity, intervention and adherence, are considered. New approaches are needed to ensure fidelity integrity in pragmatic trials and which can be sustained thereafter.

Application of the Expert Recommendations for Implementing Change (ERIC) compilation of strategies to health intervention implementation in low- and middle-income countries: a systematic review

BACKGROUND

The Expert Recommendations for Implementing Change (ERIC) project developed a compilation of implementation strategies that are intended to standardize reporting and evaluation. Little is known about the application of ERIC in low- and middle-income countries (LMICs). We systematically reviewed the literature on the use and specification of ERIC strategies for health intervention implementation in LMICs to identify gaps and inform future research.

METHODS

We searched peer-reviewed articles published through March 2023 in any language that (1) were conducted in an LMIC and (2) cited seminal ERIC articles or (3) mentioned ERIC in the title or abstract. Two co-authors independently screened all titles, abstracts, and full-text articles, then abstracted study, intervention, and implementation strategy characteristics of included studies.

RESULTS

The final sample included 60 studies describing research from all world regions, with over 30% published in the final year of our review period. Most studies took place in healthcare settings (n = 52, 86.7%), while 11 (18.2%) took place in community settings and four (6.7%) at the policy level. Across studies, 548 distinct implementation strategies were identified with a median of six strategies (range 1-46 strategies) included in each study. Most studies (n = 32, 53.3%) explicitly matched implementation strategies used for the ERIC compilation. Among those that did, 64 (87.3%) of the 73 ERIC strategies were represented. Many of the strategies not cited included those that target systems- or policy-level barriers. Nearly 85% of strategies included some component of strategy specification, though most only included specification of their action (75.2%), actor (57.3%), and action target (60.8%). A minority of studies employed randomized trials or high-quality quasi-experimental designs; only one study evaluated implementation strategy effectiveness.

CONCLUSIONS

While ERIC use in LMICs is rapidly growing, its application has not been consistent nor commonly used to test strategy effectiveness. Research in LMICs must better specify strategies and evaluate their impact on outcomes. Moreover, strategies that are tested need to be better specified, so they may be compared across contexts. Finally, strategies targeting policy-, systems-, and community-level determinants should be further explored.

TRIAL REGISTRATION

PROSPERO, CRD42021268374.

Fidelity to the ACT SMART Toolkit: an instrumental case study of implementation strategy fidelity

BACKGROUND

Evidence-based practices (EBPs) are shown to improve a variety of outcomes for autistic children. However, EBPs often are mis-implemented or not implemented in community-based settings where many autistic children receive usual care services. A blended implementation process and capacity-building implementation strategy, developed to facilitate the adoption and implementation of EBPs for autism spectrum disorder (ASD) in community-based settings, is the Autism Community Toolkit: Systems to Measure and Adopt Research-based Treatments (ACT SMART Toolkit). Based on an adapted Exploration, Adoption decision, Preparation, Implementation, Sustainment (EPIS) Framework, the multi-phased ACT SMART Toolkit is comprised of (a) implementation facilitation, (b) agency-based implementation teams, and (c) a web-based interface. In this instrumental case study, we developed and utilized a method to evaluate fidelity to the ACT SMART Toolkit. This study responds to the need for implementation strategy fidelity evaluation methods and may provide evidence supporting the use of the ACT SMART Toolkit.

METHODS

We used an instrumental case study approach to assess fidelity to the ACT SMART Toolkit during its pilot study with six ASD community agencies located in southern California. We assessed adherence, dose, and implementation team responsiveness for each phase and activity of the toolkit at both an aggregate and individual agency level.

RESULTS

Overall, we found that adherence, dose, and implementation team responsiveness to the ACT SMART Toolkit were high, with some variability by EPIS phase and specific activity as well as by ASD community agency. At the aggregate level, adherence and dose were rated notably lowest during the preparation phase of the toolkit, which is a more activity-intensive phase of the toolkit.

CONCLUSIONS

This evaluation of fidelity to the ACT SMART Toolkit, utilizing an instrumental case study design, demonstrated the potential for the strategy to be used with fidelity in ASD community-based agencies. Findings related to the variability of implementation strategy fidelity in the present study may also inform future adaptations to the toolkit and point to broader trends of how implementation strategy fidelity may vary by content and context.

"We start where we are": a qualitative study of barriers and pragmatic solutions to the assessment and reporting of implementation strategy fidelity

BACKGROUND

Fidelity measurement of implementation strategies is underdeveloped and underreported, and the level of reporting is decreasing over time. Failing to properly measure the factors that affect the delivery of an implementation strategy may obscure the link between a strategy and its outcomes. Barriers to assessing and reporting implementation strategy fidelity among researchers are not well understood. The aims of this qualitative study were to identify barriers to fidelity measurement and pragmatic pathways towards improvement.

METHODS

We conducted in-depth interviews among researchers conducting implementation trials. We utilized a theory-informed interview approach to elicit the barriers and possible solutions to implementation strategy fidelity assessment and reporting. Reflexive-thematic analysis guided coding and memo-writing to determine key themes regarding barriers and solutions.

RESULTS

Twenty-two implementation researchers were interviewed. Participants agreed that implementation strategy fidelity was an essential element of implementation trials and that its assessment and reporting should improve. Key thematic barriers focused on (1) a current lack of validated fidelity tools with the need to assess fidelity in the short term, (2) the complex nature of some implementation strategies, (3) conceptual complications when assessing fidelity within mechanisms-focused implementation research, and (4) structural issues related to funding and publishing. Researchers also suggested pragmatic solutions to overcome each barrier. Respondents reported using specification and tracking data in the short term until validated tools become available. Participants suggested that researchers with strategy-specific content expertise lead the way in identifying core components and setting fidelity requirements for them. Addressing the third barrier, participants provided examples of what pragmatic prospective and retrospective fidelity assessments might look like along a mechanistic pathway. Finally, researchers described approaches to minimize costs of data collection, as well as more structural accountability like adopting and enforcing reporting guidelines or changing the structure of funding opportunities.

DISCUSSION

We propose short- and long-term priorities for improving the assessment and reporting of implementation strategy fidelity and the quality of implementation research.

CONCLUSIONS

A better understanding of the barriers to implementation strategy fidelity assessment may pave the way towards pragmatic solutions.

Examining the complementarity between the ERIC compilation of implementation strategies and the behaviour change technique taxonomy: a qualitative analysis

BACKGROUND

Efforts to generate evidence for implementation strategies are frustrated by insufficient description. The Expert Recommendations for Implementing Change (ERIC) compilation names and defines implementation strategies; however, further work is needed to describe the actions involved. One potentially complementary taxonomy is the behaviour change techniques (BCT) taxonomy. We aimed to examine the extent and nature of the overlap between these taxonomies.

METHODS

Definitions and descriptions of 73 strategies in the ERIC compilation were analysed. First, each description was deductively coded using the BCT taxonomy. Second, a typology was developed to categorise the extent of overlap between ERIC strategies and BCTs. Third, three implementation scientists independently rated their level of agreement with the categorisation and BCT coding. Finally, discrepancies were settled through online consensus discussions. Additional patterns of complementarity between ERIC strategies and BCTs were labelled thematically. Descriptive statistics summarise the frequency of coded BCTs and the number of strategies mapped to each of the categories of the typology.

RESULTS

Across the 73 strategies, 41/93 BCTs (44%) were coded, with 'restructuring the social environment' as the most frequently coded (n=18 strategies, 25%). There was direct overlap between one strategy (change physical structure and equipment) and one BCT ('restructuring physical environment'). Most strategy descriptions (n=64) had BCTs that were clearly indicated (n=18), and others where BCTs were probable but not explicitly described (n=31) or indicated multiple types of overlap (n=15). For some strategies, the presence of additional BCTs was dependent on the form of delivery. Some strategies served as examples of broad BCTs operationalised for implementation. For eight strategies, there were no BCTs indicated, or they did not appear to focus on changing behaviour. These strategies reflected preparatory stages and targeted collective cognition at the system level rather than behaviour change at the service delivery level.

CONCLUSIONS

This study demonstrates how the ERIC compilation and BCT taxonomy can be integrated to specify active ingredients, providing an opportunity to better understand mechanisms of action. Our results highlight complementarity rather than redundancy. More efforts to integrate these or other taxonomies will aid strategy developers and build links between existing silos in implementation science.