Fidelity to the ACT SMART Toolkit: an instrumental case study of implementation strategy fidelity

Firm, yet flexible: a fidelity debate paper with two case examples

BACKGROUND

In healthcare research and practice, intervention and implementation fidelity represent the steadfast adherence to core components of research-supported interventions and the strategies employed for their implementation. Evaluating fidelity involves determining whether these core components were delivered as intended. Without fidelity data, the results of complex interventions cannot be meaningfully interpreted. Increasingly, the necessity for firmness and strict adherence by implementers and their organizations has been questioned, with calls for flexibility to accommodate contextual conditions. This shift makes contemporary fidelity a balancing act, requiring researchers to navigate various tensions. This debate paper explores these tensions, drawing on experiences from developing fidelity assessments in two ongoing effectiveness-implementation hybrid trials.

MAIN BODY

First, given often scarce knowledge about the core components of complex interventions and implementation strategies, decisions about fidelity requirements involve a degree of subjective reasoning. Researchers should make these decisions transparent using theory or logic models. Second, because fidelity is context-dependent and applies to both interventions and implementation strategies, researchers must rethink fidelity concepts with every study while balancing firmness and flexibility. This is particularly crucial for hybrid studies, with their differing emphasis on intervention and implementation fidelity. Third, fidelity concepts typically focus on individual behaviors. However, since organizational and system factors also influence fidelity, there is a growing need to define fidelity criteria at these levels. Finally, as contemporary fidelity concepts prioritize flexible over firm adherence, building, evaluating, and maintaining fidelity in healthcare research has become more complex. This complexity calls for intensified efforts to expand the knowledge base for pragmatic and adaptive fidelity measurement in trial and routine healthcare settings.

CONCLUSION

Contemporary conceptualizations of fidelity place greater demands on how fidelity is examined, necessitating the expansion of fidelity frameworks to include organizational and system levels, the service- and study-specific conceptualizations of intervention and implementation fidelity, and the development of pragmatic approaches for assessing fidelity in research and practice. Continuing to build knowledge on how to balance requirements for firmness and flexibility remains a crucial task within the field of implementation science.

Applying the Cognitive Walkthrough for Implementation Strategies methodology to inform the redesign of a selection-quality implementation toolkit for use in schools

BACKGROUND

Implementation strategies are key to enhancing the translation of new innovations but there is a need to systematically design and tailor strategies to match the targeted implementation context and address determinants. There are increasing methods to inform the development and tailoring of implementation strategies to maximize their usability, feasibility, and appropriateness in new settings such as the Cognitive Walkthrough for Implementation Strategies (CWIS) approach. The aim of the current project is to apply the CWIS approach to inform the redesign of a multifaceted selection-quality implementation toolkit entitled Adoption of Curricular supports Toolkit: Systematic Measurement of Appropriateness and Readiness for Translation in Schools (ACT SMARTS) for use in middle and high schools.

METHODS

We systematically applied CWIS as the second part of a community-partnered iterative redesign of ACT SMARTS for schools to evaluate the usability and inform further toolkit redesign areas. We conducted three CWIS user testing sessions with key end users of school district administrators (n = 3), school principals (n = 6), and educators (n = 6).

RESULTS

Our CWIS application revealed that end users found ACT SMARTS acceptable and relevant but anticipate usability issues engaging in the ACT SMARTS process. Results informed the identification of eleven usability issues and corresponding redesign solutions to enhance the usability of ACT SMARTS for use in middle and high schools.

CONCLUSIONS

Results indicated the utility of CWIS in assessing implementation strategy usability in service of informing strategy modification as part of our broader redesign to improve alignment with end user, end recipient, and setting needs. Recommendations regarding the use of this participatory approach are discussed.

Applying the Cognitive Walkthrough for Implementation Strategies Methodology to Inform the Redesign of a Selection-Quality Implementation Toolkit for Use in Schools

Background

Implementation strategies are key to enhancing translation of new innovations but there is a need to systematically design and tailor strategies to match the targeted implementation context and address determinants. There are increasing methods to inform the redesign and tailoring of implementation strategies to maximize their usability, feasibility, and appropriateness in new settings such as the Cognitive Walkthrough for Implementation Strategies (CWIS) approach. The aim of the current project is to apply the CWIS approach to inform the redesign of a multifaceted selection-quality implementation toolkit entitled ACT SMARTS for use in middle and high schools.

Methods

We systematically applied CWIS as the second part of a community-partnered iterative redesign of ACT SMARTS for schools to evaluate the usability and inform further toolkit redesign areas. We conducted three CWIS user testing sessions with key end users of school district administrators, school principals, and educators.

Results

Our CWIS application revealed that end users found ACT SMARTS acceptable and relevant but anticipate usability issues engaging in the ACT SMARTS process. Results informed the identification of eleven usability issues and corresponding redesign solutions to enhance the usability of ACT SMARTS for use in middle and high schools.

Conclusions

Results indicated the utility of CWIS in assessing implementation strategy usability in service of informing strategy tailoring and redesign to improve alignment with user and setting needs. Recommendations regarding the use of this participatory approach are discussed.

A scoping review of theories, models and frameworks used or proposed to evaluate knowledge mobilization strategies

BACKGROUND

Evaluating knowledge mobilization strategies (KMb) presents challenges for organizations seeking to understand their impact to improve KMb effectiveness. Moreover, the large number of theories, models, and frameworks (TMFs) available can be confusing for users. Therefore, the purpose of this scoping review was to identify and describe the characteristics of TMFs that have been used or proposed in the literature to evaluate KMb strategies.

METHODS

A scoping review methodology was used. Articles were identified through searches in electronic databases, previous reviews and reference lists of included articles. Titles, abstracts and full texts were screened in duplicate. Data were charted using a piloted data charting form. Data extracted included study characteristics, KMb characteristics, and TMFs used or proposed for KMb evaluation. An adapted version of Nilsen (Implement Sci 10:53, 2015) taxonomy and the Expert Recommendations for Implementing Change (ERIC) taxonomy (Powell et al. in Implement Sci 10:21, 2015) guided data synthesis.

RESULTS

Of the 4763 search results, 505 were retrieved, and 88 articles were eligible for review. These consisted of 40 theoretical articles (45.5%), 44 empirical studies (50.0%) and four protocols (4.5%). The majority were published after 2010 (n = 70, 79.5%) and were health related (n = 71, 80.7%). Half of the studied KMb strategies were implemented in only four countries: Canada, Australia, the United States and the United Kingdom (n = 42, 47.7%). One-third used existing TMFs (n = 28, 31.8%). According to the adapted Nilsen taxonomy, process models (n = 34, 38.6%) and evaluation frameworks (n = 28, 31.8%) were the two most frequent types of TMFs used or proposed to evaluate KMb. According to the ERIC taxonomy, activities to "train and educate stakeholders" (n = 46, 52.3%) were the most common, followed by activities to "develop stakeholder interrelationships" (n = 23, 26.1%). Analysis of the TMFs identified revealed relevant factors of interest for the evaluation of KMb strategies, classified into four dimensions: context, process, effects and impacts.

CONCLUSIONS

This scoping review provides an overview of the many KMb TMFs used or proposed. The results provide insight into potential dimensions and components to be considered when assessing KMb strategies.