Strengthening methods for tracking adaptations and modifications to implementation strategies

Application of the FRAME-IS to a multifaceted implementation strategy

BACKGROUND

Research demonstrates the importance of documenting adaptations to implementation strategies that support integration of evidence-based interventions into practice. While studies have utilized the FRAME-IS [Framework for Reporting Adaptations and Modifications for Implementation Strategies] to collect structured adaptation data, they are limited by a focus on discrete implementation strategies (e.g., training), which do not reflect the complexity of multifaceted strategies like practice facilitation. In this paper, we apply the FRAME-IS to our trial evaluating the effectiveness of PF on implementation fidelity of an evidence-based technology-facilitated team care model for improved hypertension control within a federally qualified health center (FQHC).

METHODS

Three data sources are used to document adaptations: (1) implementation committee meeting minutes, (2) narrative reports completed by practice facilitators, and (3) structured notes captured on root cause analysis and Plan-Do-Study-Act worksheets. Text was extracted from the data sources according to the FRAME-IS modules and inputted into a master matrix for content analysis by two authors; a third author conducted member checking and code validation.

RESULTS

We modified the FRAME-IS to include part 2 of module 2 (what is modified) to add greater detail of the modified strategy, and a numbering system to track adaptations across the modules. This resulted in identification of 27 adaptations, of which 88.9% focused on supporting practices in identifying eligible patients and referring them to the intervention. About half (52.9%) of the adaptations were made to modify the context of the PF strategy to include a group-based format, add community health workers to the strategy, and to shift the implementation target to nurses. The adaptations were often widespread (83.9%), affecting all practices within the FQHC. While most adaptations were reactive (84.6%), they resulted from a systematic process of reviewing data captured by multiple sources. All adaptations included the FQHC in the decision-making process.

CONCLUSION

With modifications, we demonstrate the ability to document our adaptation data across the FRAME-IS modules, attesting to its applicability and value for a range of implementation strategies. Based on our experiences, we recommend refinement of tracking systems to support more nimble and practical documentation of iterative, ongoing, and multifaceted adaptations.

TRIAL REGISTRATION

Clinicaltrials.gov NCT03713515, Registration date: October 19, 2018.

Implementation of Health IT for Cancer Screening in US Primary Care: Scoping Review

BACKGROUND

A substantial percentage of the US population is not up to date on guideline-recommended cancer screenings. Identifying interventions that effectively improve screening rates would enhance the delivery of such screening. Interventions involving health IT (HIT) show promise, but much remains unknown about how HIT is optimized to support cancer screening in primary care.

OBJECTIVE

This scoping review aims to identify (1) HIT-based interventions that effectively support guideline concordance in breast, cervical, and colorectal cancer screening provision and follow-up in the primary care setting and (2) barriers or facilitators to the implementation of effective HIT in this setting.

METHODS

Following scoping review guidelines, we searched MEDLINE, CINAHL Plus, Web of Science, and IEEE Xplore databases for US-based studies from 2015 to 2021 that featured HIT targeting breast, colorectal, and cervical cancer screening in primary care. Studies were dual screened using a review criteria checklist. Data extraction was guided by the following implementation science frameworks: the Reach, Effectiveness, Adoption, Implementation, and Maintenance framework; the Expert Recommendations for Implementing Change taxonomy; and implementation strategy reporting domains. It was also guided by the Integrated Technology Implementation Model that incorporates theories of both implementation science and technology adoption. Reporting was guided by PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews).

RESULTS

A total of 101 studies met the inclusion criteria. Most studies (85/101, 84.2%) involved electronic health record-based HIT interventions. The most common HIT function was clinical decision support, primarily used for panel management or at the point of care. Most studies related to HIT targeting colorectal cancer screening (83/101, 82.2%), followed by studies related to breast cancer screening (28/101, 27.7%), and cervical cancer screening (19/101, 18.8%). Improvements in cancer screening were associated with HIT-based interventions in most studies (36/54, 67% of colorectal cancer-relevant studies; 9/14, 64% of breast cancer-relevant studies; and 7/10, 70% of cervical cancer-relevant studies). Most studies (79/101, 78.2%) reported on the reach of certain interventions, while 17.8% (18/101) of the included studies reported on the adoption or maintenance. Reported barriers and facilitators to HIT adoption primarily related to inner context factors of primary care settings (eg, staffing and organizational policies that support or hinder HIT adoption). Implementation strategies for HIT adoption were reported in 23.8% (24/101) of the included studies.

CONCLUSIONS

There are substantial evidence gaps regarding the effectiveness of HIT-based interventions, especially those targeting guideline-concordant breast and colorectal cancer screening in primary care. Even less is known about how to enhance the adoption of technologies that have been proven effective in supporting breast, colorectal, or cervical cancer screening. Research is needed to ensure that the potential benefits of effective HIT-based interventions equitably reach diverse primary care populations.

Application of the FRAME-IS to a Multifaceted Implementation Strategy

Background

Research demonstrates the importance of documenting adaptations to implementation strategies that support integration of evidence-based interventions into practice. While studies have utilized the FRAME-IS [Framework for Reporting Adaptations and Modifications for Implementation Strategies] to collect structured adaptation data, they are limited by a focus on discrete implementation strategies (e.g., training), which do not reflect the complexity of multifaceted strategies like practice facilitation (PF). In this paper, we apply the FRAME-IS to our trial evaluating the effectiveness of PF on implementation fidelity of an evidence-based technology-facilitated team care model for improved hypertension control within a federally qualified health center (FQHC).

Methods

Three data sources are used to document adaptations: (1) implementation committee meeting minutes, (2) narrative reports completed by practice facilitators, and (3) structured notes captured on root cause analysis and Plan-Do-Study-Act worksheets. Text was extracted from the data sources according to the FRAME-IS modules and inputted into a master matrix for content analysis by two authors; a third author conducted member checking and code validation.

Results

We modified the FRAME-IS to include part 2 of module 2 (what is modified) to add greater detail of the modified strategy, and a numbering system to track adaptations across the modules. This resulted in identification of 27 adaptations, of which 88.9% focused on supporting practices in identifying eligible patients and referring them to the intervention. About half (52.9%) of the adaptations were made to modify the context of the PF strategy to include a group-based format, add community health workers to the strategy, and to shift the implementation target to nurses. The adaptations were often widespread (83.9%), affecting all practices within the FQHC. While most adaptations were reactive (84.6%), they resulted from a systematic process of reviewing data captured by multiple sources. All adaptations included the FQHC in the decision-making process.

Conclusion

With modifications, we demonstrate the ability to document our adaptation data across the FRAME-IS modules, attesting to its applicability and value for a range of implementation strategies. Based on our experiences, we recommend refinement of tracking systems to support more nimble and practical documentation of iterative, ongoing, and multifaceted adaptations.

Trial Registration

clinicaltrials.gov NCT03713515, Registration date: October 19, 2018.

Analysis of FRAME data (A-FRAME): An analytic approach to assess the impact of adaptations on health services interventions and evaluations

Introduction

Tracking adaptations during implementation can help assess and interpret outcomes. The framework for reporting adaptations and modifications-expanded (FRAME) provides a structured approach to characterize adaptations. We applied the FRAME across multiple health services projects, and developed an analytic approach to assess the impact of adaptations.

Methods

Mixed methods analysis of research diaries from seven quality improvement (QI) and research projects during the early stages of the COVID-19 pandemic. Using the FRAME as a codebook, discrete adaptations were described and categorized. We then conducted a three-step analysis plan: (1) calculated the frequency of adaptations by FRAME categories across projects; (2) qualitatively assessed the impact of adaptations on project goals; and (3) qualitatively assessed relationships between adaptations within projects to thematically consolidate adaptations to generate more explanatory value on how adaptations influenced intervention progress and outcomes.

Results

Between March and July 2020, 42 adaptations were identified across seven health services projects. The majority of adaptations related to training or evaluation (52.4%) with the goal of maintaining the feasibility (66.7%) of executing projects during the pandemic. Five FRAME constructs offered the most explanatory benefit to assess the impact of adaptations on program and evaluation goals, providing the basis for creating an analytic approach dubbed the "A-FRAME," analysis of FRAME data. Using the A-FRAME, the 42 adaptations were consolidated into 17 succinct adaptations. Two QI projects discontinued altogether. Intervention adaptations related to staffing, training, or delivery, while evaluation adaptations included design, recruitment, and data collection adjustments.

Conclusions

By sifting qualitative data about adaptations into the A-FRAME, implementers and researchers can succinctly describe how adaptations affect interventions and their evaluations. The simple and concise presentation of information using the A-FRAME matrix can help implementers and evaluators account for the influence of adaptations on program outcomes.

Champion and audit and feedback strategy fidelity and their relationship to depression intervention fidelity: A mixed method study

Background

Globally, mental health disorders rank as the greatest cause of disability. Low and middle-income countries (LMICs) hold a disproportionate share of the mental health burden, especially as it pertains to depression. Depression is highly prevalent among those with non-communicable diseases (NCDs), creating a barrier to successful treatment. While some treatments have proven efficacy in LMIC settings, wide dissemination is challenged by multiple factors, leading researchers to call for implementation strategies to overcome barriers to care provision. However, implementation strategies are often not well defined or documented, challenging the interpretation of study results and the uptake and replication of strategies in practice settings. Assessing implementation strategy fidelity (ISF), or the extent to which a strategy was implemented as designed, overcomes these challenges. This study assessed fidelity of two implementation strategies (a 'basic' champion strategy and an 'enhanced' champion + audit and feedback strategy) to improve the integration of a depression intervention, measurement based care (MBC), at 10 NCD clinics in Malawi. The primary goal of this study was to assess the relationship between the implementation strategies and MBC fidelity using a mixed methods approach.

Methods

We developed a theory-informed mixed methods fidelity assessment that first combined an implementation strategy specification technique with a fidelity framework. We then created corresponding fidelity indicators to strategy components. Clinical process data and one-on-one in-depth interviews with 45 staff members at 6 clinics were utilized as data sources. Our final analysis used descriptive statistics, reflexive-thematic analysis (RTA), data merging, and triangulation to examine the relationship between ISF and MBC intervention fidelity.

Results

Our mixed methods analysis revealed how ISF may moderate the relationship between the strategies and MBC fidelity. Leadership engagement and implementation climate were critical for clinics to overcome implementation barriers and preserve implementation strategy and MBC fidelity. Descriptive statistics determined champion strategy fidelity to range from 61 to 93% across the 10 clinics. Fidelity to the audit and feedback strategy ranged from 82 to 91% across the 5 clinics assigned to that condition. MBC fidelity ranged from 54 to 95% across all clinics. Although correlations between ISF and MBC fidelity were not statistically significant due to the sample of 10 clinics, associations were in the expected direction and of moderate effect size. A coefficient for shared depression screening among clinicians had greater face validity compared to depression screening coverage and functioned as a proximal indicator of implementation strategy success.

Conclusion

Fidelity to the basic and enhanced strategies varied by site and were influenced by leadership engagement and implementation climate. Champion strategies may benefit from the addition of leadership strategies to help address implementation barriers outside the purview of champions. ISF may moderate the relationship between strategies and implementation outcomes.

The Longitudinal Implementation Strategy Tracking System (LISTS): feasibility, usability, and pilot testing of a novel method

BACKGROUND

Systematic approaches are needed to accurately characterize the dynamic use of implementation strategies and how they change over time. We describe the development and preliminary evaluation of the Longitudinal Implementation Strategy Tracking System (LISTS), a novel methodology to document and characterize implementation strategies use over time.

METHODS

The development and initial evaluation of the LISTS method was conducted within the Improving the Management of SymPtoms during And following Cancer Treatment (IMPACT) Research Consortium (supported by funding provided through the NCI Cancer MoonshotSM). The IMPACT Consortium includes a coordinating center and three hybrid effectiveness-implementation studies testing routine symptom surveillance and integration of symptom management interventions in ambulatory oncology care settings. LISTS was created to increase the precision and reliability of dynamic changes in implementation strategy use over time. It includes three components: (1) a strategy assessment, (2) a data capture platform, and (3) a User's Guide. An iterative process between implementation researchers and practitioners was used to develop, pilot test, and refine the LISTS method prior to evaluating its use in three stepped-wedge trials within the IMPACT Consortium. The LISTS method was used with research and practice teams for approximately 12 months and subsequently we evaluated its feasibility, acceptability, and usability using established instruments and novel questions developed specifically for this study.

RESULTS

Initial evaluation of LISTS indicates that it is a feasible and acceptable method, with content validity, for characterizing and tracking the use of implementation strategies over time. Users of LISTS highlighted several opportunities for improving the method for use in future and more diverse implementation studies.

CONCLUSIONS

The LISTS method was developed collaboratively between researchers and practitioners to fill a research gap in systematically tracking implementation strategy use and modifications in research studies and other implementation efforts. Preliminary feedback from LISTS users indicate it is feasible and usable. Potential future developments include additional features, fewer data elements, and interoperability with alternative data entry platforms. LISTS offers a systematic method that encourages the use of common data elements to support data analysis across sites and synthesis across studies. Future research is needed to further adapt, refine, and evaluate the LISTS method in studies with employ diverse study designs and address varying delivery settings, health conditions, and intervention types.

Application of the Expert Recommendations for Implementing Change (ERIC) compilation of strategies to health intervention implementation in low- and middle-income countries: a systematic review

BACKGROUND

The Expert Recommendations for Implementing Change (ERIC) project developed a compilation of implementation strategies that are intended to standardize reporting and evaluation. Little is known about the application of ERIC in low- and middle-income countries (LMICs). We systematically reviewed the literature on the use and specification of ERIC strategies for health intervention implementation in LMICs to identify gaps and inform future research.

METHODS

We searched peer-reviewed articles published through March 2023 in any language that (1) were conducted in an LMIC and (2) cited seminal ERIC articles or (3) mentioned ERIC in the title or abstract. Two co-authors independently screened all titles, abstracts, and full-text articles, then abstracted study, intervention, and implementation strategy characteristics of included studies.

RESULTS

The final sample included 60 studies describing research from all world regions, with over 30% published in the final year of our review period. Most studies took place in healthcare settings (n = 52, 86.7%), while 11 (18.2%) took place in community settings and four (6.7%) at the policy level. Across studies, 548 distinct implementation strategies were identified with a median of six strategies (range 1-46 strategies) included in each study. Most studies (n = 32, 53.3%) explicitly matched implementation strategies used for the ERIC compilation. Among those that did, 64 (87.3%) of the 73 ERIC strategies were represented. Many of the strategies not cited included those that target systems- or policy-level barriers. Nearly 85% of strategies included some component of strategy specification, though most only included specification of their action (75.2%), actor (57.3%), and action target (60.8%). A minority of studies employed randomized trials or high-quality quasi-experimental designs; only one study evaluated implementation strategy effectiveness.

CONCLUSIONS

While ERIC use in LMICs is rapidly growing, its application has not been consistent nor commonly used to test strategy effectiveness. Research in LMICs must better specify strategies and evaluate their impact on outcomes. Moreover, strategies that are tested need to be better specified, so they may be compared across contexts. Finally, strategies targeting policy-, systems-, and community-level determinants should be further explored.

TRIAL REGISTRATION

PROSPERO, CRD42021268374.

Creating a Statistical Analysis Plan to Continually Evaluate Intervention Adaptations that Arise in Real-World Implementation

Evidence-based health interventions are frequently translated into real-world settings where practical needs drive changes to intervention protocols. Due to logistical and resource constraints, these naturally arising adaptations are rarely assessed for comparative effectiveness using a randomized trial. Nevertheless, when observational data are available, it is still possible to identify beneficial adaptations using statistical methods that adjust for differences among intervention groups. As implementation continues and more data are collected and assessed, we also require analysis methods that ensure low statistical error rates as multiple comparisons are made over time. This paper describes how to create a statistical analysis plan for evaluating adaptations to an intervention during ongoing implementation. This can be done by combining methods commonly used in platform clinical trials with methods used for real-world data. We also demonstrate how to use simulations based on previous data to decide the frequency with which to conduct statistical analyses. The illustration uses data from large-scale implementation of a school-based resilience and skill-building preventive intervention to which several adaptations were made. The proposed statistical analysis plan for evaluating the school-based intervention has potential to improve population-level outcomes as implementation scales up further and additional adaptations are anticipated.

Characterizing provider-led adaptations to mobile phone delivery of the Adolescent Transition Package (ATP) in Kenya using the Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies (FRAME-IS): a mixed methods approach

BACKGROUND

The COVID-19 pandemic resulted in disruptions to routine HIV services for youth living with HIV (YLH), provoking rapid adaptation to mitigate interruptions in care. The Adolescent Transition to Adult Care for HIV-infected adolescents (ATTACH) study (NCT03574129) was a hybrid I cluster randomized trial testing the effectiveness of a healthcare worker (HCW)-delivered disclosure and transition intervention - the Adolescent Transition Package (ATP). During the pandemic, HCWs leveraged phone delivery of the ATP and were supported to make adaptations. We characterized real-time, provider-driven adaptations made to support phone delivery of the ATP.

METHODS

We conducted continuous quality improvement (CQI) meetings with HCWs involved in phone delivery of the ATP at 10 intervention sites. CQI meetings used plan-do-study-act (PDSA) cycles and were audio-recorded. Adaptations were coded by two-independent coders using the Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies (FRAME-IS). Adaptation testing outcomes (adopt, retest, or abandon) and provider experience implementing the adaptations were also recorded. We summarized adaptation characteristics, provider experience, and outcomes.

RESULTS

We identified 72 adaptations, 32 were unique. Overall, adaptations included modification to context (53%, n = 38), content (49%, n = 35), and evaluation processes (13%, n = 9). Context adaptations primarily featured changes to personnel, format, and setting, while content and evaluation adaptations were frequently achieved by simple additions, repetition, and tailoring/refining of the phone delivery strategy. Nine adaptations involved abandoning, then returning to phone delivery. HCWs sought to increase reach, improve fidelity, and intervention fit within their context. Most adaptations (96%, n = 69) were perceived to increase the feasibility of phone delivery when compared to before the changes were introduced, and HCWs felt 83% (n = 60) of adaptations made phone delivery easier. Most adaptations were either incorporated into routine workflows (47%) or tested again (47%).

CONCLUSION

Adaptation of phone delivery was a feasible and effective way of addressing challenges with continuity of care for YLH during the COVID-19 pandemic. Adaptations were primarily context adaptions. While FRAME-IS was apt for characterizing adaptations, more use cases are needed to explore the range of its utility.

TRIAL REGISTRATION

Trial registered on ClinicalTrial.gov as NCT03574129.

Connecting the science and practice of implementation - applying the lens of context to inform study design in implementation research

The saying "horses for courses" refers to the idea that different people and things possess different skills or qualities that are appropriate in different situations. In this paper, we apply the analogy of "horses for courses" to stimulate a debate about how and why we need to get better at selecting appropriate implementation research methods that take account of the context in which implementation occurs. To ensure that implementation research achieves its intended purpose of enhancing the uptake of research-informed evidence in policy and practice, we start from a position that implementation research should be explicitly connected to implementation practice. Building on our collective experience as implementation researchers, implementation practitioners (users of implementation research), implementation facilitators and implementation educators and subsequent deliberations with an international, inter-disciplinary group involved in practising and studying implementation, we present a discussion paper with practical suggestions that aim to inform more practice-relevant implementation research.

Identification of Implementation Strategies Using the CFIR-ERIC Matching Tool to Mitigate Barriers in a Primary Care Model for Older Veterans

BACKGROUND AND OBJECTIVES

As the proportion of the U.S. population over 65 and living with complex chronic conditions grows, understanding how to strengthen the implementation of age-sensitive primary care models for older adults, such as the Veterans Health Administration's Geriatric Patient-Aligned Care Teams (GeriPACT), is critical. However, little is known about which implementation strategies can best help to mitigate barriers to adopting these models. We aimed to identify barriers to GeriPACT implementation and strategies to address these barriers using the Consolidated Framework for Implementation Research-Expert Recommendations for Implementing Change (CFIR-ERIC) Matching Tool.

RESEARCH DESIGN AND METHODS

We conducted a content analysis of qualitative responses obtained from a web-based survey sent to GeriPACT members. Using a matrix approach, we grouped similar responses into key barrier categories. After mapping barriers to CFIR, we used the Tool to identify recommended strategies.

RESULTS

Across 53 Veterans Health Administration hospitals, 32% of team members (n = 197) responded to our open-ended question about barriers to GeriPACT care. Barriers identified include Available Resources, Networks & Communication, Design Quality & Packaging, Knowledge & Beliefs, Leadership Engagement, and Relative Priority. The Tool recommended 12 Level 1 (e.g., conduct educational meetings) and 24 Level 2 ERIC strategies (e.g., facilitation). Several strategies (e.g., conduct local consensus discussions) cut across multiple barriers.

DISCUSSION AND IMPLICATIONS

Strategies identified by the Tool can inform on-going development of the GeriPACT model's effective implementation and sustainment. Incorporating cross-cutting implementation strategies that mitigate multiple barriers at once may further support these next steps.

The Implementation Playbook: study protocol for the development and feasibility evaluation of a digital tool for effective implementation of evidence-based innovations

BACKGROUND

Evidence-based innovations can improve health outcomes, but only if successfully implemented. Implementation can be complex, highly susceptible to failure, costly and resource intensive. Internationally, there is an urgent need to improve the implementation of effective innovations. Successful implementation is best guided by implementation science, but organizations lack implementation know-how and have difficulty applying it. Implementation support is typically shared in static, non-interactive, overly academic guides and is rarely evaluated. In-person implementation facilitation is often soft-funded, costly, and scarce. This study seeks to improve effective implementation by (1) developing a first-in-kind digital tool to guide pragmatic, empirically based and self-directed implementation planning in real-time; and (2) exploring the tool's feasibility in six health organizations implementing different innovations.

METHODS

Ideation emerged from a paper-based resource, The Implementation Game©, and a revision called The Implementation Roadmap©; both integrate core implementation components from evidence, models and frameworks to guide structured, explicit, and pragmatic planning. Prior funding also generated user personas and high-level product requirements. This study will design, develop, and evaluate the feasibility of a digital tool called The Implementation Playbook©. In Phase 1, user-centred design and usability testing will inform tool content, visual interface, and functions to produce a minimum viable product. Phase 2 will explore the Playbook's feasibility in six purposefully selected health organizations sampled for maximum variation. Organizations will use the Playbook for up to 24 months to implement an innovation of their choosing. Mixed methods will gather: (i) field notes from implementation team check-in meetings; (ii) interviews with implementation teams about their experience using the tool; (iii) user free-form content entered into the tool as teams work through implementation planning; (iv) Organizational Readiness for Implementing Change questionnaire; (v) System Usability Scale; and (vi) tool metrics on how users progressed through activities and the time required to do so.

DISCUSSION

Effective implementation of evidence-based innovations is essential for optimal health. We seek to develop a prototype digital tool and demonstrate its feasibility and usefulness across organizations implementing different innovations. This technology could fill a significant need globally, be highly scalable, and potentially valid for diverse organizations implementing various innovations.

Creating research-ready partnerships: the initial development of seven implementation laboratories to advance cancer control

BACKGROUND

In 2019-2020, with National Cancer Institute funding, seven implementation laboratory (I-Lab) partnerships between scientists and stakeholders in 'real-world' settings working to implement evidence-based interventions were developed within the Implementation Science Centers in Cancer Control (ISC3) consortium. This paper describes and compares approaches to the initial development of seven I-Labs in order to gain an understanding of the development of research partnerships representing various implementation science designs.

METHODS

In April-June 2021, members of the ISC3 Implementation Laboratories workgroup interviewed research teams involved in I-Lab development in each center. This cross-sectional study used semi-structured interviews and case-study-based methods to collect and analyze data about I-Lab designs and activities. Interview notes were analyzed to identify a set of comparable domains across sites. These domains served as the framework for seven case descriptions summarizing design decisions and partnership elements across sites.

RESULTS

Domains identified from interviews as comparable across sites included engagement of community and clinical I-Lab members in research activities, data sources, engagement methods, dissemination strategies, and health equity. The I-Labs use a variety of research partnership designs to support engagement including participatory research, community-engaged research, and learning health systems of embedded research. Regarding data, I-Labs in which members use common electronic health records (EHRs) leverage these both as a data source and a digital implementation strategy. I-Labs without a shared EHR among partners also leverage other sources for research or surveillance, most commonly qualitative data, surveys, and public health data systems. All seven I-Labs use advisory boards or partnership meetings to engage with members; six use stakeholder interviews and regular communications. Most (70%) tools or methods used to engage I-Lab members such as advisory groups, coalitions, or regular communications, were pre-existing. Think tanks, which two I-Labs developed, represented novel engagement approaches. To disseminate research results, all centers developed web-based products, and most (n = 6) use publications, learning collaboratives, and community forums. Important variations emerged in approaches to health equity, ranging from partnering with members serving historically marginalized populations to the development of novel methods.

CONCLUSIONS

The development of the ISC3 implementation laboratories, which represented a variety of research partnership designs, offers the opportunity to advance understanding of how researchers developed and built partnerships to effectively engage stakeholders throughout the cancer control research lifecycle. In future years, we will be able to share lessons learned for the development and sustainment of implementation laboratories.

Design and management considerations for control groups in hybrid effectiveness-implementation trials: Narrative review & case studies

Hybrid effectiveness-implementation studies allow researchers to combine study of a clinical intervention's effectiveness with study of its implementation with the aim of accelerating the translation of evidence into practice. However, there currently exists limited guidance on how to design and manage such hybrid studies. This is particularly true for studies that include a comparison/control arm that, by design, receives less implementation support than the intervention arm. Lack of such guidance can present a challenge for researchers both in setting up but also in effectively managing participating sites in such trials. This paper uses a narrative review of the literature (Phase 1 of the research) and comparative case study of three studies (Phase 2 of the research) to identify common themes related to study design and management. Based on these, we comment and reflect on: (1) the balance that needs to be struck between fidelity to the study design and tailoring to emerging requests from participating sites as part of the research process, and (2) the modifications to the implementation strategies being evaluated. Hybrid trial teams should carefully consider the impact of design selection, trial management decisions, and any modifications to implementation processes and/or support on the delivery of a controlled evaluation. The rationale for these choices should be systematically reported to fill the gap in the literature.

Tracking dynamic changes in implementation strategies over time within a hybrid type 2 trial of an electronic patient-reported oncology symptom and needs monitoring program

Background

Longitudinal tracking of implementation strategies is critical in accurately reporting when and why they are used, for promoting rigor and reproducibility in implementation research, and could facilitate generalizable knowledge if similar methods are used across research projects. This article focuses on tracking dynamic changes in the use of implementation strategies over time within a hybrid type 2 effectiveness-implementation trial of an evidence-based electronic patient-reported oncology symptom assessment for cancer patient-reported outcomes in a single large healthcare system.

Methods

The Longitudinal Implementation Strategies Tracking System (LISTS), a timeline follow-back procedure for documenting strategy use and modifications, was applied to the multiyear study. The research team used observation, study records, and reports from implementers to complete LISTS in an electronic data entry system. Types of modifications and reasons were categorized. Determinants associated with each strategy were collected as a justification for strategy use and a potential explanation for strategy modifications.

Results

Thirty-four discrete implementation strategies were used and at least one strategy was used from each of the nine strategy categories from the Expert Recommendations for Implementing Change (ERIC) taxonomy. Most of the strategies were introduced, used, and continued or discontinued according to a prospective implementation plan. Relatedly, a small number of strategies were introduced, the majority unplanned, because of the changing healthcare landscape, or to address an emergent barrier. Despite changing implementation context, there were relatively few modifications to the way strategies were enacted, such as a change in the actor, action, or dose. Few differences were noted between the trial's three regional units under investigation.

Conclusion

This study occurred within the ambulatory oncology clinics of a large, academic medical center and was supported by the Quality team of the health system to ensure greater uptake, uniformity, and implementation within established practice change processes. The centralized nature of the implementation likely contributed to the relatively low proportion of modified strategies and the high degree of uniformity across regions. These results demonstrate the potential of LISTS in gathering the level of data needed to understand the impact of the many implementation strategies used to support adoption and delivery of a multilevel innovation.

Clinical trial registration

https://clinicaltrials.gov/ct2/show/NCT04014751, identifier: NCT04014751.

"We start where we are": a qualitative study of barriers and pragmatic solutions to the assessment and reporting of implementation strategy fidelity

BACKGROUND

Fidelity measurement of implementation strategies is underdeveloped and underreported, and the level of reporting is decreasing over time. Failing to properly measure the factors that affect the delivery of an implementation strategy may obscure the link between a strategy and its outcomes. Barriers to assessing and reporting implementation strategy fidelity among researchers are not well understood. The aims of this qualitative study were to identify barriers to fidelity measurement and pragmatic pathways towards improvement.

METHODS

We conducted in-depth interviews among researchers conducting implementation trials. We utilized a theory-informed interview approach to elicit the barriers and possible solutions to implementation strategy fidelity assessment and reporting. Reflexive-thematic analysis guided coding and memo-writing to determine key themes regarding barriers and solutions.

RESULTS

Twenty-two implementation researchers were interviewed. Participants agreed that implementation strategy fidelity was an essential element of implementation trials and that its assessment and reporting should improve. Key thematic barriers focused on (1) a current lack of validated fidelity tools with the need to assess fidelity in the short term, (2) the complex nature of some implementation strategies, (3) conceptual complications when assessing fidelity within mechanisms-focused implementation research, and (4) structural issues related to funding and publishing. Researchers also suggested pragmatic solutions to overcome each barrier. Respondents reported using specification and tracking data in the short term until validated tools become available. Participants suggested that researchers with strategy-specific content expertise lead the way in identifying core components and setting fidelity requirements for them. Addressing the third barrier, participants provided examples of what pragmatic prospective and retrospective fidelity assessments might look like along a mechanistic pathway. Finally, researchers described approaches to minimize costs of data collection, as well as more structural accountability like adopting and enforcing reporting guidelines or changing the structure of funding opportunities.

DISCUSSION

We propose short- and long-term priorities for improving the assessment and reporting of implementation strategy fidelity and the quality of implementation research.

CONCLUSIONS

A better understanding of the barriers to implementation strategy fidelity assessment may pave the way towards pragmatic solutions.

Implementation mapping to plan for a hybrid trial testing the effectiveness and implementation of a behavioral intervention for HIV medication adherence and care retention

Background

Implementation mapping is a systematic, collaborative, and contextually-attentive method for developing implementation strategies. As an exemplar, we applied this method to strategy development for Managed Problem Solving Plus (MAPS+), an adapted evidence-based intervention for HIV medication adherence and care retention that will be delivered by community health workers and tested in an upcoming trial.

Methods

In Step 1: Conduct Needs Assessment, we interviewed 31 stakeholders to identify determinants of MAPS+ implementation in 13 clinics serving people with HIV in Philadelphia County. In Step 2: Develop Logic Model, we used these determinants as inputs for a working logic model guided by the Consolidated Framework for Implementation Research. In Step 3: Operationalize Implementation Strategies, our team held a virtual stakeholder meeting to confirm determinants. We synthesized stakeholder feedback, then identified implementation strategies that conceptually matched to determinants using the Expert Recommendations for Implementing Change taxonomy. Next, we operationalized implementation strategies with specific examples for clinic settings. We linked strategies to behavior change theories to allow for a mechanistic understanding. We then held a second virtual stakeholder meeting to present the implementation menu for feedback and glean generalizable insights for how these strategies could be operationalized in each stakeholder's clinic. In Step 4: Protocolize Strategies, we incorporated stakeholder feedback and finalized the implementation strategy menu.

Findings

Implementation mapping produced a menu of 39 strategies including revise professional roles, identify and prepare champions, use warm handoffs, and change record systems. The process of implementation mapping generated key challenges for implementation strategy development: lack of implementation strategies targeting the outer setting (i.e., sociopolitical context); tension between a one-size-fits-all and individualized approach for all clinics; conceptual confusion between facilitators and strategies; and challenges in translating the implementation science lexicon for partners.

Implications

This case exemplar advances both MAPS+ implementation and implementation science methods by furthering our understanding of the use of implementation mapping to develop strategies that enhance uptake of evidence-based interventions. The implementation menu will inform MAPS+ deployment across Philadelphia in an upcoming hybrid trial. We will carry out Step 5: Test Strategies to test the effectiveness and implementation of MAPS+.

Five Common Myths Limiting Engagement in HIV-Related Implementation Research

ABSTRACT

HIV-related implementation research holds great promise in achieving the potential of efficacious prevention and treatment tools in reducing the incidence of HIV and improving HIV treatment outcomes among people living with HIV. From the perspectives of HIV-related implementation research training and academia and through consultations with funders and investigators new to implementation research, we identified 5 myths that act as barriers to engagement in implementation research among new investigators. Prevailing myths broadly include (1) one must rigidly apply all aspects of an implementation framework for it to be valid, (2) implementation research limits the type of designs available to researchers, (3) implementation strategies cannot be patient-level or client-level approaches, (4) only studies prioritizing implementation outcomes are "true" implementation research, and (5) if not explicitly labeled implementation research, it may have limited impact on implementation. We offer pragmatic approaches to negotiate these myths with the goal of encouraging dialog, ensuring high-quality research, and fostering a more inclusive and dynamic field of implementation research. Ultimately, the goal of dispelling these myths was to lower the perceived bar to engagement in HIV-related implementation research while still ensuring quality in the methods and measures used.

Adaptation of communication assessment tool for community pharmacists in medication adherence and minor diseases management

Aim: To develop two versions of the Communication Assessment Tool (CAT) skilled for the setting of community pharmacy and to pilot test it on a selected sample.

Materials: Development of two versions of CAT-tool for community pharmacists. Validity and reliability assessments were required to determine the psychometric properties of developed tool versions. To investigate the construct validity of each adapted tool item, confirmatory factor analysis was performed. Reliability was assessed with the Cronbach’s Alpha evaluation, internal validity by submitting tool versions to patients of eleven pharmacies from North, Center, and South of Italy for pilot testing.

Results: Two CAT versions were developed and tested: CAT-Pharm-community Adherence to therapy and Minor Disease Management versions. First to evaluate pharmacist-patient communication following the dispensing of a prescription drug, second a consultation for minor disease management.

Conclusion: Communication tools are useful to implement optimal management of chronic diseases to minimize non-adherence and patients’ negative health outcomes.

A Modified Implementation Mapping Methodology for Evaluating and Learning From Existing Implementation

When empirically supported interventions are implemented in real-world practice settings, the process of how these interventions are implemented is highly relevant for their potential success. Implementation Mapping is a method that provides step-by-step guidance for systematically designing implementation processes that fit the respective intervention and context. It includes needs assessments among relevant stakeholders, the identification of implementation outcomes and determinants, the selection and design of appropriate implementation strategies, the production of implementation protocols and an implementation outcome evaluation. Implementation Mapping is generally conceptualized as a tool to prospectively guide implementation. However, many implementation efforts build on previous or ongoing implementation efforts, i.e., “existing implementation.” Learnings from existing implementation may offer insights critical to the success of further implementation activities. In this article, we present a modified Implementation Mapping methodology to be applied when evaluating existing implementation. We illustrate the methodology using the example of evaluating ongoing organized colorectal cancer screening programs in Switzerland. Through this example, we describe how we identify relevant stakeholders, implementation determinants and outcomes as well as currently employed implementation strategies. Moreover, we describe how we compare the types of strategies that are part of existing implementation efforts with those that implementation science would suggest as being suited to address identified implementation determinants. The results can be used for assessing the current state of implementation outcomes, refining ongoing implementation strategies, and informing future implementation efforts.

The case for prioritizing implementation strategy fidelity measurement: benefits and challenges

Implementation strategies are systematic approaches to improve the uptake and sustainability of evidence-based interventions. They frequently focus on changing provider behavior through the provision of interventions such as training, coaching, and audit-and-feedback. Implementation strategies often impact intermediate behavioral outcomes like provider guideline adherence, in turn improving patient outcomes. Fidelity of implementation strategy delivery is defined as the extent to which an implementation strategy is carried out as it was designed. Implementation strategy fidelity measurement is under-developed and under-reported, with the quality of reporting decreasing over time. Benefits of fidelity measurement include the exploration of the extent to which observed effects are moderated by fidelity, and critical information about Type-III research errors, or the likelihood that null findings result from implementation strategy fidelity failure. Reviews of implementation strategy efficacy often report wide variation across studies, commonly calling for increased implementation strategy fidelity measurement to help explain variations. Despite the methodological benefits of rigorous fidelity measurement, implementation researchers face multi-level challenges and complexities. Challenges include the measurement of a complex variable, multiple data collection modalities with varying precision and costs, and the need for fidelity measurement to change in-step with adaptations. In this position paper, we weigh these costs and benefits and ultimately contend that implementation strategy fidelity measurement and reporting should be improved in trials of implementation strategies. We offer pragmatic solutions for researchers to make immediate improvements like the use of mixed methods or innovative data collection and analysis techniques, the inclusion of implementation strategy fidelity assessment in reporting guidelines, and the staged development of fidelity tools across the evolution of an implementation strategy. We also call for additional research into the barriers and facilitators of implementation strategy fidelity measurement to further clarify the best path forward.

The quick pivot: Capturing real world modifications for the re-implementation of an early psychosis program transitioning to virtual delivery

Background

Team-based Early Psychosis Intervention (EPI) services is standard of care for youth with psychosis. The COVID-19 pandemic required most EPI services to mount an unplanned, rapid pivot to virtual delivery, with limited guidance on how to deliver virtual clinical services or whether quality of re-implementation and treatment outcomes would be impacted. We used a structured approach to identify essential modifications for the delivery of core components and explored facilitators and barriers for re-implementation and fidelity of a virtually delivered EPI intervention.

Materials and methods

NAVIGATE is a structured approach to team-based EPI. It provides detailed modules to guide delivery of core components including medication management, psychoeducation and psychotherapies, supported employment/education, and family education. Having initially implemented NAVIGATE at the Centre for Addiction and Mental Health (CAMH) in 2017, the EPI service transitioned to virtual delivery amid the COVID pandemic. Using a practice profile developed to support implementation, we detailed how core components of NAVIGATE were rapidly modified for virtual delivery as reported in structured group meetings with clinicians. The Framework for Reporting Adaptations and Modifications for Evidence-Based Interventions (FRAME) was used to describe modifications. Fidelity to the EPI standards of care was assessed by the First Episode Psychosis Fidelity Scale (FEPS-FS). Re-implementation barriers and facilitators and subsequent mitigation strategies were explored using structured clinician interviews guided by the Consolidated Framework for Implementation Research (CFIR).

Results

Identified modifications related to the intervention process, context, and training. We identified contextual factors affecting the re-implementation of virtually delivered NAVIGATE and then documented mitigating strategies that addressed these barriers. Findings can inform the implementation of virtual EPI services elsewhere, including guidance on processes, training and technology, and approaches to providing care virtually.

Discussion

This study identified modifications, impacts and mitigations to barriers emerging from rapid, unplanned virtual delivery of EPI services. These findings can support delivery of high-quality virtual services to youth with psychosis when virtual care is indicated.

Context-specific adaptation of an eHealth-facilitated, integrated care model and tailoring its implementation strategies-A mixed-methods study as a part of the SMILe implementation science project

Background

Contextually adapting complex interventions and tailoring their implementation strategies is key to a successful and sustainable implementation. While reporting guidelines for adaptations and tailoring exist, less is known about how to conduct context-specific adaptations of complex health care interventions.

Aims

To describe in methodological terms how the merging of contextual analysis results (step 1) with stakeholder involvement, and considering overarching regulations (step 2) informed our adaptation of an Integrated Care Model (ICM) for SteM cell transplantatIon faciLitated by eHealth (SMILe) and the tailoring of its implementation strategies (step 3).

Methods

Step 1: We used a mixed-methods design at University Hospital Basel, guided by the Basel Approach for coNtextual ANAlysis (BANANA). Step 2: Adaptations of the SMILe-ICM and tailoring of implementation strategies were discussed with an interdisciplinary team (n = 28) by considering setting specific and higher-level regulatory scenarios. Usability tests were conducted with patients (n = 5) and clinicians (n = 4). Step 3: Adaptations were conducted by merging our results from steps 1 and 2 using the Framework for Reporting Adaptations and Modifications-Enhanced (FRAME). We tailored implementation strategies according to the Expert Recommendations for Implementing Change (ERIC) compilation.

Results

Step 1: Current clinical practice was mostly acute-care-driven. Patients and clinicians valued eHealth-facilitated ICMs to support trustful patient-clinician relationships and the fitting of eHealth components to context-specific needs. Step 2: Based on information from project group meetings, adaptations were necessary on the organizational level (e.g., delivery of self-management information). Regulations informed the tailoring of SMILe-ICM`s visit timepoints and content; data protection management was adapted following Swiss regulations; and steering group meetings supported infrastructure access. The usability tests informed further adaptation of technology components. Step 3: Following FRAME and ERIC, SMILe-ICM and its implementation strategies were contextually adapted and tailored to setting-specific needs.

Discussion

This study provides a context-driven methodological approach on how to conduct intervention adaptation including the tailoring of its implementation strategies. The revealed meso-, and macro-level differences of the contextual analysis suggest a more targeted approach to enable an in-depth adaptation process. A theory-guided adaptation phase is an important first step and should be sufficiently incorporated and budgeted in implementation science projects.