Evaluating agency-led adaptions to an evidence-based lifestyle intervention for adults with serious mental illness

The adaptation and fidelity tool to support social service practitioners in balancing fidelity and adaptations: Longitudinal, mixed-method evaluation study

Background

Evidence-based interventions (EBIs) seldom fit seamlessly into a setting and are often adapted. The literature identifies practitioners' management of fidelity and adaptations as problematic but offers little guidance. This study aimed to investigate practitioners' perceptions of the feasibility and usability of an intervention aimed to support them in fidelity and adaptation management when working with EBIs.

Methods

The intervention, the adaptation and fidelity tool (A-FiT), was developed based on the literature, along with input from social service practitioners and social services' Research and Development units' personnel. The intervention consisted of two workshops where the participants were guided through a five-step process to manage fidelity and adaptations. It was tested in a longitudinal mixed-method intervention study with 103 practitioners from 19 social service units in Stockholm, Sweden. A multimethod data collection was employed, which included interviews at follow-up, questionnaires at baseline and follow-up (readiness for change and self-rated knowledge), workshop evaluation questionnaires (usability and feasibility) after each workshop, and documentation (participants' notes on worksheets). To analyze the data, qualitative content analysis, Kruskal-Wallis tests, and Wilcoxon rank-sum tests were performed.

Results

Overall, the practitioners had a positive perception of the intervention and perceived it as relevant for fidelity and adaptation management (mean ratings over 7.0 on usability and feasibility). The workshops also provided new knowledge and skills to manage fidelity and adaptations. Furthermore, the intervention provided insights into the practitioners' understanding about adaptation and fidelity through a more reflective approach.

Conclusion

Practical tools are needed to guide professionals not only to adhere to intervention core elements but also to help them to manage fidelity and adaptation. The proposed A-FiT intervention for practitioners' management of both fidelity and adaptation is a novel contribution to the implementation literature. Potentially, the next step is an evaluation of the intervention's impact in an experimental design.

Use of the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework to guide iterative adaptations: Applications, lessons learned, and future directions

Introduction

Implementation science frameworks have been used widely for planning and evaluation, but seldom to guide adaptations during program implementation. There is great potential for these frameworks to be used to inform conceptual and data-driven decisions about adaptations.

Methods

We summarize recent applications using Iterative RE-AIM to capture and guide adaptations. Iterative RE-AIM can be repeated at multiple time points customized to each project and involves the following activities: identification of key implementation partners; rating importance of and progress on each RE-AIM dimension (reach, effectiveness, adoption, implementation, and maintenance); use of summary data on ratings to identify one or two RE-AIM dimensions for adaptations and implementation strategies; and evaluation of progress and impact of adaptations. We summarize recent and ongoing Iterative RE-AIM applications across multiple care coordination and pain management projects within the Veterans Health Administration, a hypertension control trial in Guatemala, a hospital-based lung ultrasound implementation pilot, and a colorectal cancer screening program in underserved communities.

Results

Iterative RE-AIM appears feasible, helpful, and broadly applicable across diverse health care issues, interventions, contexts, and populations. In general, the RE-AIM dimension showing the largest gap between importance and progress has been Reach. The dimensions most frequently selected for improvement have been Reach and Implementation. We discuss commonalities, differences and lessons learned across these various applications of Iterative RE-AIM. Challenges include having objective real time data on which to make decisions, having key implementation staff available for all assessments, and rapidly scoring and providing actionable feedback. We discuss print and online resources and materials to support Iterative RE-AIM.

Conclusions

The use of Iterative RE-AIM to guide and support understanding of adaptations has proven feasible across diverse projects and in multiple case studies, but there are still questions about its strengths, limitations, essential components, efficiency, comparative effectiveness, and delivery details. Future directions include investigating the optimal frequency and timing for iterative applications; adding contextual assessments; developing more continuous and rapid data on which to make adaptation decisions; identifying opportunities to enhance health equity; and determining the level of facilitation that is most cost-effective.

Using a longitudinal multi-method approach to document, assess, and understand adaptations in the Veterans Health Administration Advanced Care Coordination program

Background

Understanding adaptations supports iterative refinement of the implementation process and informs scale out of programs. Systematic documentation of adaptations across the life course of programs is not routinely done, and efficient capture of adaptations in real world studies is not well understood.

Methods

We used a multi-method longitudinal approach to systematically document adaptations during pre-implementation, implementation, and sustainment for the Veteran Health Administration (VA) Advanced Care Coordination program. This approach included documenting adaptations through a real-time tracking instrument, process maps, Implementation and Evaluation (I&E) team meeting minutes, and adaptation interviews. Data collection was guided by the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) enhanced framework for reporting adaptations and modifications to evidence-based interventions (FRAME) model. Adaptations were evaluated across 9 categories, and analytic team consensus and member-checking were used to validate the results.

Results

A total of 144 individual adaptations were identified across two implementation sites and the four data sources; analytic team consensus and member-checking processes resulted in 50 unique adaptations. Most adaptations took place during the early implementation and mid-implementation phases and were: 1) planned; 2) made to address changes in program delivery; 3) made to extend a component; 4) related to the core component of the intervention concerning notification of the community emergency department visit; 5) initiated by the entire or most of the I&E team; 6) made on the basis of: pragmatic/practical considerations; 7) made with an intent to improve implementation domain (to make the intervention delivered more consistently; to better fit the local practice, patient flow or Electronic Health Record (EHR) and/or for practical reasons); 8) a result of internal influences; 9) perceived to impact the RE-AIM implementation dimension (consistent delivery of quality care or costs). I&E team meeting minutes and process maps captured the highest numbers of unique adaptations (n = 19 and n = 13, respectively).

Conclusion

Our longitudinal, multi-method approach provided a feasible way to collect adaptations data through engagement of multiple I&E team members, allowing and a broader understanding of adaptations that took place. Recommendations for future research include pragmatic assessment of the impact of adaptations and meaningful data collection without overburdening the implementing teams and front-line staff.