The Veterans Health Administration (VHA) Innovators Network: Evaluation design, methods and lessons learned through an embedded research approach

Diffusion of excellence: evaluating a system to identify, replicate, and spread promising innovative practices across the Veterans health administration

Introduction

The Veterans Health Administration (VHA) Diffusion of Excellence (DoE) program provides a system to identify, replicate, and spread promising practices across the largest integrated healthcare system in the United States. DoE identifies innovations that have been successfully implemented in the VHA through a Shark Tank style competition. VHA facility and regional directors bid resources needed to replicate promising practices. Winning facilities/regions receive external facilitation to aid in replication/implementation over the course of a year. DoE staff then support diffusion of successful practices across the nationwide VHA.

Methods

Organized around the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework, we summarize results of an ongoing long-term mixed-methods implementation evaluation of DoE. Data sources include: Shark Tank application and bid details, tracking practice adoptions through a Diffusion Marketplace, characteristics of VHA facilities, focus groups with Shark Tank bidders, structured observations of DoE events, surveys of DoE program participants, and semi-structured interviews of national VHA program office leaders, VHA healthcare system/facility executives, practice developers, implementation teams and facilitators.

Results

In the first eight Shark Tanks (2016-2022), 3,280 Shark Tank applications were submitted; 88 were designated DoE Promising Practices (i.e., practices receive facilitated replication). DoE has effectively spread practices across the VHA, with 1,440 documented instances of adoption/replication of practices across the VHA. This includes 180 adoptions/replications in facilities located in rural areas. Leadership decisions to adopt innovations are often based on big picture considerations such as constituency support and linkage to organizational goals. DoE Promising Practices that have the greatest national spread have been successfully replicated at new sites during the facilitated replication process, have close partnerships with VHA national program offices, and tend to be less expensive to implement. Two indicators of sustainment indicate that 56 of the 88 Promising Practices are still being diffused across the VHA; 56% of facilities originally replicating the practices have sustained them, even up to 6 years after the first Shark Tank.

Conclusion

DoE has developed a sustainable process for the identification, replication, and spread of promising practices as part of a learning health system committed to providing equitable access to high quality care.

Supporting Veteran's Administration Medical Center Directors' Decisions When Adopting Innovative Practices: Development and Implementation of the "QuickView" and "WishList" Tools

Background Since 2015, the Veterans Health Administration (VHA) Diffusion of Excellence Program has supported spread of practices developed by frontline employees. Shark Tank-style competitions encourage "Sharks" nationwide (VHA medical center/regional directors) to bid for the opportunity to implement practices at their institutions. Methods The authors evaluated bidding strategies (2016-2020), developing the "QuickView" practice comparator to promote informed bidding. Program leaders distributed QuickView and revised versions in subsequent competitions. Our team utilized in-person observation, online chats after the competition, bidder interviews, and bid analysis to evaluate QuickView use. Bids were ranked based on demonstrated understanding of resources required for practice implementation. Results Sharks stated that QuickView supported preparation before the competition and suggested improvements. Our revised tool reported necessary staff time and incorporated a "WishList" from practice finalists detailing minimum requirements for successful implementation. Bids from later years reflected increased review of facilities' current states before the competition and increased understanding of the resources needed for implementation. Percentage of bids describing local need for the practice rose from 2016 to 2020: 4.7% (6/127); 62.1% (54/87); 78.3% (36/46); 80.6% (29/36); 89.7% (26/29). Percentage of bids committing specific resources rose following QuickView introduction: 81.1% (103/127) in 2016, 69.0% (60/87) in 2017, then 73.9% (34/46) in 2018, 88.9% (32/36) in 2019, and 89.7% (26/29) in 2020. Discussion In the years following QuickView/WishList implementation, bids reflected increased assessment before the competition of both local needs and available resources. Conclusion Selection of a new practice for implementation requires an understanding of local need, necessary resources, and fit. QuickView and WishList appear to support these determinations.

Rolling out PRIDE in All Who Served: Barriers and Facilitators for Sites Implementing an LGBTQ+ Health Education Group for Military Veterans

BACKGROUND/OBJECTIVE

The Veterans Health Administration (VHA) PRIDE in All Who Served health education group (PRIDE) was developed to improve health equity and access to care for military veterans who are lesbian, gay, bisexual, transgender, queer, and/or other sexual/gender-diverse identities (LGBTQ+). This 10-week program rapidly spread to over 30 VHA facilities in 4 years. Veterans receiving PRIDE experience improved LGBTQ+ identity-related resilience and reductions in suicide attempt likelihood. Despite PRIDE's rapid spread across facilities, information is lacking on implementation determinants. The current study's goal was to clarify determinants of PRIDE group implementation and sustainment.

METHODS

A purposive sample of VHA staff (N = 19) with experience delivering or implementing PRIDE completed teleconference interviews January-April 2021. The interview guide was informed by the Consolidated Framework for Implementation Research. Rapid qualitative matrix analysis was completed with methods to ensure rigor (e.g., triangulation and investigator reflexivity).

RESULTS

Key barriers and facilitators of PRIDE implementation were heavily related to facility inner setting (what is happening inside the facility), including implementation readiness (e.g., leadership support for LGBTQ+-affirming programming, access to LGBTQ+-affirming care training) and facility culture (e.g., systemic anti-LGBTQ+ stigma). Several implementation process facilitators enhanced engagement at sites, such as a centrally facilitated PRIDE learning collaborative and a formal process of contracting/training for new PRIDE sites.

DISCUSSION/CONCLUSION

Although aspects of the outer setting and larger societal influences were mentioned, the majority of factors impacting implementation success were at the VHA facility level and therefore may be more readily addressable through tailored implementation support. The importance of LGBTQ+ equity at the facility level indicates that implementation facilitation should ideally address institutional equity in addition to implementation logistics. Combining effective interventions with attention to local implementation needs will be required before LGBTQ+ veterans in all areas will benefit from PRIDE and other health equity-focused interventions.

The journey to a learning health system in primary care: a qualitative case study utilising an embedded research approach

BACKGROUND

Healthcare systems may be resilient and adaptive, but they are not fit for purpose in their current state. Increasing threats to health system sustainability have underscored the need to move towards a learning health system in which research and data are used routinely in clinical practice to facilitate system improvement. This study aimed to establish which elements of the learning health system were being realised within a university-based general practice and determine acceptability from staff to embrace further the transition towards a learning health system.

METHODS

Semi-structured interviews were conducted with practice staff, including clinical and administrative staff, to determine the current state of the learning health system in the practice. An embedded researcher was placed within the general practice on a part-time basis to investigate the learning health system model. Interviews were transcribed and thematically analysed based on the National Academy of Medicine's framework of learning health systems.

RESULTS

In total, 32 (91%) practice staff were interviewed, comprising general practitioners (n = 15), nurses (n = 3), administrative staff (n = 13), and a psychologist (n = 1). Participants indicated that the practice was operating with several characteristics of a learning health system (e.g., emphasising science and informatics; focusing on patient-clinician partnerships; applying incentives; supporting a continuous learning culture; and establishing structures and governance for learning). These measures were supported by the university-based setting, and resultant culture of learning. Nevertheless, there were areas of the practice where the learning health system could be strengthened, specifically relating to the use of patient data and informatics. Staff generally expressed willingness to engage with the process of strengthening the learning health system within their practice.

CONCLUSION

Although the idea of a learning health system has been gaining traction in recent years, there are comparatively few empirical studies presented in the literature. This research presents a case study of a general practice that is operating as a learning health system and highlights the utility of using the learning health system framework.

Balancing reality in embedded research and evaluation: Low vs high embeddedness

Embedding research and evaluation into organizations is one way to generate “practice‐based” evidence needed to accelerate implementation of evidence‐based innovations within learning health systems. Organizations and researchers/evaluators vary greatly in how they structure and operationalize these collaborations. One key aspect is the degree of embeddedness: from low embeddedness where researchers/evaluators are located outside organizations (eg, outside evaluation consultants) to high embeddedness where researchers/evaluators are employed by organizations and thus more deeply involved in program evolution and operations. Pros and cons related to the degree of embeddedness (low vs high) must be balanced when developing these relationships. We reflect on this process within the context of an embedded, mixed‐methods evaluation of the Veterans Health Administration (VHA) Diffusion of Excellence (DoE) program. Considerations that must be balanced include: (a) low vs high alignment of goals; (b) low vs high involvement in strategic planning; (c) observing what is happening vs being integrally involved with programmatic activities; (d) reporting findings at the project's end vs providing iterative findings and recommendations that contribute to program evolution; and (e) adhering to predetermined aims vs adapting aims in response to evolving partner needs.