Long-Term Evaluation of a Course on Evidence-Based Public Health in the U.S. and Europe

Implementing evidence ecosystems in the public health service: Development of a framework for designing tailored training programs

The COVID-19 pandemic has highlighted the importance of local evidence ecosystems in which academia and practice in the Public Health Service (PHS) are interconnected. However, appropriate organizational structures and well-trained staff are lacking and evidence use in local public health decision-making has to be integrated into training programs in Germany. To address this issue, we developed a framework incorporating a toolbox to conceptualize training programs designed to qualify public health professionals for working at the interface between academia and practice. We conducted a scoping review of training programs, key-informant interviews with public health experts, and a multi-professional stakeholder workshop and triangulated their output. The resulting toolbox consists of four core elements, encompassing 15 parameters: (1) content-related aspects, (2) context-related aspects, (3) aspects relevant for determining the training format, and (4) aspects relevant for consolidation and further development. Guiding questions with examples supports the application of the toolbox. Additionally, we introduced a how-to-use guidance to streamline the creation of new training programs, fostering knowledge transfer at the academia-practice interface, equipping public health researchers and practitioners with relevant skills for needs-based PHS research. By promoting collaborative training development across institutions, our approach encourages cross-institutional cooperation, enhances evidence utilization, and enables efficient resource allocation. This collaborative effort in developing training programs within local evidence ecosystems not only strengthens the scientific and practical impact but also lays a foundation for implementing complex public health measures effectively at the local level.

Understanding Misimplementation in U.S. State Health Departments: An Agent-Based Model

INTRODUCTION

The research goal of this study is to explore why misimplementation occurs in public health agencies and how it can be reduced. Misimplementation is ending effective activities prematurely or continuing ineffective ones, which contributes to wasted resources and suboptimal health outcomes.

METHODS

The study team created an agent-based model that represents how information flow, filtered through organizational structure, capacity, culture, and leadership priorities, shapes continuation decisions. This agent-based model used survey data and interviews with state health department personnel across the U.S. between 2014 and 2020; model design and analyses were conducted with substantial input from stakeholders between 2019 and 2021. The model was used experimentally to identify potential approaches for reducing misimplementation.

RESULTS

Simulations showed that increasing either organizational evidence-based decision-making capacity or information sharing could reduce misimplementation. Shifting leadership priorities to emphasize effectiveness resulted in the largest reduction, whereas organizational restructuring did not reduce misimplementation.

CONCLUSIONS

The model identifies for the first time a specific set of factors and dynamic pathways most likely driving misimplementation and suggests a number of actionable strategies for reducing it. Priorities for training the public health workforce include evidence-based decision making and effective communication. Organizations will also benefit from an intentional shift in leadership decision-making processes. On the basis of this initial, successful application of agent-based model to misimplementation, this work provides a framework for further analyses.

Program adaptation by health departments

Introduction

The dissemination of evidence-based interventions (i.e., programs, practices, and policies) is a core function of US state health departments (SHDs). However, interventions are originally designed and tested with a specific population and context. Hence, adapting the intervention to meet the real-world circumstances and population's needs can increase the likelihood of achieving the expected health outcomes for the target population from the implemented intervention. This study identified how SHD employees decide to adapt public health programs and what influences decisions on how to adapt them.

Materials and methods

SHD employees (n = 45) were interviewed using a qualitative semi-structured interview guide. Telephone interviews were audio-recorded and transcribed verbatim. The transcripts were consensus-coded and themes were identified using thematic analysis. Several themes aligned with the Model for Adaptation Design and Impact.

Results

Data, outcomes, and health department evaluations influenced decisions to adapt a program (pre-adaptation), and reasons to adapt a program included organizational and sociopolitical contextual factors. SHD middle-level managers, program managers and staff, and local agencies were involved in the decisions to adapt the programs. Finally, the goals for adapting a program included enhancing effectiveness/outcomes, reach and satisfaction with the program; funding; and partner engagement. After SHD employees decided to adapt a program, data and evidence guided the changes. Program staff and evaluators were engaged in the adaptation process. Program managers consulted partners to gather ideas on how best to adapt a program based on partners' experiences implementing the program and obtaining community input. Lastly, program managers also received input on adapting content and context from coalition meetings and periodic technical assistance calls.

Discussion

The findings related to decisions to adapt public health programs provide practitioners with considerations for adapting them. Findings reaffirm the importance of promoting public health competencies in program evaluation and adaptation, as well as systematically documenting and evaluating the adaptation processes. In addition, the themes could be studied in future research as mechanisms, mediators, and moderators to implementation outcomes.

How to "Start Small and Just Keep Moving Forward": Mixed Methods Results From a Stepped-Wedge Trial to Support Evidence-Based Processes in Local Health Departments

Background

Local health departments (LHDs) in the United States are charged with preventing disease and promoting health in their respective communities. Understanding and addressing what supports LHD's need to foster a climate and culture supportive of evidence-based decision making (EBDM) processes can enhance delivery of effective practices and services.

Methods

We employed a stepped-wedge trial design to test staggered delivery of implementation supports in 12 LHDs (Missouri, USA) to expand capacity for EBDM processes. The intervention was an in-person training in EBDM and continued support by the research team over 24 months (March 2018-February 2020). We used a mixed-methods approach to evaluate: (1) individuals' EBDM skills, (2) organizational supports for EBDM, and (3) administered evidence-based interventions. LHD staff completed a quantitative survey at 4 time points measuring their EBDM skills, organizational supports, and evidence-based interventions. We selected 4 LHDs with high contact and engagement during the intervention period to interview staff (n = 17) about facilitators and barriers to EBDM. We used mixed-effects linear regression to examine quantitative survey outcomes. Interviews were transcribed verbatim and coded through a dual independent process.

Results

Overall, 519 LHD staff were eligible and invited to complete quantitative surveys during control periods and 593 during intervention (365 unique individuals). A total of 434 completed during control and 492 during intervention (83.6 and 83.0% response, respectively). In both trial modes, half the participants had at least a master's degree (49.7-51.7%) and most were female (82.1-83.8%). No significant intervention effects were found in EBDM skills or in implementing evidence-based interventions. Two organizational supports scores decreased in intervention vs. control periods: awareness (-0.14, 95% CI -0.26 to -0.01, p < 0.05) and climate cultivation (-0.14, 95% CI -0.27 to -0.02, p < 0.05) but improved over time among all participants. Interviewees noted staff turnover, limited time, resources and momentum as challenges to continue EBDM work. Setting expectations, programmatic reviews, and pre-existing practices were seen as facilitators.

Conclusions

Challenges (e.g., turnover, resources) may disrupt LHDs' abilities to fully embed organizational processes which support EBDM. This study and related literature provides understanding on how best to support LHDs in building capacity to use and sustain evidence-based practices.