Expect the Unexpected: A Qualitative Study of the Ripple Effects of Children's Mental Health Services Implementation Efforts

Conceptualizing patient-level adverse effects in implementation trials

BACKGROUND

Identifying and monitoring adverse effects (AEs) are integral to ensuring patient safety in clinical trials. Research sponsors and regulatory bodies have put into place a variety of policies and procedures to guide researchers in protecting patient safety during clinical trials. However, it remains unclear how these policies and procedures should be adapted for trials in implementation science. As a starting point, we develop a conceptual model that traces causal pathways leading from implementation strategies to AEs, propose a definition and classification of such effects, and provide recommendations for monitoring and oversight.

MAIN TEXT

We propose four major types of adverse effects for implementation trials. First, we characterize implementation strategies that lead to "proper use" of an intervention that align with AEs as conceptualized and reported in clinical trials. Second, we characterize a strategy's AEs mediated through "misuse" which involves inappropriate utilization of an evidence-based intervention (EBI). Third, we characterize a strategy which focuses on one EBI and may inadvertently cause the inappropriate discontinuation or "disuse" of other EBIs already in place, thus inducing AEs. Finally, we characterize strategies that may cause AEs by reducing the use of an EBI in the target population (i.e., "nonuse"). Based on these considerations, we propose an extended definition of adverse effects that includes harms that are causally related to implementation strategies, termed Implementation strategy Adverse Effects (IAEs). We recommend researchers, oversight committees, sponsors, and other stakeholders work together prior to trials to determine the best approaches for identifying, monitoring, and reporting IAEs.

CONCLUSIONS

In this paper, we develop a conceptual model to identify four types of AEs in implementation trials clarifying the mechanisms linking implementation strategies to patterns of use of the EBI and potential patient-level harms. We propose a new definition that links implementation strategies to AEs that can be used to guide conceptualization, monitoring, and oversight of potential harms in future implementation trials. Our work represents an important step towards understanding adverse effects in implementation trials and lays the groundwork for future advancement in the conceptualization of other types of adverse effects (e.g., harms to providers) encountered in implementation trials.

Leveraging group model building to operationalize implementation strategies across implementation phases: an exemplar related to child maltreatment intervention selection

BACKGROUND

Implementation strategies can help support the adoption and implementation of health interventions that are appropriate for a local context and acceptable to decision makers and community members. Implementation strategies should be designed to handle the complexity of the multi-level, dynamic contexts in which interventions are implemented. Systems science theories and methods explicitly attend to complexity and can be valuable for specifying implementation strategies. Group Model Building (GMB) combines research partner engagement strategies with systems science to support researchers' and partners' learning about complex problems and to identify solutions through consensus. This paper specifies how GMB can operationalize implementation strategies - methods for supporting evidence implementation in real-world practice - and describes how GMB can aid in selecting and tailoring both health interventions and implementation strategies. A case study in child maltreatment prevention planning is provided to illustrate how GMB was used to specify the "actions" - strategy activities - for three implementation strategies (conduct local consensus discussions; build a coalition; model and simulate change) during the earliest implementation phases, with the goal of supporting intervention selection decisions. Examples are provided of generalizable research products that can be produced concurrently through GMB, in addition to contextually-driven implementation support.

METHODS

Participants (n = 8) were engaged over four sessions using tailored GMB activities. Participants generated a qualitative system dynamics model that described their theory of change for how to prevent child maltreatment in their communities. This theory of change reflected a dynamic understanding of the interconnected determinants of child maltreatment.

RESULTS

GMB was acceptable to participants and resulted in products that could be used for implementation planning (e.g., to model and simulate change) and future research. GMB fostered trust and idea sharing between participants.

CONCLUSION

GMB can facilitate learning about which outcomes are (or are not) impacted by interventions, which resources and approaches are required for quality implementation (e.g., implementation strategies), and tradeoffs in outcomes and resources between interventions. GMB also provides a structured, effective process to generate a shared implementation vision amongst participants. Lessons learned include methods for developing trust with and between participants, and the need for researchers to tailor GMB actions for participant and project needs.

Impact of research activity on performance of general practices: a qualitative study

BACKGROUND

There is evidence that engaging in research is directly associated with better performance. If this relationship is to be strengthened, it is necessary to understand the mechanisms that might underlie that relationship.

AIM

To explore the perspectives of staff and wider stakeholders about mechanisms by which research activity may impact on the performance of general practices.

DESIGN & SETTING

Qualitative study using semi-structured interviews with general practice professionals and wider stakeholders in England.

METHOD

Individual interviews with 41 purposively sampled staff in 'research-ready' or 'research-active' general practices, and 21 other stakeholders. Interviews were independently coded by three researchers using a framework approach.

RESULTS

Participants described potential 'direct' and 'indirect' impacts on their work. 'Direct' impacts included improved knowledge and skills that could change practice work (for example, additional records searches for particular conditions); bringing in additional resources (for example, access to investigations or staff); and improving relationships with patients. 'Indirect' impacts included job satisfaction (for example, perception of practice as a centre of excellence and innovation, and the variety afforded by research activity reducing burnout); and staff recruitment (increasing the attractiveness of the practice as a place to work). Responders identified few negative impacts.

CONCLUSION

Staff and stakeholders identified a range of potential impacts of research activity on practice performance, with impacts on their working lives most salient. Negative impacts were not generally raised. Nevertheless, responders generally discussed potential impacts rather than providing specific examples of those impacts. This may reflect the type of research activity conducted in general practice, often led by external collaborators.

A research agenda to advance the study of implementation mechanisms

BACKGROUND

Implementation science scholars have made significant progress identifying factors that enable or obstruct the implementation of evidence-based interventions, and testing strategies that may modify those factors. However, little research sheds light on how or why strategies work, in what contexts, and for whom. Studying implementation mechanisms-the processes responsible for change-is crucial for advancing the field of implementation science and enhancing its value in facilitating equitable policy and practice change. The Agency for Healthcare Research and Quality funded a conference series to achieve two aims: (1) develop a research agenda on implementation mechanisms, and (2) actively disseminate the research agenda to research, policy, and practice audiences. This article presents the resulting research agenda, including priorities and actions to encourage its execution.

METHOD

Building on prior concept mapping work, in a semi-structured, 3-day, in-person working meeting, 23 US-based researchers used a modified nominal group process to generate priorities and actions for addressing challenges to studying implementation mechanisms. During each of the three 120-min sessions, small groups responded to the prompt: "What actions need to be taken to move this research forward?" The groups brainstormed actions, which were then shared with the full group and discussed with the support of facilitators trained in structured group processes. Facilitators grouped critical and novel ideas into themes. Attendees voted on six themes they prioritized to discuss in a fourth, 120-min session, during which small groups operationalized prioritized actions. Subsequently, all ideas were collated, combined, and revised for clarity by a subset of the authorship team.

RESULTS

From this multistep process, 150 actions emerged across 10 priority areas, which together constitute the research agenda. Actions included discrete activities, projects, or products, and ways to shift how research is conducted to strengthen the study of implementation mechanisms.

CONCLUSIONS

This research agenda elevates actions to guide the selection, design, and evaluation of implementation mechanisms. By delineating recommended actions to address the challenges of studying implementation mechanisms, this research agenda facilitates expanding the field of implementation science, beyond studying what works to how and why strategies work, in what contexts, for whom, and with which interventions.

Specifying cross-system collaboration strategies for implementation: a multi-site qualitative study with child welfare and behavioral health organizations

BACKGROUND

Cross-system interventions that integrate health, behavioral health, and social services can improve client outcomes and expand community impact. Successful implementation of these interventions depends on the extent to which service partners can align frontline services and organizational operations. However, collaboration strategies linking multiple implementation contexts have received limited empirical attention. This study identifies, describes, and specifies multi-level collaboration strategies used during the implementation of Ohio Sobriety Treatment and Reducing Trauma (Ohio START), a cross-system intervention that integrates services across two systems (child welfare and evidence-based behavioral health services) for families that are affected by co-occurring child maltreatment and parental substance use disorders.

METHODS

In phase 1, we used a multi-site qualitative design with 17 counties that implemented Ohio START. Qualitative data were gathered from 104 staff from child welfare agencies, behavioral health treatment organizations, and regional behavioral health boards involved in implementation via 48 small group interviews about collaborative approaches to implementation. To examine cross-system collaboration strategies, qualitative data were analyzed using an iterative template approach and content analysis. In phase 2, a 16-member expert panel met to validate and specify the cross-system collaboration strategies identified in the interviews. The panel was comprised of key child welfare and behavioral health partners and scholars.

RESULTS

In phase 1, we identified seven cross-system collaboration strategies used for implementation. Three strategies were used to staff the program: (1) contract for expertise, (2) provide joint supervision, and (3) co-locate staff. Two strategies were used to promote service access: (4) referral protocols and (5) expedited access agreements. Two strategies were used to align case plans: (6) shared decision-making meetings, and (7) sharing data. In phase 2, expert panelists specified operational details of the cross-system collaboration strategies, and explained the processes by which strategies were perceived to improve implementation and service system outcomes.

CONCLUSIONS

We identified a range of cross-system collaboration strategies that show promise for improving staffing, service access, and case planning. Leaders, supervisors, and frontline staff used these strategies during all phases of implementation. These findings lay the foundation for future experimental and quasi-experimental studies that test the effectiveness of cross-system collaboration strategies.

Developing ethical standards for dissemination and implementation research: a roadmap for consensus and guidance

BACKGROUND

As a relatively new field, dissemination and implementation research has not been included as a separate study design category for ethical consideration compared with clinical and social/behavioral research, yet it should be based on unique study designs, targets of intervention, and corresponding risks.

MAIN TEXT

Research teams conducting dissemination and implementation research have raised important questions related to the responsible conduct of research such as collecting informed consent, site monitoring, identifying and mitigating risks of unintended consequences, and adverse event ascertainment and reporting in dissemination and implementation research. In this commentary, we highlight the need for guidance and consensus standards on ethical issues in dissemination and implementation research and describe some ethical domains and relevant questions in dissemination and implementation research. Additionally, we propose a process for conceptual development and a research agenda to create consensus standards for the responsible conduct of research for dissemination and implementation research.

CONCLUSION

Thorough research is needed to understand the depth of ethical issues in dissemination and implementation research. A consensus-seeking process will be needed to develop new bioethical standards that carefully identify, measure, and mitigate unintended consequences in dissemination and implementation research.

Strategies for enacting health policy codesign: a scoping review and direction for research

BACKGROUND

Strategies for supporting evidence-informed health policy are a recognized but understudied area of policy dissemination and implementation science. Codesign describes a set of strategies potentially well suited to address the complexity presented by policy formation and implementation. We examine the health policy literature describing the use of codesign in initiatives intended to combine diverse sources of knowledge and evidence in policymaking.

METHODS

The search included PubMed, MEDLINE, PsychInfo, CINAHL, Web of Science, and Google Scholar in November 2022 and included papers published between 1996 and 2022. Terms included codesign, health, policy, and system terminology. Title and abstracts were reviewed in duplicate and included if efforts informed policy or system-level decision-making. Extracted data followed scoping review guidelines for location, evaluation method, health focus, codesign definition, description, level of health system user input, sectors involved, and reported benefits and challenges.

RESULTS

From 550 titles, 23 citations describing 32 policy codesign studies were included from multiple continents (Australia/New Zealand, 32%; UK/Europe, 32%; South America, 14%; Africa, 9%; USA/Canada 23%). Document type was primarily case study (77%). The area of health focus was widely distributed. Policy type was more commonly little p policy (47%), followed by big p policy (25%), and service innovations that included policy-enabled funding (25%). Models and frameworks originated from formal design (e.g., human-centered or participatory design (44%), political science (38%), or health service research (16%). Reported outcomes included community mobilization (50%), policy feasibility (41%), improved multisector alignment (31%), and introduction of novel ideas and critical thinking (47%). Studies engaging policy users in full decision-making roles self-reported higher levels of community mobilization and community needs than other types of engagement.

DISCUSSION

Policy codesign is theoretically promising and is gaining interest among diverse health sectors for addressing the complexity of policy formation and implementation. The maturity of the science is just emerging. We observed trends in the association of codesign strategies and outcomes that suggests a research agenda in this area could provide practical insights for tailoring policy codesign to respond to local contextual factors including values, needs, and resources.