Participatory logic modeling in a multi-site initiative to advance implementation science

BACKGROUND

Logic models map the short-term and long-term outcomes that are expected to occur with a program, and thus are an essential tool for evaluation. Funding agencies, especially in the United States (US), have encouraged the use of logic models among their grantees. They also use logic models to clarify expectations for their own funding initiatives. It is increasingly recognized that logic models should be developed through a participatory approach which allows input from those who carry out the program being evaluated. While there are many positive examples of participatory logic modeling, funders have generally not engaged grantees in developing the logic model associated with their own initiatives. This article describes an instance where a US funder of a multi-site initiative fully engaged the funded organizations in developing the initiative logic model. The focus of the case study is Implementation Science Centers in Cancer Control (ISC3), a multi-year initiative funded by the National Cancer Institute.

METHODS

The reflective case study was collectively constructed by representatives of the seven centers funded under ISC3. Members of the Cross-Center Evaluation (CCE) Work Group jointly articulated the process through which the logic model was developed and refined. Individual Work Group members contributed descriptions of how their respective centers reviewed and used the logic model. Cross-cutting themes and lessons emerged through CCE Work Group meetings and the writing process.

RESULTS

The initial logic model for ISC3 changed in significant ways as a result of the input of the funded groups. Authentic participation in the development of the logic model led to strong buy-in among the centers, as evidenced by their utilization. The centers shifted both their evaluation design and their programmatic strategy to better accommodate the expectations reflected in the initiative logic model.

CONCLUSIONS

The ISC3 case study demonstrates how participatory logic modeling can be mutually beneficial to funders, grantees and evaluators of multi-site initiatives. Funded groups have important insights about what is feasible and what will be required to achieve the initiative's stated objectives. They can also help identify the contextual factors that either inhibit or facilitate success, which can then be incorporated into both the logic model and the evaluation design. In addition, when grantees co-develop the logic model, they have a better understanding and appreciation of the funder's expectations and thus are better positioned to meet those expectations.

Surfacing, as well as testing, "elliptical assumptions" in a theory of change: Principled discovery

While the testing of known assumptions in a theory of change is important, so too is the discovery or surfacing of previously unrecognized assumptions. This paper describes and illustrates the surfacing of "elliptical assumptions," which involve the unknown ingredients that are necessary for a program to be effective. Identifying the ingredients of program success is important for several reasons, which include (a) guiding the development of an improved theory of change, which in turn can guide program improvement, and (b) informing efforts to transfer the program to other settings and populations. However, when an observed pattern, such as differential program effects, points to the existence of a previously unidentified but important ingredient, this might be a just-so story, a seemingly compelling but inaccurate account. Accordingly, the testing of previously unidentified elliptical assumptions is recommended and illustrated.

COVID-19 - an opportunity to improve access to primary care through organizational innovations? A qualitative multiple case study in Quebec and Nova Scotia (Canada)

Background

COVID-19 catalyzed a rapid and substantial reorganization of primary care, accelerating the spread of existing strategies and fostering a proliferation of innovations. Access to primary care is an essential component of a healthcare system, particularly during a pandemic. We describe organizational innovations aiming to improve access to primary care and related contextual changes during the first 18 months of the COVID-19 pandemic in two Canadian provinces, Quebec and Nova Scotia.

Methods

We conducted a multiple case study based on 63 semi-structured interviews (n = 33 in Quebec, n = 30 in Nova Scotia) conducted between October 2020 and May 2021 and 71 documents from both jurisdictions. We recruited a diverse range of provincial and regional stakeholders (e.g., policy-makers, decision-makers, family physicians, nurses) involved in reorganizing primary care during the COVID-19 pandemic using purposeful sampling (e.g., based on role, region). Interviews were transcribed verbatim and thematic analysis was conducted in NVivo12. Emerging results were discussed by team members to identify salient themes and organized into logic models.

Results

We identified and analyzed six organizational innovations. Four of these – centralized public online booking systems, centralized access centers for unattached patients, interim primary care clinics for unattached patients, and a community connector to health and social services for older adults – pre-dated COVID-19 but were accelerated by the pandemic context. The remaining two innovations were created to specifically address pandemic-related needs: COVID-19 hotlines and COVID-dedicated primary healthcare clinics.

Innovation spread and proliferation was influenced by several factors, such as a strengthened sense of community amongst providers, decreased patient demand at the beginning of the first wave, renewed policy and provider interest in population-wide access (versus attachment of patients only), suspended performance targets (e.g., continuity ≥80%) in Quebec, modality of care delivery, modified fee codes, and greater regional flexibility to implement tailored innovations.

Conclusion

COVID-19 accelerated the uptake and creation of organizational innovations to potentially improve access to primary healthcare, removing, at least temporarily, certain longstanding barriers. Many stakeholders believed this reorganization would have positive impacts on access to primary care after the pandemic. Further studies should analyze the effectiveness and sustainability of innovations adapted, developed, and implemented during the COVID-19 pandemic.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-022-08140-w.

Exploring the implementation and underlying mechanisms of centralized referral systems to access specialized health services in Quebec

BACKGROUND

In 2016, Quebec, a Canadian province, implemented a program to improve access to specialized health services (Accès priorisé aux services spécialisés (APSS)), which includes single regional access points for processing requests to such services via primary care (Centre de répartition des demandes de services (CRDS)). Family physicians fill out and submit requests for initial consultations with specialists using a standardized form with predefined prioritization levels according to listed reasons for consultations, which is then sent to the centralized referral system (the CRDS) where consultations with specialists are assigned. We 1) described the APSS-CRDS program in three Quebec regions using logic models; 2) compared similarities and differences in the components and processes of the APSS-CRDS models; and 3) explored contextual factors influencing the models' similarities and differences.

METHODS

We relied on a qualitative study to develop logic models of the implemented APSS-CRDS program in three regions. Semi-structured interviews with health administrators (n = 9) were conducted. The interviews were analysed using a framework analysis approach according to the APSS-CRDS's components included in the initially designed program, Mitchell and Lewis (2003)'s logic model framework, and Chaudoir and colleagues (2013)'s framework on contextual factors' influence on an innovation's implementation.

RESULTS

Findings show the APSS-CRDS program's regional variability in the implementation of its components, including its structure (centralized/decentralized), human resources involved in implementation and operation, processes to obtain specialists' availability and assess/relay requests, as well as monitoring methods. Variability may be explained by contextual factors' influence, like ministerial and medical associations' involvement, collaborations, the context's implementation readiness, physician practice characteristics, and the program's adaptability.

INTERPRETATION

Findings are useful to inform decision-makers on the design of programs like the APSS-CRDS, which aim to improve access to specialists, the essential components for the design of these types of interventions, and how contextual factors may influence program implementation. Variability in program design is important to consider as it may influence anticipated effects, a next step for the research team. Results may also inform stakeholders should they wish to implement similar programs to increase access to specialized health services via primary care.

How should we evaluate sweetened beverage tax policies? A review of worldwide experience

Over 45 jurisdictions globally have implemented sweetened beverage taxes. Researchers and policymakers need to assess whether and how these taxes change beverage demand and supply, their intended and unanticipated health, economic and equity impacts. Lessons from such evaluations can maximise the policies' success and impact on non-communicable disease prevention globally. We discuss key theoretical, design and methodological considerations to help policymakers, funders and researchers commission and conduct rigorous evaluations of these policies and related disease prevention efforts. We encourage involving the perspectives of various stakeholders on what evaluations are needed given the specific context, what data and methods are appropriate, readily available or can be collected within time and budget constraints. A logic model /conceptual system map of anticipated implications across sectors and scales should help identify optimal study design, analytical techniques and measures. These models should be updated when synthesising findings across diverse methods and integrating findings across subpopulations using similar methods.

Program Evaluation Through the Use of Logic Models

INTRODUCTION

In order to effectively design and evaluate the effectiveness of a new clinical program or intervention, pharmacists must be equipped with the skills and knowledge that are obtained by familiarity and use of a logic model. Currently, most pharmacy school curricula do not include logic model exercises to instill these necessary skills into the knowledge base of doctor of pharmacy students. This report provides understanding of how a logic model can be permanently implemented into pharmacy curricula in order to develop critical thinking skills that will allow students to become more well-rounded in their future practice of pharmacy.

METHODS

A 23-point questionnaire was developed by the principal investigator, primarily based on feedback from the student reflection papers and areas of interest with the author's prior experience and use of logic models. They were distributed to assess the knowledge, attitudes, and perspectives of students enrolled or previously enrolled in the course.

RESULTS

Questionnaires were received from 128 students, representing approximately 32% of those provided the opportunity to participate. The majority of students (72.98%) viewed the potential benefits of learning about logic models favorably. Overall, 64.86% of students agreed that the experience gained through constructing their own logic model was an intellectually stimulating activity.

CONCLUSIONS

The logic model is an effective tool that can be used to teach pharmacy students how planned program development would contribute to combating various public health issues.

The Implementation Research Logic Model: a method for planning, executing, reporting, and synthesizing implementation projects

BACKGROUND

Numerous models, frameworks, and theories exist for specific aspects of implementation research, including for determinants, strategies, and outcomes. However, implementation research projects often fail to provide a coherent rationale or justification for how these aspects are selected and tested in relation to one another. Despite this need to better specify the conceptual linkages between the core elements involved in projects, few tools or methods have been developed to aid in this task. The Implementation Research Logic Model (IRLM) was created for this purpose and to enhance the rigor and transparency of describing the often-complex processes of improving the adoption of evidence-based interventions in healthcare delivery systems.

METHODS

The IRLM structure and guiding principles were developed through a series of preliminary activities with multiple investigators representing diverse implementation research projects in terms of contexts, research designs, and implementation strategies being evaluated. The utility of the IRLM was evaluated in the course of a 2-day training to over 130 implementation researchers and healthcare delivery system partners.

RESULTS

Preliminary work with the IRLM produced a core structure and multiple variations for common implementation research designs and situations, as well as guiding principles and suggestions for use. Results of the survey indicated a high utility of the IRLM for multiple purposes, such as improving rigor and reproducibility of projects; serving as a "roadmap" for how the project is to be carried out; clearly reporting and specifying how the project is to be conducted; and understanding the connections between determinants, strategies, mechanisms, and outcomes for their project.

CONCLUSIONS

The IRLM is a semi-structured, principle-guided tool designed to improve the specification, rigor, reproducibility, and testable causal pathways involved in implementation research projects. The IRLM can also aid implementation researchers and implementation partners in the planning and execution of practice change initiatives. Adaptation and refinement of the IRLM are ongoing, as is the development of resources for use and applications to diverse projects, to address the challenges of this complex scientific field.

Advancing complexity science in healthcare research: the logic of logic models

Background

Logic models are commonly used in evaluations to represent the causal processes through which interventions produce outcomes, yet significant debate is currently taking place over whether they can describe complex interventions which adapt to context. This paper assesses the logic models used in healthcare research from a complexity perspective. A typology of existing logic models is proposed, as well as a formal methodology for deriving more flexible and dynamic logic models.

Analysis

Various logic model types were tested as part of an evaluation of a complex Patient Experience Toolkit (PET) intervention, developed and implemented through action research across six hospital wards/departments in the English NHS. Three dominant types of logic model were identified, each with certain strengths but ultimately unable to accurately capture the dynamics of PET. Hence, a fourth logic model type was developed to express how success hinges on the adaption of PET to its delivery settings. Aspects of the Promoting Action on Research Implementation in Health Services (PARIHS) model were incorporated into a traditional logic model structure to create a dynamic “type 4” logic model that can accommodate complex interventions taking on a different form in different settings.

Conclusion

Logic models can be used to model complex interventions that adapt to context but more flexible and dynamic models are required. An implication of this is that how logic models are used in healthcare research may have to change. Using logic models to forge consensus among stakeholders and/or provide precise guidance across different settings will be inappropriate in the case of complex interventions that adapt to context. Instead, logic models for complex interventions may be targeted at facilitators to enable them to prospectively assess the settings they will be working in and to develop context-sensitive facilitation strategies. Researchers should be clear as to why they are using a logic model and experiment with different models to ensure they have the correct type.

Electronic supplementary material

The online version of this article (10.1186/s12874-019-0701-4) contains supplementary material, which is available to authorized users.

A programme theory for liaison mental health services in England

Background

Mechanisms by which liaison mental health services (LMHS) may bring about improved patient and organisational outcomes are poorly understood. A small number of logic models have been developed, but they fail to capture the complexity of clinical practice.

Method

We synthesised data from a variety of sources including a large national survey, 73 in-depth interviews with acute and liaison staff working in hospitals with different types of liaison mental health services, and relevant local, national and international literature. We generated logic models for two common performance indicators used to assess organisational outcomes for LMHS: response times in the emergency department and hospital length of stay for people with mental health problems.

Results

We identified 8 areas of complexity that influence performance, and 6 trade-offs which drove the models in different directions depending upon the balance of the trade-off. The logic models we developed could only be captured by consideration of more than one pass through the system, the complexity in which they operated, and the trade-offs that occurred.

Conclusions

Our findings are important for commissioners of liaison services. Reliance on simple target setting may result in services that are unbalanced and not patient-centred. Targets need to be reviewed on a regular basis, together with other data that reflect the wider impact of the service, and any external changes in the system that affect the performance of LMHS, which are beyond their control.

Social Interface Model: Theorizing Ecological Post-Delivery Processes for Intervention Effects

Successful prevention programs depend on a complex interplay among aspects of the intervention, the participant, the specific intervention setting, and the broader set of contexts with which a participant interacts. There is a need to theorize what happens as participants bring intervention ideas and behaviors into other life-contexts, and theory has not yet specified how social interactions about interventions may influence outcomes. To address this gap, we use an ecological perspective to develop the social interface model. This paper presents the key components of the model and its potential to aid the design and implementation of prevention interventions. The model is predicated on the idea that intervention message effectiveness depends not only on message aspects but also on the participants' adoption and adaptation of the message vis-à-vis their social ecology. The model depicts processes by which intervention messages are received and enacted by participants through social processes occurring within and between relevant microsystems. Mesosystem interfaces (negligible interface, transference, co-dependence, and interdependence) can facilitate or detract from intervention effects. The social interface model advances prevention science by theorizing that practitioners can create better quality interventions by planning for what occurs after interventions are delivered.

How Decision Support Systems Can Benefit from a Theory of Change Approach

Decision support systems are now mostly computer and internet-based information systems designed to support land managers with complex decision-making. However, there is concern that many environmental and agricultural decision support systems remain underutilized and ineffective. Recent efforts to improve decision support systems use have focused on enhancing stakeholder participation in their development, but a mismatch between stakeholders' expectations and the reality of decision support systems outputs continues to limit uptake. Additional challenges remain in problem-framing and evaluation. We propose using an outcomes-based approach called theory of change in conjunction with decision support systems development to support both wider problem-framing and outcomes-based monitoring and evaluation. The theory of change helps framing by placing the decision support systems within a wider context. It highlights how decision support systems use can "contribute" to long-term outcomes, and helps align decision support systems outputs with these larger goals. We illustrate the benefits of linking decision support systems development and application with a theory of change approach using an example of pest rabbit management in Australia. We develop a theory of change that outlines the activities required to achieve the outcomes desired from an effective rabbit management program, and two decision support systems that contribute to specific aspects of decision making in this wider problem context. Using a theory of change in this way should increase acceptance of the role of decision support systems by end-users, clarify their limitations and, importantly, increase effectiveness of rabbit management. The use of a theory of change should benefit those seeking to improve decision support systems design, use and, evaluation.

Conceptualising the public health role of actors operating outside of formal health systems: The case of social enterprise

This paper focuses on the role of actors that operate outside formal health systems, but nevertheless have a vital, if often under-recognised, role in supporting public health. The specific example used is the 'social enterprise', an organisation that seeks, through trading, to maximise social returns, rather than the distribution of profits to shareholders or owners. In this paper we advance empirical and theoretical understanding of the causal pathways at work in social enterprises, by considering them as a particularly complex form of public health 'intervention'. Data were generated through qualitative, in depth, semi-structured interviews and a focus group discussion, with a purposive, maximum variation sample of social enterprise practitioners (n = 13) in an urban setting in the west of Scotland. A method of analysis inspired by critical realism - Causation Coding - enabled the identification of a range of explanatory mechanisms and potential pathways of causation between engagement in social enterprise-led activity and various outcomes, which have been grouped into physical health, mental health and social determinants. The findings then informed the construction of an empirically-informed conceptual model to act as a platform upon which to develop a future research agenda. The results of this work are considered to not only encourage a broader and more imaginative consideration of what actually constitutes a public health intervention, but also reinforces arguments that actors within the Third Sector have an important role to play in addressing contemporary and future public health challenges.

Public health program planning logic model for community engaged type 2 diabetes management and prevention

Diabetes remains a growing epidemic with widening health inequity gaps in disease management, self-management knowledge, access to care and outcomes. Yet there is a paucity of evaluation tools for community engaged interventions aimed at closing the gaps and improving health. The Guide to Community Preventive Services (the Community Guide) developed by the Task Force on Community Preventive Services (the Task Force) at the Centers for Disease Control and Prevention (CDC) recommends two healthcare system level interventions, case management interventions and disease management programs, to improve glycemic control. However, as a public health resource guide for diabetes interventions a model for community engagement is a glaringly absent component of the Community Guide recommendations. In large part there are few evidence-based interventions featuring community engagement as a practice and system-level focus of chronic disease and Type 2 diabetes management. The central argument presented in this paper is that the absence of these types of interventions is due to the lack of tools for modeling and evaluating such interventions, especially among disparate and poor populations. A conceptual model emphasizing action-oriented micro-level community engagement is needed to complement the Community Guide and serve as the basis for testing and evaluation of these kinds of interventions. A unique logic model advancing the Community Guide diabetes recommendations toward measureable and sustainable community engagement for improved Type 2 diabetes outcomes is presented.