Defining, illustrating and reflecting on logic analysis with an example from a professional development program

Building a Logic Model to Foster Engagement and Learning Using the Case of a Province-Wide Multispecies Antimicrobial Use Monitoring System

Successfully designing and implementing a program is complex; it requires a reflexive balance between the available resources and the priorities of various stakeholders, both of which change over time. Logic models are theory-based evaluation approaches used to identify and address key challenges of a program. This article describes the process of building a logic model on advanced theories in complexity studies. The models aim to support a province-wide multispecies monitoring system of antimicrobial use (AMU), designed in collaboration with the animal health sector in Quebec (Canada). Based on a rigorous theoretical foundation, the logic model is built in three steps: (1) mapping, a narrative review of literature on similar programs in other jurisdictions; (2) framing, iterative consultations with project members to elaborate the logic model; (3) shaping, hypotheses based on the logic model. The model emerges from the reflexive balancing of current scientific knowledge and empirical insights to gather relevant information about stakeholders from interdisciplinary experts that led a 3-year consensus-building process within the community. Recognizing the challenge of unpacking theories for practical use, we illustrate how the process of an "open" logic model building could enable governance coordination in complex processes. Logic models are useful for evaluating public, private, and academic partnerships in One Health programs that characterize an adaptive governance process.

Healthcare professionals' longitudinal perceptions of group phenomena as determinants of self-assessed learning in organizational communities of practice

BACKGROUND

Given the importance of continuous learning as a response to the increasing complexity of health care practice, there is a need to better understand what makes communities of practice in health effective at fostering learning. Despite the conceptual stance that communities of practice facilitate individual learning, the scientific literature does not offer much evidence for this. Known factors associated with the effectiveness of communities of practice - such as collaboration, psychological safety within the community, and commitment to the community - have been studied in cross-sectional qualitative designs. However, no studies to date have used a quantitative predictive design. The objective of this study is to assess how members of a community of practice perceive interactions among themselves and determine the extent to which these interactions predict self-assessed learning over time.

METHODS

Data was collected using validated questionnaires from six communities of practice (N = 83) in four waves of measures over the course of 36 months and was analysed by means of General Estimating Equations. This allowed to build a longitudinal model of the associations between perceptions of collaboration, psychological safety within the community, commitment to the community and self-assessed learning over time.

RESULTS

Perception of collaboration in the community of practice, a personal sense of psychological safety and a commitment to the community of practice are predictors longitudinally associated with self-assessed learning.

CONCLUSIONS

In terms of theory, conceptual links can be made between intensity of collaboration and learning over time in the context of a community of practice. Recent work on psychological safety suggests that it is still unclear whether psychological safety acts as a direct enhancer of learning or as a remover of barriers to learning. This study's longitudinal results suggest that psychological safety may enhance how and to what extent professionals feel they learn over time. Commitment towards the community of practice is a strong predictor of learning over time, which hints at differential effects of affective, normative and continuance commitment. Communities of practice can therefore apply these findings by making collaboration, psychological safety, commitment and learning regular reflexive topics of discussion.

Designing centralized waiting lists for attachment to a primary care provider: Considerations from a logic analysis

Access to a regular primary care provider is essential to quality care. In Canada, where 15 % of patients are unattached (i.e., without a regular provider), centralized waiting lists (CWLs) help attach patients to a primary care provider (family physician or nurse practitioner). Previous studies reveal mechanisms needed for CWLs to work, but focus mostly on CWLs for specialized health care. We aim to better understand how to design CWLs for unattached patients in primary care. In this study, a logic analysis compares empirical evidence from a qualitative case study of CWLs for unattached patients in seven Canadian provinces to programme theory derived from a realist review on CWLs. Data is analyzed using context-intervention-mechanism-outcome configurations. Results identify mechanisms involved in three components of CWL design: patient registration, patient prioritization, and patient assignment to a provider for attachment. CWL programme theory is revised to integrate mechanisms specific to primary care, where patients, rather than referring providers, are responsible for registering on the CWL, where prioritization must consider a broad range of conditions and characteristics, and where long-term acceptability of attachment is important. The study provides new insight into mechanisms that enable CWLs for unattached patients to work.

Five ways to get a grip on the shortcomings of logic models in program evaluation

Logic models are perhaps the most widely used tools in program evaluation work. They provide reasonably straightforward, visual illustrations of plausible links between program activities and outcomes. Consequently, they are employed frequently in stakeholder engagement, communication, and evaluation project planning. However, their relative simplicity comes with multiple drawbacks that can compromise the integrity of evaluation studies. In this Black Ice article, we outline key considerations and provide practical strategies that can help those engaged in evaluation work to identify and mitigate some limitations of logic models.

Adapting a health equity tool to meet professional needs (Québec, Canada)

While numerous tools are available to better incorporate equity into population health actions, they are limited mainly by their lack of adaptation to professional practices and organizational realities. A study was conducted in Québec to identify and understand, from the perspective of future users, conditions that would facilitate use of a tool (Reflex-ISS) targeted at supporting collaborative action to improve consideration of social inequalities in health (SIH) within population health actions. Concept mapping and focus groups were implemented as complementary methods for investigating the conditions. Significant results that emerged were strong participant interest in the tool and the need for resources to better take SIH into account. The conditions for use that were identified referred to the tool itself (user-friendliness and literacy) and to resources for appropriating the tool, competency development, as well as the role and responsibilities of organizations and policies in promoting use of the tool in daily activities and more fundamentally in acting against SIH in general. Models for organizational innovation give an idea of the dimensions that need to be considered to strengthen the integration of equity into organizations and to support the changes in practice that result from using the tool. They provide a reminder that a health equity tool cannot be the cornerstone of an organizational strategy to fight against SIH; rather, it must be incorporated as part of a systemic strategy of professional and organizational development.

Using collaborative logic analysis evaluation to test the program theory of an intensive interdisciplinary pain treatment for youth with pain‐related disability

Intensive interdisciplinary pain treatment (IIPT) involves multiple stakeholders. Mapping the program components to its anticipated outcomes (ie, its theory) can be difficult and requires stakeholder engagement. Evidence is lacking, however, on how best to engage them. Logic analysis, a theory‐based evaluation, that tests the coherence of a program theory using scientific evidence and experiential knowledge may hold some promise. Its use is rare in pediatric pain interventions, and few methodological details are available. This article provides a description of a collaborative logic analysis methodology used to test the theoretical plausibility of an IIPT designed for youth with pain‐related disability. A 3‐step direct logic analysis process was used. A 13‐member expert panel, composed of clinicians, teachers, managers, youth with pain‐related disability, and their parents, were engaged in each step. First, a logic model was constructed through document analysis, expert panel surveys, and focus‐group discussions. Then, a scoping review, focused on pediatric self‐management, building self‐efficacy, and fostering participation, helped create a conceptual framework. An examination of the logic model against the conceptual framework by the expert panel followed, and recommendations were formulated. Overall, the collaborative logic analysis process helped raiseawareness of clinicians’ assumptions about the program causal mechanisms, identified program components most valued by youth and their parents, recognized the program features supported by scientific and experiential knowledge, detected gaps, and highlighted emerging trends. In addition to providing a consumer‐focused program evaluation option, collaborative logic analysis methodology holds promise as a strategy to engage stakeholders and to translate pediatric pain rehabilitation evaluation research knowledge to key stakeholders.

Evaluating complex interventions in real context: Logic analysis of a case management program for frequent users of healthcare services

Case management programs for frequent users of healthcare services are complex interventions which implementation and application are challenging to evaluate. The aim of this article was to conduct a logic analysis to evaluate a case management program for frequent users of healthcare services. The study proceeded in three phases: 1) establishing causal links between the program's components by the construction of a logic model, 2) developing an integrated framework from a realistic synthesis, and 3) making a new reading of the case management program in regard of the integrated framework. The study demonstrated, on one hand, strengths and weaknesses of the actual case management program, and, on the other hand, how logic analysis can create a constructive dialogue between theory and practice. The evaluative process with decision-makers, clinicians and patients has helped to make connexions between theory, practice, experience and services organization.

Optimizing the development and evaluation of complex interventions: lessons learned from the BetterBirth Program and associated trial

BACKGROUND

Despite extensive efforts to develop and refine intervention packages, complex interventions often fail to produce the desired health impacts in full-scale evaluations. A recent example of this phenomenon is BetterBirth, a complex intervention designed to implement the World Health Organization's Safe Childbirth Checklist and improve maternal and neonatal health. Using data from the BetterBirth Program and its associated trial as a case study, we identified lessons to assist in the development and evaluation of future complex interventions.

METHODS

BetterBirth was refined across three sequential development phases prior to being tested in a matched-pair, cluster randomized trial in Uttar Pradesh, India. We reviewed published and internal materials from all three development phases to identify barriers hindering the identification of an optimal intervention package and identified corresponding lessons learned. For each lesson, we describe its importance and provide an example motivated by the BetterBirth Program's development to illustrate how it could be applied to future studies.

RESULTS

We identified three lessons: (1) develop a robust theory of change (TOC); (2) define optimization outcomes, which are used to assess the effectiveness of the intervention across development phases, and corresponding criteria for success, which determine whether the intervention has been sufficiently optimized to warrant full-scale evaluation; and (3) create and capture variation in the implementation intensity of components. When applying these lessons to the BetterBirth intervention, we demonstrate how a TOC could have promoted more complete data collection. We propose an optimization outcome and related criteria for success and illustrate how they could have resulted in additional development phases prior to the full-scale trial. Finally, we show how variation in components' implementation intensities could have been used to identify effective intervention components.

CONCLUSION

These lessons learned can be applied during both early and advanced stages of complex intervention development and evaluation. By using examples from a real-world study to demonstrate the relevance of these lessons and illustrating how they can be applied in practice, we hope to encourage future researchers to collect and analyze data in a way that promotes more effective complex intervention development and evaluation.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT02148952; registered on May 29, 2014.

[Organizational learning and health promotion: a Quebec project]

OBJECTIVE

This article focuses on health promotion laboratories, a Quebec professional development program offered by the Public Health Department of the Montréal Region to teams of professionals and managers working in health promotion within local public health organizations. The objective is to examine the process of translating the knowledge gained by participants as a result of the program over the longer term within the organization.

METHOD

This was a qualitative descriptive study. The work was guided by Nonaka’s Organizational Knowledge Creation Model. Data were collected from participants at several types of discussion and development events held in the three months following the end of the pilot project. A thematic content analysis was performed using a grid derived from Nonaka’s model.

RESULTS

The analysis revealed the presence of both externalization and internalization in two of the sites, as well as a considerable volume of combinations in the four sites studied. In the latter case, the learnings reused over the longer term were similar to those that had been transferred in the short term (e.g. ideas and methods relating to partnership, planning, etc.).

CONCLUSION

These results are important, in that they confirm the laboratories’ potential to propagate the learnings throughout the organization, beyond the short-term gains made by participants during the laboratories. These learnings could potentially pave the way for new practices.

Professional development programs in health promotion: tools and processes to favor new practices

Developing innovative interventions that are in sync with a health promotion paradigm often represents a challenge for professionals working in local public health organizations. Thus, it is critical to have both professional development programs that favor new practices and tools to examine these practices. In this case study, we analyze the health promotion approach used in a pilot intervention addressing children's vulnerability that was developed and carried out by participants enrolled in a public health professional development program. More specifically, we use a modified version of Guichard and Ridde's (Une grille d'analyse des actions pour lutter contre les inégalités sociales de santé. In Potvin, L., Moquet, M.-J. and Jones, C. M. (eds), Réduire les Inégalités Sociales en Santé. INPES, Saint-Denis Cedex, pp. 297-312, 2010) analytical grid to assess deductively the program participants' use of health promotion practices in the analysis and planning, implementation, evaluation, sustainability and empowerment phases of the pilot intervention. We also seek evidence of practices involving (empowerment, participation, equity, holism, an ecological approach, intersectorality and sustainability) in the intervention. The results are mixed: our findings reveal evidence of the application of several dimensions of health promotion (equity, holism, an ecological approach, intersectorality and sustainability), but also a lack of integration of two key dimensions; that is, empowerment and participation, during various phases of the pilot intervention. These results show that the professional development program is associated with the adoption of a pilot intervention integrating multiple but not all dimensions of health promotion. We make recommendations to facilitate a more complete integration. This research also shows that the Guichard and Ridde grid proves to be a thorough instrument to document the practices of participants.

An analysis of the adaptability of a professional development program in public health: results from the ALPS Study

Background

Professional development is a key component of effective public health infrastructures. To be successful, professional development programs in public health and health promotion must adapt to practitioners’ complex real-world practice settings while preserving the core components of those programs’ models and theoretical bases. An appropriate balance must be struck between implementation fidelity, defined as respecting the core nature of the program that underlies its effects, and adaptability to context to maximize benefit in specific situations. This article presents a professional development pilot program, the Health Promotion Laboratory (HPL), and analyzes how it was adapted to three different settings while preserving its core components. An exploratory analysis was also conducted to identify team and contextual factors that might have been at play in the emergence of implementation profiles in each site.

Methods

This paper describes the program, its core components and adaptive features, along with three implementation experiences in local public health teams in Quebec, Canada. For each setting, documentary sources were analyzed to trace the implementation of activities, including temporal patterns throughout the project for each program component. Information about teams and their contexts/settings was obtained through documentary analysis and semi-structured interviews with HPL participants, colleagues and managers from each organization.

Results

While each team developed a unique pattern of implementing the activities, all the program’s core components were implemented. Differences of implementation were observed in terms of numbers and percentages of activities related to different components of the program as well as in the patterns of activities across time. It is plausible that organizational characteristics influencing, for example, work schedule flexibility or learning culture might have played a role in the HPL implementation process.

Conclusions

This paper shows how a professional development program model can be adapted to different contexts while preserving its core components. Capturing the heterogeneity of the intervention’s exposure, as was done here, will make possible in-depth impact analyses involving, for example, the testing of program–context interactions to identify program outcomes predictors. Such work is essential to advance knowledge on the action mechanisms of professional development programs.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-015-0903-3) contains supplementary material, which is available to authorized users.

Learning reflexively from a health promotion professional development program in Canada

In recent decades, reflexivity has received much attention in the professional education and training literature, especially in the public health and health promotion fields. Despite general agreement on the importance of reflexivity, there appears to be no consensus on how to assess reflexivity or to conceptualize the different forms developed among professionals and participants of training programs. This paper presents an analysis of the reflexivity outcomes of the Health Promotion Laboratory, an innovative professional development program aimed at supporting practice changes among health professionals by fostering competency development and reflexivity. More specifically, this paper explores the difference between two levels of reflexivity (formative and critical) and highlights some implications of each for practice. Data were collected through qualitative interviews with participants from two intervention sites. Results showed that involvement in the Health Promotion Laboratory prompted many participants to modify their vision of their practice and professional role, indicating an impact on reflexivity. In many cases, new understandings seem to have played a formative function in enabling participants to improve their practice and their role as health promoters. The reflective process also served a critical function culminating in a social and moral understanding of the impacts on society of the professionals' practices and roles. This type of outcome is greatly desired in health promotion, given the social justice and equity concerns of this field of practice. By redefining the theoretical concept of reflexivity on two levels and discussing their impacts on practice, this study supports the usefulness of both levels of reflexivity.