Protocol for the Optimizing Naloxone Dispensing in Pharmacies (ONDP) Online Continuing Education Program: A Randomized Controlled Trial

The number of opioid-related deaths in Canada has steadily increased since 2016 and the COVID-19 pandemic has worsened this trend. Naloxone has been pivotal for reducing opioid-related harms and death, and pharmacists play a crucial role in ensuring the supply of naloxone to Canadians through community pharmacies. However, naloxone dispensing by pharmacists is not optimal; in fact, in Ontario, only 50% of pharmacists offer naloxone, despite national guidelines that pharmacists should offer naloxone to everyone with an opioid prescription. When asked why pharmacists do not proactively offer naloxone, recent research has identified that pharmacists need continuing education to boost confidence and knowledge on how to start conversations with patients. The study involves a delayed start, double-blind randomized controlled trial, for Canadian licensed pharmacists and pharmacy technicians. The goals of the program are to increase Canadian pharmacy professional’s knowledge, confidence, and motivation to proactively offer naloxone, as well as to decrease stigma associated with naloxone. The program incorporates behaviour change techniques from the Theoretical Domains Framework and the Theory of Planned Behaviour. The intervention program includes three modules that focus on improving pharmacists’ communication skills by teaching them how to proactively offer naloxone, while the control group will complete a reading assignment on the naloxone consensus guidelines. The program will involve a process and outcome evaluation in addition to a contribution analysis. This program is important for breaking down previously identified barriers and knowledge gaps for why pharmacists currently do not proactively offer naloxone. This study will provide important new information about what behaviour change techniques are successful in improving confidence and motivation in the pharmacy profession and in an online environment. Findings from this study can be used to produce a national naloxone education program that can also be implemented into current pharmacy school curriculum.

Evaluation in health professions education-Is measuring outcomes enough?

INTRODUCTION

In an effort to increase the rigour of evaluation in health professions education (HPE), a range of evaluation approaches are used. These largely focus on outcome evaluation as opposed to programme evaluation. We aim to review and critique the use of outcome evaluation models, using the Kirkpatrick Model as an example given its wide acceptance and use, and advocate for the use of programme evaluation models that help us understand how and why outcomes are occurring.

METHODS

We systematically searched OVID medline, Scopus, CINAHL and Pubmed, and hand searched six leading HPE journals to provide an overview of the use of the Kirkpatrick Model as well as a range of programme evaluation models in HPE. In addition to this, we synthesised the existing critiques of the Kirkpatrick Model as an example of outcome evaluation, to highlight the limitations of such models.

RESULTS

The use of the Kirkpatrick Model in HPE is widespread and increasing; however, studies focus on categorising outcomes, rather than explaining how and why they occur. The main criticisms of the model are as follows: it is outcomes focused and fails to consider factors that can impact training outcomes; it assumes positive casual linkages between the levels; there is an assumption that the higher-level outcomes are more important; and unintended impacts are not considered. The use of the Kirkpatrick Model by the MERSQI, BEME and WHO contribute to the myth that the Kirkpatrick Model is the gold standard for programme evaluation.

DISCUSSION

Moving forward, evaluations of HPE interventions must shift from focusing largely on measuring outcomes of interventions with little consideration for how and why these outcomes are occurring to programme evaluation that investigates what contributes to these outcomes. Other models that facilitate the evaluation of the complex processes that occur in HPE should be used instead of Kirkpatrick's.

Beyond Indicators and Success Stories: An Emerging Method to Assess Social Learning in Large-Scale Transdisciplinary Research Programs

Facilitated learning approaches are increasingly being used as a means to enhance climate and sustainability collaborations working across disciplines, regions, and scales. With investments into promoting and supporting inter- and transdisciplinary learning in major programs on complex global challenges like climate change on the rise, scholars and practitioners are calling for a more grounded and empirical understanding of learning processes and their outcomes. Yet, methodologies for studying the interplay between learning and change in these initiatives remain scarce, owing to both the "hard to measure" nature of learning and the complexity of large-scale program implementation and evaluation. This paper proposes a new method for studying social learning in the context of large research programs. It aims to analyze the social learning of researchers and practitioners engaged in these programs and assess the contributions of this learning to the resilience of the natural and social systems that these programs seek to influence. We detail the theoretical basis for this new approach and set out six steps for developing multi-layered contribution pathways and contribution stories with stakeholders to document both the process and outcomes of social learning. The proposed method, we argue, can strengthen our analytical capacity to uncover the structural drivers and barriers to social learning that are often masked by the complexity of large-scale programs. An illustrative example, drawn from a large-scale climate adaptation research program, provides evidence on how this method might advance our methodological strategies for studying learning in these programs. We conclude by highlighting two key methodological contributions brought about through this approach, and by reflecting on opportunities for further methodological development. Enriching our understanding of learning and change processes, we argue, is an important avenue for understanding how we can pursue transformations for sustainability.

A refined method for theory-based evaluation of the societal impacts of research

With high and increasing expectations for research to have social and environmental impact, there is a corresponding need for appropriate methods to demonstrate (for accountability) and analyze (for learning) whether and how research projects contribute to change processes. Evaluation is especially challenging for problem-oriented research that employs inter- and transdisciplinary approaches and intervenes in complex systems, where experimental and statistical approaches to causal inference are inappropriate. Instead, theory-based evaluation can be applied to identify and test causal processes. This paper presents a detailed explanation of the Outcome Evaluation approach applied in Belcher et al. (2019b). It draws on concepts and approaches used in theory-based program evaluation and the more limited experience of theory-based research evaluation, providing a brief overview of conceptual strengths and limitations of other methods. The paper offers step-by-step guidance on application of the Outcome Evaluation approach, detailing how to: document a theory of change; determine data needs and sources; collect data; manage and analyze data; and present findings. This approach provides a clear conceptual and analytical framework in addition to actor-specific and impact pathway analyses for more precision in the assessment of outcomes. Specifically, the Outcome Evaluation approach: •Conceptualizes research within a complex system and explicitly recognizes the role of other actors, context, and external processes;•Utilizes a detailed actor-centred theory of change (ToC) as the analytical framework; and•Explicitly tests a set of hypotheses about the relationship between the research process/outputs and outcomes.

Contribution of the Alive & Thrive-UNICEF advocacy efforts to improve infant and young child feeding policies in Southeast Asia

Evaluating the impact of advocacy for policy change presents many challenges. Recent advances in the field of evaluation, such as contribution analysis (CA), offer guidance on how to make credible claims regarding such impact. The purposes of this article are (a) to detail the application of CA to assess the contribution of an advocacy initiative to improve infant and young child feeding policies and (b) to present the emergent theory of change and contribution story of how progress was achieved. An evaluation applying developmental evaluation and CA was conducted on the Alive & Thrive (A&T)-UNICEF initiative in seven Southeast Asian countries to document the extent to which policy objectives were achieved and identify key drivers of policy change. A contribution story was developed based on these experiences. The advocacy approach, which involved a four-part process, contributed directly to (a) set the agenda of various actors and (b) create a strategic group; and indirectly to (a) set and maintain the issue on the agenda at all stages of the policy cycle, (b) support the government to carry out a set of critical tasks, and (c) extend commitment. All of this helped to achieve progress towards policy change. External influences were at play. The flexibility of A&T allowed key actors to utilize the positive external influences and address some of the negative ones through developing responsive strategies mitigating their effects. The emerging contribution story supports that A&T-UNICEF initiative contributed to the progress achieved in the participating countries.

Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research

Background

Measuring the policy and practice impacts of research is becoming increasingly important. Policy impacts can be measured from two directions – tracing forward from research and tracing backwards from a policy outcome. In this review, we compare these approaches and document the characteristics of studies assessing research impacts on policy and the policy utilisation of research.

Methods

Keyword searches of electronic databases were conducted in December 2016. Included studies were published between 1995 and 2016 in English and reported methods and findings of studies measuring policy impacts of specified health research, or research use in relation to a specified health policy outcome, and reviews reporting methods of research impact assessment. Using an iterative data extraction process, we developed a framework to define the key elements of empirical studies (assessment reason, assessment direction, assessment starting point, unit of analysis, assessment methods, assessment endpoint and outcomes assessed) and then documented the characteristics of included empirical studies according to this framework.

Results

We identified 144 empirical studies and 19 literature reviews. Empirical studies were derived from two parallel streams of research of equal size, which we termed ‘research impact assessments’ and ‘research use assessments’. Both streams provided insights about the influence of research on policy and utilised similar assessment methods, but approached measurement from opposite directions. Research impact assessments predominantly utilised forward tracing approaches while the converse was true for research use assessments. Within each stream, assessments focussed on narrow or broader research/policy units of analysis as the starting point for assessment, each with associated strengths and limitations. The two streams differed in terms of their relative focus on the contributions made by specific research (research impact assessments) versus research more generally (research use assessments) and the emphasis placed on research and the activities of researchers in comparison to other factors and actors as influencers of change.

Conclusions

The Framework presented in this paper provides a mechanism for comparing studies within this broad field of research enquiry. Forward and backward tracing approaches, and their different ways of ‘looking’, tell a different story of research-based policy change. Combining approaches may provide the best way forward in terms of linking outcomes to specific research, as well as providing a realistic picture of research influence.

Electronic supplementary material

The online version of this article (10.1186/s12961-018-0310-4) contains supplementary material, which is available to authorized users.