A conceptual framework for implementation fidelity

Theory-Driven Process Evaluation of the SHINE Trial Using a Program Impact Pathway Approach

Two reasons for the lack of success of programs or interventions are poor alignment of interventions with the causes of the problem targeted by the intervention, leading to poor efficacy (theory failure), and failure to implement interventions as designed (program failure). These failures are important for both public health programs and randomized trials. In the Sanitation Hygiene and Infant Nutrition Efficacy (SHINE) Trial, we utilize the program impact pathway (PIP) approach to track intervention implementation and behavior uptake. In this article, we present the SHINE PIP including definitions and measurements of key mediating domains, and discuss the implications of this approach for randomized trials. Operationally, the PIP can be used for monitoring and strengthening intervention delivery, facilitating course-correction at various stages of implementation. Analytically, the PIP can facilitate a richer understanding of the mediating and modifying determinants of intervention impact than would be possible from an intention-to-treat analysis alone.

High impact of implementation on school-based smoking prevention: the X:IT study-a cluster-randomized smoking prevention trial

BACKGROUND

Implementation fidelity describes how well an intervention is implemented in the real-world setting. Assessing implementation fidelity is essential in the understanding of intervention results. In most studies, implementation fidelity is measured insufficiently, though, not taking into account the complexity of the concept nor the intervention. The objective of the present study was to develop an overall quantitative measure of implementation fidelity, to examine the degree of implementation fidelity and the association of implementation and effect of a randomized school-based smoking prevention trial-the X:IT study.

METHODS

A cluster-randomized trial testing is a multi-component intervention to prevent smoking among adolescents in 94 Danish elementary schools (51 intervention, 43 control schools). Participants were grade 7 pupils (mean age 12.5 years). Data was collected by electronic questionnaires among pupils at baseline (n = 4161), the first follow-up (n = 3764), and the second follow-up (n = 3269) and among school coordinators at intervention schools at the first and second follow-up (50 and 39 coordinators).

INTERVENTION

The intervention included three components: (1) smoke-free school grounds, (2) smoke-free curriculum, and (3) parental involvement, contracts, and dialogues. Implementation fidelity was assessed by four domains: adherence, dose, quality of delivery, and participant responsiveness. These were combined into an overall school-wise implementation index. The association of implementation and smoking was examined by logistic regression analyses.

RESULTS

One fourth of the schools was characterized as high implementers of the program (all three components) at both first (12 schools, 24.0 %) and second follow-up (11 schools, 28.2 %). Implementation fidelity was strongly associated with smoking at the first and second follow-up, e.g., the odds for smoking at schools with high implementation both years were OR = 0.44 (95 % CI 0.32 to 0.68).

CONCLUSIONS

Using an overall measure based on several aspects of implementation fidelity, we showed a negative graded association between implementation and smoking. This study suggests that higher degrees of implementation will improve the effect of the X:IT intervention. Studying the association between implementation and effect is extremely important; only by doing so, we can distinguish the quality of the intervention from the success of the implementation.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN77415416.

Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies

BACKGROUND

Numerous types of digital health interventions (DHIs) are available to patients and the public but many factors affect their ability to engage and enrol in them. This systematic review aims to identify and synthesise the qualitative literature on barriers and facilitators to engagement and recruitment to DHIs to inform future implementation efforts.

METHODS

PubMed, MEDLINE, CINAHL, Embase, Scopus and the ACM Digital Library were searched for English language qualitative studies from 2000 - 2015 that discussed factors affecting engagement and enrolment in a range of DHIs (e.g. 'telemedicine', 'mobile applications', 'personal health record', 'social networking'). Text mining and additional search strategies were used to identify 1,448 records. Two reviewers independently carried out paper screening, quality assessment, data extraction and analysis. Data was analysed using framework synthesis, informed by Normalization Process Theory, and Burden of Treatment Theory helped conceptualise the interpretation of results.

RESULTS

Nineteen publications were included in the review. Four overarching themes that affect patient and public engagement and enrolment in DHIs emerged; 1) personal agency and motivation; 2) personal life and values; 3) the engagement and recruitment approach; and 4) the quality of the DHI. The review also summarises engagement and recruitment strategies used. A preliminary DIgital Health EnGagement MOdel (DIEGO) was developed to highlight the key processes involved. Existing knowledge gaps are identified and a number of recommendations made for future research. Study limitations include English language publications and exclusion of grey literature.

CONCLUSION

This review summarises and highlights the complexity of digital health engagement and recruitment processes and outlines issues that need to be addressed before patients and the public commit to digital health and it can be implemented effectively. More work is needed to create successful engagement strategies and better quality digital solutions that are personalised where possible and to gain clinical accreditation and endorsement when appropriate. More investment is also needed to improve computer literacy and ensure technologies are accessible and affordable for those who wish to sign up to them.

SYSTEMATIC REVIEW REGISTRATION

International Prospective Register of Systematic Reviews CRD42015029846.

Process evaluation of five tailored programs to improve the implementation of evidence-based recommendations for chronic conditions in primary care

Background

Although there is evidence that tailored implementation strategies can be effective, there is little evidence on which methods of tailoring improve the effect. We designed and evaluated five tailored programs (TPs) each consisting of various strategies. The aim of this study was to examine (a) how determinants of practice prioritized in the design phase of the TPs were perceived by health care professionals who had been exposed to the TPs and whether they suggested other important determinants of practice and (b) how professionals used the offered strategies and whether they suggested other strategies that might have been more effective.

Methods

We conducted a mixed-method process evaluation linked to five cluster-randomized trials carried out in five European countries to implement recommendations for five chronic conditions in primary care settings. The five TPs used a total of 28 strategies which aimed to address 38 determinants of practice. Interviews of professionals in the intervention groups and a survey of professionals in the intervention and control groups were performed. Data collection was conducted by each research team in the respective national language. The interview data were first analyzed inductively by each research team, and subsequently, a meta-synthesis was conducted. The survey was analyzed descriptively.

Results

We conducted 71 interviews; 125 professionals completed the survey. The survey showed that 76 % (n = 29) of targeted determinants of practice were perceived as relevant and 95 % (n = 36) as being modified by the implementation interventions by 66 to 100 % of professionals. On average, 47 % of professionals reported using the strategies and 51 % considered them helpful, albeit with substantial variance between countries and strategies. In the interviews, 89 determinants of practice were identified, of which 70 % (n = 62) had been identified and 45 % (n = 40) had been prioritized in the design phase. The interviewees suggested 65 additional strategies, of which 54 % (n = 35) had been identified and 20 % (n = 13) had been prioritized, but not selected in the final programs.

Conclusions

This study largely confirmed the perceived relevance of the targeted determinants of practice. This contrasts with the fact that no impact of the trials on the implementation of the recommendations could be observed. The findings suggest that better methods for prioritization of determinants and strategies are needed.

Trial registration

Each of the five trials was registered separately in recognized trial registries. Details are given in the respective trial outcome papers.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-016-0473-8) contains supplementary material, which is available to authorized users.

A systematic review of the effectiveness of discharge care bundles for patients with COPD

BACKGROUND

A COPD discharge bundle is a set of evidence-based practices aimed at improving patient outcomes after discharge from acute care settings following an exacerbation. We conducted a systematic review on the effectiveness of COPD discharge bundles and summarised their individual care elements.

METHODS

Biomedical electronic databases and clinical trial registries were searched from database inception through April 2016 to identify experimental studies evaluating care bundles offered to patients with COPD at discharge. Random-effects meta-analyses of clinical trials data were conducted for hospital readmissions, mortality, and quality of life (QoL).

RESULTS

The review included 14 studies (5 clinical trials, 7 uncontrolled trials, and 2 interrupted time series). A total of 26 distinct elements of care were included in the bundles of individual studies. Evidence from four clinical trials with moderate-to-high risk of bias showed that COPD discharge bundles reduced hospital readmissions (pooled risk ratio (RR): 0.80; 95% CI 0.65 to 0.99). There was insufficient evidence that care bundles influence long-term mortality (RR: 0.74; 95% CI 0.43 to 1.28; four trials) or QoL (mean difference in St. George's Respiratory Questionnaire: 1.84; 95% CI -2.13 to 5.8).

CONCLUSIONS

Discharge bundles for patients with COPD led to fewer readmissions but did not significantly improve mortality or QoL. Future studies should employ higher quality research methods, fully report care bundle elements, implementation strategies and intervention fidelity to better evaluate the effectiveness of packaging evidence-based interventions together to improve outcomes of patients with COPD discharged from acute care settings.

Real-world implementation of electronic patient-reported outcomes in outpatient pediatric cancer care

OBJECTIVE

The KLIK method is an online tool that monitors and discusses electronic patient-reported outcomes (ePROs), which has been shown to enhance outcomes. This study aimed (1) to determine the fidelity (ie, extent to which used as intended) of the KLIK method as implemented in outpatient pediatric cancer care and (2) to study health care professional (HCP)-reported barriers and facilitators for implementation.

METHODS

Two hundred five children with newly diagnosed cancer (enrollment rate 85%) participated. At 1 (T1), 3 (T2), and 6 (T3) months after diagnosis, patients (8-18 years) or parents (of patients 0-7 years) completed health-related quality of life (HRQoL) questionnaires, which were transformed into an ePROfile and discussed by their HCP during consultations. Fidelity was determined by the following: percentage of website registrations, HRQoL questionnaires completed, and ePROfiles discussed. Implementation determinants were assessed with HCPs after the final T3 with the Measurement Instrument for Determinants of Innovations.

RESULTS

Depending on the time point (T1-T3), fidelity was 86% to 89% for website registration, 66-85% for completed HRQoL questionnaires, and 56% to 62% for ePROfile discussion. Barriers were mainly related to organizational issues (eg, organizational change) and less frequently to users (eg, motivation to comply) or the intervention (compatibility). Facilitators were related to the user (eg, positive outcome expectations) and intervention (simplicity) but not to the organization.

CONCLUSIONS

When implementing ePROs in outpatient pediatric oncology practice, HCPs report determinants that influence ePRO integration. To improve implementation and outcomes, tailored organizational (eg, formal ratification by management and time) and specific local (eg, individualized assessments) strategies should be developed to achieve optimal ePRO discussion.

Qualitative study on the implementation of professional pharmacy services in Australian community pharmacies using framework analysis

BACKGROUND

Multiple studies have explored the implementation process and influences, however it appears there is no study investigating these influences across the stages of implementation. Community pharmacy is attempting to implement professional services (pharmaceutical care and other health services). The use of implementation theory may assist the achievement of widespread provision, support and integration. The objective was to investigate professional service implementation in community pharmacy to contextualise and advance the concepts of a generic implementation framework previously published.

METHODS

Purposeful sampling was used to investigate implementation across a range of levels of implementation in community pharmacies in Australia. Twenty-five semi-structured interviews were conducted and analysed using a framework methodology. Data was charted using implementation stages as overarching themes and each stage was thematically analysed, to investigate the implementation process, the influences and their relationships. Secondary analyses were performed of the factors (barriers and facilitators) using an adapted version of the Consolidated Framework for Implementation Research (CFIR), and implementation strategies and interventions, using the Expert Recommendations for Implementing Change (ERIC) discrete implementation strategy compilation.

RESULTS

Six stages emerged, labelled as development or discovery, exploration, preparation, testing, operation and sustainability. Within the stages, a range of implementation activities/steps and five overarching influences (pharmacys' direction and impetus, internal communication, staffing, community fit and support) were identified. The stages and activities were not applied strictly in a linear fashion. There was a trend towards the greater the number of activities considered, the greater the apparent integration into the pharmacy organization. Implementation factors varied over the implementation stages, and additional factors were added to the CFIR list and definitions modified/contextualised for pharmacy. Implementation strategies employed by pharmacies varied widely. Evaluations were lacking.

CONCLUSIONS

The process of implementation and five overarching influences of professional services implementation in community pharmacy have been outlined. Framework analysis revealed, outside of the five overarching influences, factors influencing implementation varied across the implementation stages. It is proposed at each stage, for each domain, the factors, strategies and evaluations should be considered. The Framework for the Implementation of Services in Pharmacy incorporates the contextualisation of implementation science for pharmacy.

The "Retrofitting" Approach to Adapting Evidence-Based Interventions: A Case Study of Pediatric Asthma Care Coordination, United States, 2010-2014

Adaptation of evidence-based interventions upon implementation into new practice settings is universal, yet poorly understood. During a cross-site evaluation of the implementation of a proven intervention for pediatric asthma care coordination into 4 resource-challenged settings, we conducted in-depth interviews with site representatives, who reported how and why they modified intervention components. Interview notes were coded for themes. We focused on a single theme from a respondent who described the adaptation process as "backing" the intervention into ongoing services; we found evidence of a similar process at other sites. We labeled this process "retrofitting" to signify adaptation that consists of altering existing services to align with intervention components, rather than modifying the intervention to fit a new setting. Advantages of retrofitting may include allowing organizations to keep what works, capitalizing on existing support for program activities, elevating the role of local knowledge, and potentially promoting the sustainability of effective innovations.

Evaluating complex health financing interventions: using mixed methods to inform further implementation of a novel PBI intervention in rural Malawi

Background

Gaps remain in understanding how performance-based incentive (PBI) programs affect quality of care and service quantity, whether programs are cost effective and how programs could be tailored to meet client and provider needs while remaining operationally viable. In 2014, Malawi’s Ministry of Health launched the Service Delivery Integration-PBI (SSDI-PBI) program. The program is unique in that no portion of performance bonuses are paid to individual health workers, and it shifts responsibility for infrastructure and equipment procurement from facility staff to implementing partners. This protocol outlines an approach that analyzes processes and outcomes, considers expected and unexpected consequences of the program and frames the program’s outputs relative to its costs. Findings from this evaluation will inform the intended future scale-up of PBI in Malawi.

Methods/design

This study employs a prospective controlled before-and-after triangulation design to assess effects of the PBI program by analyzing quantitative and qualitative data from intervention and control facilities. Guided by a theoretical framework, the evaluation consists of four main components: service provision, health worker motivation, implementation processes and costing. Quality and access outcomes are assessed along four dimensions: (1) structural elements (related to equipment, drugs, staff); (2) process elements (providers’ compliance with standards); (3) outputs (service utilization); (4) experiential elements (experiences of service delivery). The costing component includes costs related to start-up, ongoing management, and the cost of incentives themselves. The cost analysis considers costs incurred within the Ministry of Health, funders, and the implementing agency. The evaluation relies on primary data (including interviews and surveys) and secondary data (including costing and health management information system data).

Discussion

Through the lens of a PBI program, we illustrate how complex interventions can be evaluated via not only primary, mixed-methods data collection, but also through a wealth of secondary data from program implementers (including monitoring, evaluation and financial data), and the health system (including service utilization and service readiness data). We also highlight the importance of crafting a theory and using theory to inform the nature of data collected. Finally, we highlight the need to be responsive to stakeholders in order to enhance a study’s relevance.

Behaviour Centred Design: towards an applied science of behaviour change

Behaviour change has become a hot topic. We describe a new approach, Behaviour Centred Design (BCD), which encompasses a theory of change, a suite of behavioural determinants and a programme design process. The theory of change is generic, assuming that successful interventions must create a cascade of effects via environments, through brains, to behaviour and hence to the desired impact, such as improved health. Changes in behaviour are viewed as the consequence of a reinforcement learning process involving the targeting of evolved motives and changes to behaviour settings, and are produced by three types of behavioural control mechanism (automatic, motivated and executive). The implications are that interventions must create surprise, revalue behaviour and disrupt performance in target behaviour settings. We then describe a sequence of five steps required to design an intervention to change specific behaviours: Assess, Build, Create, Deliver and Evaluate. The BCD approach has been shown to change hygiene, nutrition and exercise-related behaviours and has the advantages of being applicable to product, service or institutional design, as well as being able to incorporate future developments in behaviour science. We therefore argue that BCD can become the foundation for an applied science of behaviour change.

Factors influencing implementation of a survivorship care plan-a quantitative process evaluation of the ROGY Care trial

Purpose

The aim of this study is to investigate the factors that influence implementation of Survivorship Care Plans (SCPs) in the intervention arm of the ROGY Care trial by (1) assessing the level of SCP receipt in the ROGY Care trial and (2) identifying patient- and provider-level factors that influence SCP receipt.

Methods

Between 2011 and 2015, a pragmatic cluster randomized-controlled-trial was conducted on the effects of automatically generated SCPs. Endometrial (N = 117) and ovarian (N = 61) cancer patients were allocated to ‘SCP care’, as provided by their SCP care providers (N = 10). Associations between SCP receipt (self-reported SCP receipt and actually generated SCPs), patient-factors (socio-demographic-, clinical-, and personality factors), and care provider factors (profession and a-priori motivation regarding SCP provision) were tested in univariate analysis. The odds ratios of factors influencing self-reported SCP receipt were estimated with a multivariate regression model.

Results

Of all patients in the SCP care arm (N = 178), SCPs were generated by the care provider for 90 % of the patients and 70 % of the patients reported that they had received an SCP. Patients with older age, ovarian cancer, type D (distressed) personality, and patients that completed the questionnaire a longer period of time after the SCP consult were more likely to report no SCP receipt.

Conclusions

SCP receipt was influenced by patient- but not care-provider factors.

Implications for cancer survivors

Certain patient groups were less likely to report SCP receipt. Whether all patients are in need of an SCP, requires further investigation. If they do, more efforts need to be made towards the implementation of SCPs.

Fidelity of Implementation of a State Antibullying Policy With a Focus on Protected Social Classes

Bullying threatens the mental and educational well-being of students. All states have enacted antibullying laws. This study surveyed 634 educators about the implementation of the North Carolina School Violence Prevention Act, which enumerated social classes protected from bullying: race, national origin, gender, socioeconomic status, sexual orientation, gender identity, appearance, and disability. Results showed that local antibullying policies most often included race as a protected class and least often included sexual orientation and gender identity. More educators had been trained on bullying based on race than any other social class. Students were more often informed that bullying based on race was prohibited and were least often informed about prohibitions regarding sexual orientation and gender identity. Reporting, investigating, and remediating bullying was highest for racial bullying, followed by disability bullying, and was lowest for bullying based on sexual orientation and gender identity.

A modified theoretical framework to assess implementation fidelity of adaptive public health interventions

BACKGROUND

One of the major debates in implementation research turns around fidelity and adaptation. Fidelity is the degree to which an intervention is implemented as intended by its developers. It is meant to ensure that the intervention maintains its intended effects. Adaptation is the process of implementers or users bringing changes to the original design of an intervention. Depending on the nature of the modifications brought, adaptation could either be potentially positive or could carry the risk of threatening the theoretical basis of the intervention, resulting in a negative effect on expected outcomes. Adaptive interventions are those for which adaptation is allowed or even encouraged. Classical fidelity dimensions and conceptual frameworks do not address the issue of how to adapt an intervention while still maintaining its effectiveness.

DISCUSSION

We support the idea that fidelity and adaptation co-exist and that adaptations can impact either positively or negatively on the intervention's effectiveness. For adaptive interventions, research should answer the question how an adequate fidelity-adaptation balance can be reached. One way to address this issue is by looking systematically at the aspects of an intervention that are being adapted. We conducted fidelity research on the implementation of an empowerment strategy for dengue prevention in Cuba. In view of the adaptive nature of the strategy, we anticipated that the classical fidelity dimensions would be of limited use for assessing adaptations. The typology we used in the assessment-implemented, not-implemented, modified, or added components of the strategy-also had limitations. It did not allow us to answer the question which of the modifications introduced in the strategy contributed to or distracted from outcomes. We confronted our empirical research with existing literature on fidelity, and as a result, considered that the framework for implementation fidelity proposed by Carroll et al. in 2007 could potentially meet our concerns. We propose modifications to the framework to assess both fidelity and adaptation. The modified Carroll et al.'s framework we propose may permit a comprehensive assessment of the implementation fidelity-adaptation balance required when implementing adaptive interventions, but more empirical research is needed to validate it.

Protocol of a mixed method, randomized controlled study to assess the efficacy of a psychosocial intervention to reduce fatigue in patients with End-Stage Renal Disease (ESRD)

BACKGROUND

Patients with end-stage renal disease (ESRD) commonly suffer from severe fatigue, which strongly impacts their quality of life (QoL). Although fatigue is often attributed to disease- and treatment characteristics, research also shows that behavioural, psychological and social factors affect perceived fatigue in dialysis patients. Whereas studies on fatigue in other chronic patient groups suggest that psychological or psychosocial interventions are effective in reducing fatigue, such interventions are not yet available for ESRD patients on dialysis treatment. The objective of this study is to examine the efficacy of a psychosocial intervention for dialysis patients aimed at reducing fatigue (primary outcome) and improving QoL (secondary outcome). The intervention consists of counselling sessions led by a social worker. The implementation process and patients' and social workers' expectations and experiences with the intervention will also be evaluated.

METHODS/DESIGN

This study follows a mixed-methods design in which both quantitative and qualitative data will be collected. A multi-centre, randomised controlled trial (RCT) with repeated measures will be conducted to quantitatively assess the efficacy of the psychosocial intervention in reducing fatigue and improving QoL in ESRD patients. Additional secondary outcomes and medical parameters will be assessed. Outcomes will be compared to patients receiving usual care. A sample of 74 severely fatigued dialysis patients will be recruited from 10 dialysis centres. Patients will be randomly assigned to the intervention or control group. Outcomes will be assessed at baseline, post intervention/16 weeks, and at three and six-month follow-ups. A qualitative process evaluation will be conducted parallel to/following the effectiveness RCT. Interviews and focus groups will be conducted to gain insight into patients' and social workers' perspectives on outcomes and implementation procedures. Implementation fidelity will be assessed by audio-taped and written registrations. Participatory methods ensure the continuous input of experiential knowledge, improving the quality of study procedures and the applicability of outcomes.

DISCUSSION

This is the first mixed method study (including an RCT and qualitative process evaluation) to examine the effect and implementation process of a psychosocial intervention on reducing fatigue and improving QoL in ESRD patients on dialysis treatment.

TRIAL REGISTRATION

NTR5366 , The Netherlands National Trial Register (NTR), registered August 26, 2015.

Assessing Fidelity of a Community-Based Opioid Overdose Prevention Program: Modification of the Fidelity Checklist

BACKGROUND

The evaluation and maintenance of treatment fidelity is an important methodological consideration in intervention research. Treatment fidelity is the degree to which interventions are delivered as outlined by the program developers. A lack of fidelity to the intervention model has the potential to be a large source of error, including type I and type II error. Monitoring and assuring fidelity is critical to assuring the validity of the interventions. The Fidelity Checklist is a reliable and valid tool designed to measure fidelity in a group-based parenting program. The Fidelity Checklist assesses group leaders' maintenance of the intervention protocol (adherence) and their group facilitation and process skills (competence).

OBJECTIVES

The purpose of this article is to describe the systematic process of modifying the Fidelity Checklist for use in a community-based opioid overdose prevention group educational intervention.

METHODS

A biphasic approach was used to modify the Fidelity Checklist. Phase 1 included engaging key stakeholders during a full-day meeting to determine adherence subscale components. Phase 2 included: (a) the modification of the competence subscale of the Fidelity Checklist, (b) adaptation of the coding manual defining the components of the Fidelity Checklist, and (c) creating a fidelity checklist to guide interventionists.

RESULTS

The biphasic systematic modification approach resulted in a revised Fidelity Checklist that was successfully used to measure treatment fidelity in a community-based opioid overdose prevention program.

CLINICAL RELEVANCE

The findings of this study provide information for nurse researchers and nurse educators for modifying a fidelity checklist that can be used to enhance community-based educational interventions.

Evaluating Measures of Fidelity for Substance Abuse Group Treatment With Incarcerated Adolescents

The evaluation of treatment fidelity has become increasingly important as the demand for evidence-based practice grows. The purpose of the present study is to describe the psychometric properties of two measures of treatment fidelity that can be used by therapists and supervisors - one for group-based cognitive-behavioral therapy (CBT) and one for combined Substance Education and Twelve-Step Introduction (SET) for adolescent substance use. At the end of group sessions (CBT n=307; SET n=279), therapists and supervisors completed an evaluation measure assessing adherence to certain core components of the intervention. The supervisor version of the fidelity measure also included items for rating the level of competency the therapist demonstrated when providing each component of the intervention. Results from split-half cross-validation analyses provide strong support for an 11-item, three-factor CBT fidelity measure. Somewhat less consistent but adequate support for a nine-item, two-factor SET fidelity measure was found. Internal consistencies ranged from acceptable to good for both the CBT and SET adherence scales and from acceptable to good for the CBT and SET competency scales, with the exception of the CBT practices competency scale. Preliminary validation of the measures suggests that both measures have adequate to strong factor structure, reliability, and concurrent and discriminant validity. The results of this study have implications for research and clinical settings, including the supervision process.

Problems, Solutions, and Strategies Reported by Users of Transcutaneous Electrical Nerve Stimulation for Chronic Musculoskeletal Pain: Qualitative Exploration Using Patient Interviews

BACKGROUND

Transcutaneous electrical nerve stimulation (TENS) could offer a non-drug form of pain relief, but there is no consensus regarding its effectiveness for chronic musculoskeletal pain or chronic low back pain. A recent review of previous trial methods identified significant problems with low treatment fidelity. There is little information available to inform the development of a pragmatic implementation design for a TENS evaluation.

OBJECTIVES

The aim of this study was to explore the experiences of patients who were receiving secondary care in a pain clinic and who had expertise in using TENS to manage chronic musculoskeletal pain. These key informants were selected because they had the potential to generate knowledge that could inform research design and clinical practice.

DESIGN

A qualitative method using individual semistructured interviews with open questions was selected for its capacity to generate rich data.

METHODS

Semistructured interviews were conducted with 9 patients (6 women, 3 men). Thematic analysis was used as the primary data analysis method, and this analysis was enhanced by a case-level analysis of the context and processes of TENS use of each individual.

RESULTS

Data analysis indicated that patients learned to address a range of problems in order to optimize TENS use. Patients may need to personalize the positioning of electrodes and the TENS settings and to readjust them over time. Patients learned to use TENS in a strategic manner, and the outcomes of each strategy varied.

CONCLUSIONS

The findings indicated that a pragmatic TENS evaluation may need to incorporate a learning phase to allow patients to optimize this complex pattern of TENS usage, and evaluation may need to be sensitive to the outcomes of strategic use. These findings also have implications for clinical practice.

Evaluation of the implementation process and outcomes of a professional pharmacy service in a community pharmacy setting. A case report

Pharmacist-led medication review services are recognized as a key to medicines management. This case study describes the implementation process of a medication review with follow-up service in a community pharmacy setting and evaluates its initial outcomes. An implementation-effectiveness hybrid study was undertaken in a community pharmacy setting. The implementation process was divided into four different phases: Exploration and adoption, program installation, initial implementation, and full operation. A core set of implementation outcomes was measured, including penetration, implementation costs, feasibility, fidelity, acceptability, appropriateness and efficiency. The penetration rate of the service was nearly 62.5%, and the implementation costs were 57,359.67€. There was a high retention-participation rate of patients. For every month of service provision, there was a 1.27 increase in the number of patients requesting the service, compared to the number of patients being offered the service. The time spent on service provision was 171.7 min per patient. Average patient satisfaction with the service was 4.82 (SD: 0.39, scale 1-5), and the acceptance rate of care plans by patients and general medical practitioners were 96.99% and 96.46%, respectively. There were 408 negative outcomes associated with the use of medications were identified during the study (3.09 per patient), of which 96.3% were resolved. The average time per patient spent on service provision significantly decreased along the 18 months of service provision. This case report can assist individual pharmacists and professional organizations interested in implementing evidence-based services by offering an example on how to approach the implementation process in a systematic way.

Development and testing of two implementation tools to measure components of professional pharmacy service fidelity

RATIONALE, AIMS AND OBJECTIVES

There is a need to evaluate both service process and implementation outcomes as professional services are being implemented into pharmacy practice. Fidelity is an implementation outcome, which may be used for service optimization, by associating service components to patient outcomes, as well as use in process evaluation. The objective of this study was to develop tools to measure components of fidelity, specifically, an adherence index (adherence of the service provider to the elements of the service) and a patient responsiveness scale for the professional pharmacy service, medication review with follow-up.

METHODS

The procedure described by DeVellis was followed to develop the tools. An expert panel was used to create items and establish content validity. Primary data were collected from 190 service provider pharmacists from 128 pharmacies across 11 provinces of Spain using Spanish version tools as part of an ongoing implementation study (English translations appended to the online version of the article as supplementary material). An initial assessment of item functionality was performed using descriptive statistics and item discrimination for both tools. The patient responsiveness scale's internal consistency was confirmed by calculating Cronbach's alpha coefficient and inter-item correlations. In addition, for the patient responsiveness scale, the number of factors to retain was based on Kaiser criterion, parallel analysis and Cattell's scree test and the number of items was optimized as guided by iterative exploratory factor analysis (EFA).

RESULTS

Acceptability of both tools was high. An adherence index of 39 items was developed. After five EFA iterations, four items were removed, resulting in a reliable, 12-item, two-factor patient responsiveness scale, explaining 53.9% of total variance.

CONCLUSIONS

Two tools for measuring implementation fidelity, an adherence index and a patient responsiveness scale, have been developed and tested. Future assessment, in particular to establish criterion validity, is recommended.

Implementing the Health Promoting University approach in culturally different contexts: a systematic review

Introduction:

Universities represent a valuable opportunity to promote health and well-being. Based on the setting approach, the Health Promoting Universities concept has been developed in different countries and contexts. However, the implementation process remains poorly documented. This systematic review aims to describe how universities have implemented the Health Promoting University concept in different cultural contexts.

Methods:

Pubmed, Medline, Lilacs and Scielo were searched for articles on Health Promoting Universities, published between 1995 and 2015. Studies detailing the implementation of a Health Promoting University approach were included. Selected articles were content analysed paying attention to: (a) the definition of a Health Promoting University; (b) priority areas of action; (c) items of work; (d) coordination of the project; (e) evaluation; and (f) adaptation to the cultural context.

Results:

Twelve studies were identified for in-depth analysis. Of those, three were theoretical papers, and nine were intervention studies. The programmes described in the selected studies are mostly based on the guidelines of the Edmonton Charter. They incorporated the main areas of action and items of works proposed by the Health Promoting University framework. The implementation of healthy policies and incorporation of health promotion in the curriculum are remaining challenges. Strategies to facilitate adaptation to context include: stakeholder participation in planning and implementation, adaptation of educational material and analysis of needs.

Conclusions:

The review suggests that most of the universities work towards similar goals, relying on the Health Promoting University framework, yet that the way in which initiatives are implemented depends on the context.

Predictive Factors of Spontaneous Reporting of Adverse Drug Reactions among Community Pharmacists

Purpose

To evaluate the association between spontaneous reporting (SR) and the knowledge, attitude, and needs of community pharmacists (CPs), using a questionnaire following a conceptual model known as the mixed model of knowledge-attitude-practices and the satisfaction of needs.

Methods

Self-administered questionnaires were used with a nationwide convenience sample of CPs between September 1, 2014 and November 25, 2014 in Korea. The association between SR and the predictive factors was evaluated using multivariate logistic regression analysis.

Results

In total, 1,001 questionnaires were analyzed. The mean age of the respondents and the number of years spent in community pharmacy practice were 45.6 years and 15.3 years, respectively. CPs with experience of SR was 29.4%. Being older than 60 (ORadj, 0.16; 95% CI, 0.06–0.42), having prior experience with adverse drug reactions (ADR) (ORadj, 6.46; 95% CI, 2.46–16.98), having higher specific knowledge of SR (ORadj, 3.58; 95% CI, 1.96–6.56), and having less concern about the obstacles to SR (ORadj, 0.36; 95% CI, 0.23–0.57) were significant contributing factors to SR. The main obstacles to SR included perception of ADRs as ‘not serious ADR’ (77.9%), ‘already well known ADR’ (81.5%), and ‘uncertain about causality’ (73.3%). CPs without reporting experience had greater concerns related to the reporting method and the liability of the pharmacy than those with reporting experience (p<0.05).

Conclusions

Findings from our study showed around one in three CPs had ADR reporting experience in Korea, while 87.1% had prior experience with ADR cases. The knowledge of SR, prior experience of ADR, and less concern about the obstacles to SR were contributing factors for reporting levels.

The Aging, Community and Health Research Unit-Community Partnership Program for older adults with type 2 diabetes and multiple chronic conditions: a feasibility study

BACKGROUND

Few studies have examined the effectiveness of community-based self-management interventions in older adults with type 2 diabetes mellitus (T2DM) and multiple chronic conditions (MCC). The objectives of this study were to examine the feasibility of implementation in practice (primary) and the feasibility of study methods and potential effectiveness (secondary) of the Aging, Community and Health-Community Partnership Program, a new 6-month interprofessional, nurse-led program to promote diabetes self-management in older adults (>65 years) with T2DM and MCC.

METHODS

This study used a prospective one-group pre-test/post-test design. Participants were recruited from a specialized diabetes clinic. They received a median of three in-home/clinic visits by certified diabetes educators (CDEs) and attended a median of three group wellness sessions provided by the CDEs in partnership with a community-based seniors' association. The primary outcome was the feasibility of the program (acceptability, fidelity, implementation barriers/facilitators). Secondary outcomes included the feasibility of the study methods (recruitment/retention rates and procedures, eligibility criteria, data collection and analysis methods) and potential effectiveness of the program based on 6-month changes in self-reported outcomes including self-management behavior (diet, exercise, self-monitoring), health status (quality of life, mental health), and costs of service use. Analysis of feasibility outcomes was primarily based on descriptive statistics. The potential effectiveness of the program was explored using different tests, with the results expressed using descriptive statistics and effect estimates (95 % confidence intervals).

RESULTS

In total, 45 (88 %) of 51 eligible persons consented to participate. Of these, 37 (82 %) completed the 6-month follow-up. Participants and providers viewed the program as acceptable and feasible. Participants had a higher SF-12 physical component summary score at 6 months compared with baseline (mean score difference 3.0, 95 % CI 0.2-5.8). Median costs for diabetes care increased over 6 months (reflecting inclusion of program costs), while other service costs either decreased or remained unchanged.

CONCLUSIONS

This study offers preliminary evidence that the program was feasible to deliver and acceptable to participants and providers. Initial results suggest that the program may improve physical functioning. A randomized controlled trial is feasible, with some adaptations to the program and study methods that were identified from this feasibility study.

TRIAL REGISTRATION

Clinicaltrials.gov identifier: NCT01880476.

Challenges, solutions and future directions in the evaluation of service innovations in health care and public health

Evaluating service innovations in health care and public health requires flexibility, collaboration and pragmatism; this collection identifies robust, innovative and mixed methods to inform such evaluations.

Integrating care for frequent users of emergency departments: implementation evaluation of a brief multi-organizational intensive case management intervention

Background

Addressing the needs of frequent users of emergency departments (EDs) is a health system priority in many jurisdictions. This study describes stakeholder perspectives on the implementation of a multi-organizational brief intervention designed to support integration and continuity of care for frequent ED users with mental health and addictions problems, focusing on perceived barriers and facilitators to early implementation in a large urban centre.

Methods

Coordinating Access to Care from Hospital Emergency Departments (CATCH-ED) is a brief case management intervention bridging hospital, primary and community care for frequent ED users experiencing mental illness and addictions. To examine barriers and facilitators to early implementation of this multi-organizational intervention, between July and October 2012, 47 stakeholders, including direct service providers, managers and administrators participated in 32 semi-structured qualitative interviews and one focus group exploring their experience with the intervention and factors that helped or hindered successful early implementation. Qualitative data were analyzed using thematic analysis.

Results

Stakeholders valued the intervention and its potential to support continuity of care for this population. Service delivery system factors, including organizational capacity and a history of collaborative relationships across the healthcare continuum, and support system factors, such as training and supervision, emerged as key facilitators of program implementation. Operational challenges included early low program referral rates, management of a multi-organizational initiative, variable adherence to the model among participating organizations, and scant access to specialty psychiatric resources. Factors contributing to these challenges included lack of dedicated staff in the ED and limited local system capacity to support this population, and insufficient training and technical assistance available to participating organizations.

Conclusions

A multi-organizational brief intervention is an acceptable model to support integration of hospital, primary and community care for frequent ED users. The study highlights the importance of early implementation evaluation to identify potential solutions to implementation barriers that may be applicable to many jurisdictions.

Media Competition Implementation for the Massachusetts Childhood Obesity Research Demonstration Study (MA-CORD): Adoption and Reach

The Massachusetts Childhood Obesity Research Demonstration Study (MA-CORD) was a multi-level, multi-sector community intervention with a media competition component to provide an overarching synergy and promote awareness of target behaviors to reduce childhood obesity. Students participating in the media competition were tasked with developing videos, song/rap lyrics, and artwork that reflected the goals. The aim of this study is to document the process used to develop and implement the media competition along with its reach and adoption. An adapted version of Neta and colleagues’ 2015 framework on dissemination and implementation was used to summarize the process by which the media competition was developed and implemented. Adoption was defined by whether eligible schools or afterschool programs decided to implement the media competition. Reach was defined by student participation rates within schools/programs and the number of votes cast for the finalists on the coalition website and students’ paper ballots. A total of 595 students participated in the media competition from 18 school and afterschool programs in two communities. Adoption of the media competitions ranged from 22% to 100% in programs and reach ranged from 3% to 33% of the student population. The documentation of the implementation should contribute to the replication of the media competition.

Effectiveness of the STOPP/START (Screening Tool of Older Persons' potentially inappropriate Prescriptions/Screening Tool to Alert doctors to the Right Treatment) criteria: systematic review and meta-analysis of randomized controlled studies

WHAT IS KNOWN AND OBJECTIVE

STOPP/START are explicit screening tools that identify potentially inappropriate prescribing in older adults. Our objective was to update our 2013 systematic review that showed limited evidence of impact, using new evidence from randomized controlled trials (RCTs) assessing clinical, humanistic and economic outcomes in older adults.

METHODS

We performed a search of PubMed, EMBASE, CINAHL, Web of Science and grey literature for RCTs published in English since the previous review through June 2014. The Cochrane Risk of Bias Tool was used. We performed a meta-analysis on the effect of STOPP on potentially inappropriate medication (PIM) rates and a narrative synthesis on other outcomes.

RESULTS AND DISCUSSION

Four RCTs (n = 1925 adults) from four countries were included, reporting both acute (n = 2) and long-term care (n = 2) patients. Studies differed in implementation. Two studies were judged to have low risk, and two to have moderate-to-high risk of bias in key domains. Meta-analysis found that the STOPP criteria reduced PIM rates in all four studies, but study heterogeneity (I(2) = 86·7%) prevented the calculation of a meaningful statistical summary. We found evidence that use of the criteria reduces falls, delirium episodes, hospital length-of-stay, care visits (primary and emergency) and medication costs, but no evidence of improvements in quality of life or mortality.

WHAT IS NEW AND CONCLUSION

STOPP/START may be effective in improving prescribing quality, clinical, humanistic and economic outcomes. Additional research investigating these tools is needed, especially in frail elderly and community-living patients receiving primary care.

Measuring implementation strength: lessons from the evaluation of public health strategies in low- and middle-income settings

Evaluation of strategies to ensure evidence-based, low-cost interventions reach those in need is critical. One approach is to measure the strength, or intensity, with which packages of interventions are delivered, in order to explore the association between implementation strength and public health gains. A recent systematic review suggested methodological guidance was needed. We described the approaches used in three examples of measures of implementation strength in evaluation. These addressed important public health topics with a substantial disease burden in low-and middle-income countries; they involved large-scale implementation; and featured evaluation designs without comparison areas. Strengths and weaknesses of the approaches were discussed. In the evaluation of Ethiopia’s Health Extension Programme, implementation strength scoring for each kebele (ward) was based on aggregated data from interviews with mothers of children aged 12–23 months, reflecting their reports of contact with four elements of the programme. An evaluation of the Avahan HIV prevention programme in India used the cumulative amount of Avahan funding per HIV-infected person spent each year in each district. In these cases, a single measure was developed and the association with hypothesised programme outcomes presented. In the evaluation of the Affordable Medicines Facility—malaria, several implementation strength measures were developed based on the duration of activity of the programme and the level of implementation of supporting interventions. Measuring the strength of programme implementation and assessing its association with outcomes is a promising approach to strengthen pragmatic impact evaluation. Five key aspects of developing an implementation strength measure are to: (a) develop a logic model; (b) identify aspects of implementation to be assessed; (c) design and implement data collection from a range of data sources; (d) decide whether and how to combine data into a single measure; and, (e) plan whether and how to use the measure(s) in outcome analysis.

Adherence to Tobacco Dependence Treatment Among HIV-Infected Smokers

High prevalence of tobacco use and low success in quitting remain significant problems for reducing disease burden among HIV-infected persons. This study's purpose was to examine participant responsiveness and tobacco dependence treatment adherence and their influences on tobacco abstinence among HIV-infected patients. This non-randomized study included HIV-infected smokers 18 years of age or older, who smoked at least 5 cigarettes per day, and had an interest in quitting smoking in the next 30 days. HIV-infected smokers (n = 247) received a 12-week tobacco dependence treatment intervention that included pharmacotherapy and telephone counseling. Younger age and non-White race were associated with lower adherence to pharmacotherapy. Younger age, non-White race, and increased monthly binge drinking were associated with lower adherence to telephone counseling. High participant responsiveness was associated with adherence to pharmacotherapy, counseling, and abstinence. Development and testing of interventions to improve adherence to evidence-based tobacco dependence treatment is warranted.

Figuring out fidelity: a worked example of the methods used to identify, critique and revise the essential elements of a contextualised intervention in health policy agencies

BACKGROUND

In this paper, we identify and respond to the fidelity assessment challenges posed by novel contextualised interventions (i.e. interventions that are informed by composite social and psychological theories and which incorporate standardised and flexible components in order to maximise effectiveness in complex settings). We (a) describe the difficulties of, and propose a method for, identifying the essential elements of a contextualised intervention; (b) provide a worked example of an approach for critiquing the validity of putative essential elements; and (c) demonstrate how essential elements can be refined during a trial without compromising the fidelity assessment. We used an exploratory test-and-refine process, drawing on empirical evidence from the process evaluation of Supporting Policy In health with Research: an Intervention Trial (SPIRIT). Mixed methods data was triangulated to identify, critique and revise how the intervention's essential elements should be articulated and scored.

RESULTS

Over 50 provisional elements were refined to a final list of 20 and the scoring rationalised. Six (often overlapping) challenges to the validity of the essential elements were identified. They were (1) redundant-the element was not essential; (2) poorly articulated-unclear, too specific or not specific enough; (3) infeasible-it was not possible to implement the essential element as intended; (4) ineffective-the element did not effectively deliver the change principles; (5) paradoxical-counteracting vital goals or change principles; or (6) absent or suboptimal-additional or more effective ways of operationalising the theory were identified. We also identified potentially valuable 'prohibited' elements that could be used to help reduce threats to validity.

CONCLUSIONS

We devised a method for critiquing the construct validity of our intervention's essential elements and modifying how they were articulated and measured, while simultaneously using them as fidelity indicators. This process could be used or adapted for other contextualised interventions, taking evaluators closer to making theoretically and contextually sensitive decisions upon which to base fidelity assessments.

Assessing Fidelity of Implementation (FOI) for School-Based Mindfulness and Yoga Interventions: A Systematic Review

As school-based mindfulness and yoga programs gain popularity, the systematic study of fidelity of program implementation (FOI) is critical to provide a more robust understanding of the core components of mindfulness and yoga interventions, their potential to improve specified teacher and student outcomes, and our ability to implement these programs consistently and effectively. This paper reviews the current state of the science with respect to inclusion and reporting of FOI in peer-reviewed studies examining the effects of school-based mindfulness and/or yoga programs targeting students and/or teachers implemented in grades kindergarten through twelve (K-12) in North America. Electronic searches in PsychInfo and Web of Science from their inception through May 2014, in addition to hand searches of relevant review articles, identified 312 publications, 48 of which met inclusion criteria. Findings indicated a relative paucity of rigorous FOI. Fewer than 10% of studies outlined potential core program components or referenced a formal theory of action, and fewer than 20% assessed any aspect of FOI beyond participant dosage. The emerging nature of the evidence base provides a critical window of opportunity to grapple with key issues relevant to FOI of mindfulness-based and yoga programs, including identifying essential elements of these programs that should be faithfully implemented and how we might develop rigorous measures to accurately capture them. Consideration of these questions and suggested next steps are intended to help advance the emerging field of school-based mindfulness and yoga interventions.

Randomized trial of a prevention intervention that embeds weight-related messages within a general parenting program

OBJECTIVE

To assess the extent to which an obesity prevention intervention that embeds obesity-related messages within a parenting program, compared with controls who received weekly mailings, resulted in a smaller increase in children's BMI (primary outcome) and improvements in weight-related behaviors from baseline to 9-month follow-up.

METHODS

Fifty-six families were randomly assigned to the intervention and 56 to control. Children were primarily Hispanic (58%) or Black/African American (23%). Intervention included nine weekly: group parenting sessions, children's sessions, and homework assignments. At baseline, post-intervention, and 9-month follow-up, staff assessed children's weight and height. Parents completed surveys assessing parenting skills, feeding behaviors, and children's weight-related behaviors.

RESULTS

From baseline to 9-month follow-up, BMI decreased by a mean of 0.13 kg m(-2) among children in the intervention and increased by 0.21 kg m(-2) among children in the control, resulting in a nonsignificant difference (multivariate adjusted difference = -0.36; 95% confidence interval [CI] -1.23, 0.51; P = 0.41). Parents in the intervention decreased restrictive feeding practices relative to control (-0.30; 95% CI -0.53, -0.07; P = 0.01). Intervention and control arms showed similar changes in children's weight-related behaviors.

CONCLUSIONS

The intervention improved restrictive feeding but did not influence children's BMI or weight-related behaviors compared to controls who received weekly mailings.

An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention

The valid and reliable assessment of fidelity is critical at all stages of intervention research and is particularly germane to interpreting the results of efficacy and implementation trials. Ratings of protocol adherence typically are reliable, but ratings of therapist competence are plagued by low reliability. Because family context and case conceptualization guide the therapist's delivery of interventions, the reliability of fidelity ratings might be improved if the coder is privy to client context in the form of an ecological assessment. We conducted a randomized experiment to test this hypothesis. A subsample of 46 families with 5-year-old children from a multisite randomized trial who participated in the feedback session of the Family Check-Up (FCU) intervention were selected. We randomly assigned FCU feedback sessions to be rated for fidelity to the protocol using the COACH rating system either after the coder reviewed the results of a recent ecological assessment or had not. Inter-rater reliability estimates of fidelity ratings were meaningfully higher for the assessment information condition compared to the no-information condition. Importantly, the reliability of the COACH mean score was found to be statistically significantly higher in the information condition. These findings suggest that the reliability of observational ratings of fidelity, particularly when the competence or quality of delivery is considered, could be improved by providing assessment data to the coders. Our findings might be most applicable to assessment-driven interventions, where assessment data explicitly guides therapist's selection of intervention strategies tailored to the family's context and needs, but they could also apply to other intervention programs and observational coding of context-dependent therapy processes, such as the working alliance.

Factors affecting adoption, implementation fidelity, and sustainability of the Redesigned Community Health Fund in Tanzania: a mixed methods protocol for process evaluation in the Dodoma region

Background

Despite the implementation of various initiatives to address low enrollment in voluntary micro health insurance (MHI) schemes in sub-Saharan Africa, the problem of low enrollment remains unresolved. The lack of process evaluations of such interventions makes it difficult to ascertain whether their poor results are because of design failures or implementation weaknesses.

Objective

In this paper, we describe a process evaluation protocol aimed at opening the ‘black box’ to evaluate the implementation processes of the Redesigned Community Health Fund (CHF) program in the Dodoma region of Tanzania.

Design

The study employs a cross-sectional mixed methods design and is being carried out 3 years after the launch of the Redesigned CHF program. The study is grounded in a conceptual framework which rests on the Diffusion of Innovation Theory and the Implementation Fidelity Framework. The study utilizes a mixture of quantitative and qualitative data collection tools (questionnaires, focus group discussions, in-depth interviews, and document review), and aligns the evaluation to the Theory of Intervention developed by our team. Quantitative data will be used to measure program adoption, implementation fidelity, and their moderating factors. Qualitative data will be used to explore the responses of stakeholders to the intervention, contextual factors, and moderators of adoption, implementation fidelity, and sustainability.

Discussion

This protocol describes a systematic process evaluation in relation to the implementation of a reformed MHI. We trust that the theoretical approaches and methodologies described in our protocol may be useful to inform the design of future process evaluations focused on the assessment of complex interventions, such as MHI schemes.

Good practice in school based alcohol education programmes

OBJECTIVE

To identify elements of good practice in designing and delivering alcohol education programmes in schools.

METHODS

Literature reviews and published programme evaluations were used to identify key elements of good practice.

RESULTS

Principles of good practive are identified and discussed. Five main issues are highlighted: choosing a universal or targetted approach, the need for theoretical frameworks, adopting a stand-alone or multi-component approach; issues of delivery and programme fidelity, and balancing programme fidelity and cultural relevance.

CONCLUSIONS

Programme objectives, programme fidelity and cultural context are important factors in designing programmes and will influence outcomes and evaluation of success.

PRACTICE IMPLICATIONS

In developing alcohol education programmes, there is a need to draw on the evidence and experience accrued from previous efforts. Programme development and implementation can draw on results from evaluated programmes to design alcohol education programmes suited to specific contexts, the availability of resources, the perceived needs of the target group and the problem to be addressed.

Investigating variations in implementation fidelity of an organizational-level occupational health intervention

BACKGROUND

The workplace has been suggested as an important arena for health promotion, but little is known about how the organizational setting influences the implementation of interventions.

PURPOSE

The aims of this study are to evaluate implementation fidelity in an organizational-level occupational health intervention and to investigate possible explanations for variations in fidelity between intervention units.

METHOD

The intervention consisted of an integration of health promotion, occupational health and safety, and a system for continuous improvements (Kaizen) and was conducted in a quasi-experimental design at a Swedish hospital. Implementation fidelity was evaluated with the Conceptual Framework for Implementation Fidelity and implementation factors used to investigate variations in fidelity with the Framework for Evaluating Organizational-level Interventions. A multi-method approach including interviews, Kaizen notes, and questionnaires was applied.

RESULTS

Implementation fidelity differed between units even though the intervention was introduced and supported in the same way. Important differences in all elements proposed in the model for evaluating organizational-level interventions, i.e., context, intervention, and mental models, were found to explain the differences in fidelity.

CONCLUSION

Implementation strategies may need to be adapted depending on the local context. Implementation fidelity, as well as pre-intervention implementation elements, is likely to affect the implementation success and needs to be assessed in intervention research. The high variation in fidelity across the units indicates the need for adjustments to the type of designs used to assess the effects of interventions. Thus, rather than using designs that aim to control variation, it may be necessary to use those that aim at exploring and explaining variation, such as adapted study designs.

Using the Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ) to identify practice patterns

Primary care-mental health integration (PC-MHI) is growing in popularity. To determine program success, it is essential to know if PC-MHI services are being delivered as intended. The investigation examines responses to the Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ) to explore PC-MHI provider practice patterns. Latent class analysis was used to identify clusters of PC-MHI providers based on their self-report of adherence on the PPAQ. Analysis revealed five provider clusters with varying levels of adherence to PC-MHI model components. Across clusters, adherence was typically lowest in relation to collaboration with other primary care staff. Clusters also differed significantly in regard to provider educational background and psychotherapy approach, level of clinic integration, and previous PC-MHI training. The PPAQ can be used to identify PC-MHI provider practice patterns that have relevance for future clinical effectiveness studies, development of provider training, and quality improvement initiatives.

Impact of including regional anaesthesia in enhanced recovery protocols: a scoping review

Regional anaesthesia (RA) is often included in enhanced recovery protocols (ERPs) as an important component of a bundle of interventions to improve outcomes after surgery. We sought to delineate whether the literature supports the use of RA in this setting with regard to commonly measured outcomes. We further sought to assess whether such improvements would translate into positive impacts on healthcare value as defined by the Institute for Healthcare Improvement Triple Aim. We conducted a scoping review to address our objectives. Studies of ERPs that included RA and reported at least one outcome of interest in comparison to a control group were included. MEDLINE, EMBASE, CENTRAL, CDSR, PROSPERO, and the NHS Economic Evaluation Database were searched up to May 2015. Two reviewers assessed studies and extracted data. Of 695 identified citations, 58 studies were included for analysis. The majority (53%) were in colorectal surgery. Positive impacts of RA on all outcomes were identified; however, value-based outcomes were rarely reported. Where value-based outcomes were reported, RA appears to have a positive impact on global measures of health and function and on economic outcomes. Existing literature supports a positive impact of RA on ERP outcomes, which may be reflected in improved healthcare value. In order to justify the value of RA in ERPs, a future focus on appropriate measures is needed to align research with widely accepted frameworks, such as the Triple Aim.

Developing Brief Opportunistic Interactions: practitioners facilitate patients to identify and change health risk behaviours at an early preventive stage

Aim To identify shortcomings in existing models of patient behaviour change, and present the development and testing of a novel approach using practitioner facilitation and person-focussed conversations that identifies and addresses behaviours at an earlier stage than current models.

BACKGROUND

Systematic strategies used by health professionals to change patient behaviours began with motivational interviewing and brief intervention approaches for serious addictive behaviours. Practitioners typically presume they should drive the process of patient behaviour change. Attempts to transfer these approaches to primary care, and a broader range of health risk behaviours, have been less successful. The TADS programme (Tobacco, Alcohol and Other Drugs, later Training and Development Services) began teaching motivational interviewing and brief interventions to practitioners in New Zealand in 1996. Formal and informal evaluations showed that practitioners used screening tools that patients rejected and that led to incomplete disclosure, used language that did not engage patients, failed to identify the behaviours patients wished to address and therefore misdirected interventions.

METHODS

Iterative development of new tools with input from patients and primary care clinicians. Findings The TADS programme developed a questionnaire whose results remained private to the patient, which enabled the patient to identify personal behaviours that they might choose to change (the TADS Personal Assessment Choice Tool). This was assisted by a brief conversation that facilitated and supported any change prioritised by the patient (the TADS Brief Opportunistic Interaction). The need for this approach, and its effectiveness, appeared to be similar across adults, youth, different ethnic groups and people in different socio-economic circumstances. Behaviours patients identified were often linked to other health risk behaviours or early-stage mental health disorders that were not easily detected by practitioner-driven screening or inquiry. The long-term effectiveness of this approach in different populations in primary health care settings requires further evaluation.

Building Partnership to Improve Migrants' Access to Healthcare in Mumbai

Objectives

An intervention to improve migrants’ access to healthcare was piloted in Mumbai with purpose of informing health policy and planning. This paper aims to describe the process of building partnership for improving migrants’ access to healthcare of the pilot intervention, including the role played by different stakeholders and the contextual factors affecting the intervention.

Methods

The process evaluation was based on Baranowski and Stables’ framework. Observations in community and conversations with stakeholders as recorded in daily diaries, minutes of pre-intervention workshops, and stakeholder meetings served as data sources. Data were coded using the framework and descriptive summaries of evaluation components were prepared.

Results

Recruitment of stakeholders was easier than sustaining their interest. Community representatives led the intervention assisted by government officials. They planned community-level interventions to improve access to healthcare that involved predominantly information, education, and communication activities for which pre-existing formal and informal social networks and community events were used. Although the intervention reached migrants living with families, single male migrants neither participated nor did the intervention reach them consistently. Contextual factors such as culture differences between migrants and native population and illegality in the nature of the settlement, resulting in the exclusion from services, were the barriers.

Conclusion

Inclusive multi-stakeholder partnership, including migrants themselves and using both formal and informal networks in community is a feasible strategy for health education and has potential to improve the migrants’ access to healthcare. However, there are challenges to the partnership process and new strategies to overcome these challenges need to be tested such as peer-led models for involvement of single male migrants. For sustaining such efforts and mainstreaming migrants, addressing contextual factors and having formal mechanisms for their inclusion are equally important.

Intervention fidelity in a school-based diet and physical activity intervention in the UK: Active for Life Year 5

BACKGROUND

Active for Life Year 5 (AFLY5) is an educational programme for Year 5 children (aged 9-10) designed to increase children's physical activity, decrease sedentary behaviour and increase fruit and vegetable intake. This paper reports findings from a process evaluation embedded within a randomised controlled trial evaluating the programme's effectiveness. It considers the fidelity of implementation of AFLY5 with a focus on three research questions: 1. To what extent was the intervention delivered as planned? 2. In what ways, if any, did the teachers amend the programme? and 3. What were the reasons for any amendments?

METHODS

Mixed methods were used including data collection via observation of the intervention delivery, questionnaire, teacher's intervention delivery log and semi-structured interviews with teachers and parents. Qualitative data were analysed thematically and quantitative data were summarised using descriptive statistics.

RESULTS

Following training, 42 of the 43 intervention school teachers/teaching staff (98%) were confident they could deliver the nutrition and physical activity lessons according to plan. The mean number of lessons taught was 12.3 (s.d. 3.7), equating to 77% of the intervention. Reach was high with 95% of children in intervention schools receiving lessons. A mean of 6.2 (s.d. 2.6) out of 10 homeworks were delivered. Median lesson preparation time was 10 min (IQR 10-20) and 28% of lessons were reported as having been amended. Qualitative findings revealed that those who amended the lessons did so to differentiate for student ability, update them for use with new technologies and to enhance teacher and student engagement. Teachers endorsed the aims of the intervention, but some were frustrated with having to adapt the lesson materials. Teachers also a reported tendency to delegate the physical activity lessons to other staff not trained in the intervention.

CONCLUSIONS

Fidelity of intervention implementation was good but teachers' enthusiasm for the AFLY5 programme was mixed despite them believing that the messages behind the lessons were important. This may have meant that the intervention messages were not delivered as anticipated and explain why the intervention was found not to be effective.

TRIAL REGISTRATION

ISRCTN50133740.

Direct and Indirect Benefits Reported by Users of Transcutaneous Electrical Nerve Stimulation for Chronic Musculoskeletal Pain: Qualitative Exploration Using Patient Interviews

BACKGROUND

There is no consensus regarding the effectiveness of transcutaneous electrical nerve stimulation (TENS) for management of chronic musculoskeletal pain or chronic low back pain. A recent review of previous trial methodology identified significant problems with low treatment fidelity. There is little information available to guide selection of patient-reported outcome measures appropriate for TENS evaluation.

OBJECTIVES

The purpose of this study was to explore the experiences of patients at a secondary care pain clinic who successfully used TENS to help manage chronic musculoskeletal pain. These key informants were selected because they had the potential to generate knowledge that could inform research design and clinical practice.

DESIGN

A qualitative method using individual semistructured interviews with open questions was selected for its capacity to generate rich data.

METHODS

A mini focus group informed the development of a discussion guide for semistructured interviews with 9 patients (6 women, 3 men). Thematic analysis was used as the primary data analysis method, and this analysis was enhanced by a case-level analysis of the context and processes of TENS use of each individual.

RESULTS

Data analysis indicated that distraction from pain and a reduction in the sensations associated with muscle tension or spasm should be considered as separate outcomes from pain relief. These direct benefits led to a wide range of indirect benefits dependent on patient decision making, including medication reduction, enhanced function, psychological benefits, and enhanced ability to rest.

CONCLUSIONS

The findings indicate that evaluating TENS using a unidimensional pain scale is likely to overlook potential benefits. The complex pattern of TENS usage, as well as multiple direct and indirect outcomes, indicates that TENS could be considered as a complex intervention.