Fidelity in School-Based Positive Behavioral Interventions and Supports: Current Status of Compliance

OBJECTIVE

The current study identified and compared different treatment fidelity reporting methods.

METHOD

This paper includes 2 studies. In Study 1, the researchers compared and contrasted 3 sources of fidelity obtained in a study previously published by the authors; whereas, Study 2 did the same using a structured review of the literature.

RESULTS

Fidelity reporting methods included: self-reports, peer reports, observations, artifact review, and use of standardized procedures. Study 1: a statistically significant difference in fidelity results was identified between methods. Study 2: the most common method of reporting was no reporting (46%) followed by observations (25.6%), peer ratings (25.6%), and self-report (23.1%). When studies reported that fidelity was evaluated, 57% subsequently provided specific fidelity results.

CONCLUSIONS

Given that intervention fidelity is reported differently depending on the method used, then standard guidelines are needed for how this construct should be assessed and reported in practice and research.

Evaluation of the implementation fidelity of comprehensive emergency obstetric and newborn care at University of Gondar comprehensive specialized hospital, Northwest Ethiopia: a mixed-method evaluation

BACKGROUND

Approximately 15% of births worldwide result in life-threatening complications during pregnancy, delivery, or postpartum. Comprehensive Emergency Management of Obstetric and Newborn Care (CEmONC) is intended as one of the measures for maternal healthcare services to reduce the high burden with regard to childbirth complications. However, its state of implementation fidelity has not been well investigated. Therefore, this study aimed to evaluate the implementation fidelity of CEmONC services at University of Gondar Comprehensive Specialized Hospital, Ethiopia.

METHOD

A case-study design with an embedded mixed method was employed. Adherence, quality of delivery, and participant responsiveness dimensions from Carroll's conceptual framework were used in this evaluation. Four hundred four exit interviews, 423 retrospective document reviews and 10 key informants were conducted. Moreover, a binary logistic regression model was fitted. The qualitative data were transcribed, translated, coded, and analysed using a thematic analysis approach. The overall implementation fidelity of the CEmONC was judged based on the pre-seated judgmental criteria.

RESULTS

Overall the implementation fidelity of the CEmONC service was 75.5%. Quality of delivery, participant responsiveness and adherence were 72.7%, 76.6% and 77.2% respectively. Signal functions like parenteral antibiotics and removal of retained products were insufficiently performed against the recommended protocols which was also evidenced by the key informant interviews. Healthcare providers' respect for the clients was less. Age ≥ 35 years (AOR = 0.48, 95% CI: 0.24,0.98), educational status of college and above (AOR = 2.61, 95% CI: 1.46,4.66), being government employed (AOR = 1.85, 95% CI: 1.08,3.18), having ANC follow-up (AOR = 5.50, 95% CI: 1.83, 16.47) and grand multigravida (AOR = 2.17, 95% CI: 1.08, 4.38) were factors significantly associated with participant responsiveness towards the services.

CONCLUSIONS

The overall implementation fidelity of the CEmONC services was implemented in good fidelity. Moreover, the quality of delivery was judged as implemented in fair fidelity. Parenteral antibiotics and removal of retained products were not found to be sufficiently performed. Respect for the clients was insufficiently delivered. Therefore, it is recommended that parenteral antibiotics drugs be adequately provided and training for healthcare providers regarding compassionate and respectful care shall be facilitated. Moreover, healthcare providers are strongly recommended to adhere to the recommended guidelines.

Fidelity to the ACT SMART Toolkit: an instrumental case study of implementation strategy fidelity

BACKGROUND

Evidence-based practices (EBPs) are shown to improve a variety of outcomes for autistic children. However, EBPs often are mis-implemented or not implemented in community-based settings where many autistic children receive usual care services. A blended implementation process and capacity-building implementation strategy, developed to facilitate the adoption and implementation of EBPs for autism spectrum disorder (ASD) in community-based settings, is the Autism Community Toolkit: Systems to Measure and Adopt Research-based Treatments (ACT SMART Toolkit). Based on an adapted Exploration, Adoption decision, Preparation, Implementation, Sustainment (EPIS) Framework, the multi-phased ACT SMART Toolkit is comprised of (a) implementation facilitation, (b) agency-based implementation teams, and (c) a web-based interface. In this instrumental case study, we developed and utilized a method to evaluate fidelity to the ACT SMART Toolkit. This study responds to the need for implementation strategy fidelity evaluation methods and may provide evidence supporting the use of the ACT SMART Toolkit.

METHODS

We used an instrumental case study approach to assess fidelity to the ACT SMART Toolkit during its pilot study with six ASD community agencies located in southern California. We assessed adherence, dose, and implementation team responsiveness for each phase and activity of the toolkit at both an aggregate and individual agency level.

RESULTS

Overall, we found that adherence, dose, and implementation team responsiveness to the ACT SMART Toolkit were high, with some variability by EPIS phase and specific activity as well as by ASD community agency. At the aggregate level, adherence and dose were rated notably lowest during the preparation phase of the toolkit, which is a more activity-intensive phase of the toolkit.

CONCLUSIONS

This evaluation of fidelity to the ACT SMART Toolkit, utilizing an instrumental case study design, demonstrated the potential for the strategy to be used with fidelity in ASD community-based agencies. Findings related to the variability of implementation strategy fidelity in the present study may also inform future adaptations to the toolkit and point to broader trends of how implementation strategy fidelity may vary by content and context.

A codesigned fit-for-purpose implementation framework for aged care

RATIONALE, AIMS AND OBJECTIVES

The field of implementation science is critical for embedding research evidence into healthcare practice, benefiting individuals, organizations, governments, and the broader community. Implementation science is messy and complex, underpinned by many theories and frameworks. Efficacious interventions for older people with multiple comorbidities exist, yet many lack effectiveness evaluation relevant to pragmatic implementation within aged care practice. This article outlines the conceptualization and development of an Implementation Framework for Aged Care (IFAC), fit-for-purpose for an aged care organization, Bolton Clarke, intent on embedding evidence into practice.

METHOD

A four-stage process was adopted to (1) explore context and relevant literature to conceptualize the IFAC; (2) identify key elements for a draft IFAC; (3) expand elements and refine the draft in consultation with experts and (4) apply the IFAC to three existing projects, identifying key learnings. A checklist to operationalize the IFAC was then developed.

RESULTS

The IFAC is grounded in codesign principles and encapsulated by the implementation context, from a social, cultural and political perspective. The IFAC addresses the questions of (1) why do we need to change?; (2) what do we know?; (3) who will benefit?; (4) who will make the change?; (5) what strategies will be used?; and (6) what difference are we making? Three pilot projects: early adoption of a Wellness and Reablement approach; a care worker and virtual physiotherapist-led program to prevent falls; and a therapeutic horticulture program for residential communities, highlight learnings of applying the IFAC in practice.

CONCLUSION

This fit-for-purpose IFAC was developed for a proactive and responsive aged care provider. The simplicity of the six-question IFAC is underpinned by substantial theoretical perspectives for its elements and their connections. This complexity is then consolidated into an 18-question checklist to operationalize the IFAC, necessary to advance the translation of evidence into clinical practice.

Going From Zero to 100 in Remote Dementia Research: A Practical Guide

Remote approaches for dementia research are required in the era of COVID-19, but moving a research program from in person to remote involves additional considerations. We recommend using outcome measures that have psychometric properties for remote delivery, and we recommend against adapting in-person scales for remote delivery without evidence for psychometric equivalency. We suggest remote research designs that maximize benefit for participants, which could have implications for control groups. Researchers should plan for flexibility in their methods for remote research and must not assume all participants will be able to videoconference; telephone-only research is possible. We recommend performing an assessment of information communication technology infrastructure and prior exposure to this technology with each participant before making a final choice on remote methods for research. In general, researchers should adapt their methods for remote research to each participant rather than requesting participants to adapt to the researchers. Screening for sensory loss should be conducted, and the impact of this on the use of technology for remote research should be considered. In this viewpoint, we detail how individualized training is required prior to engaging in remote research, how training plans interact with cognitive impairments and, finally, the steps involved in facilitating technology-based remote data collection.

The Effect of Pelvic Floor Muscle Interventions on Pelvic Floor Dysfunction After Gynecological Cancer Treatment: A Systematic Review

OBJECTIVE

The aim of this systematic review was to identify, evaluate, and synthesize the evidence from studies that have investigated the effect of nonsurgical, nonpharmacological, pelvic floor muscle interventions on any type of pelvic floor dysfunction or health-related quality of life in patients after any type of treatment for gynecological cancer.

METHODS

Six electronic databases (Cochrane Library 2018, CINAHL 1982-2018, MEDLINE 1950-2018, EMBASE 1980-2018, PsycINFO 1806-2018, and EMCARE 1995-2018) were systematically searched in June 2018. Reference lists of identified articles were hand searched. Randomized controlled trials (RCTs), cohort studies, and case series were included if they investigated the effects of conservative treatments, including pelvic floor muscle training or dilator training, on bladder, bowel, or sexual function in patients who had received treatment for gynecological cancer. Risk of bias was assessed using the Physiotherapy Evidence Database scale for RCTs and the Newcastle-Ottawa scale for cohort studies.

RESULTS

Five RCTs and 2 retrospective cohort studies were included (n = 886). The results provided moderate-level evidence that pelvic floor muscle training with counseling and yoga or core exercises were beneficial for sexual function (standardized mean difference = -0.96, 95% CI = -1.22 to -0.70, I2 = 0%) and health-related quality of life (standardized mean difference = 0.63, 95% CI = 0.38 to 0.88, I2 = 0%) in survivors of cervical cancer and very low-level evidence that dilator therapy reduced vaginal complications in survivors of cervical and uterine cancer (odds ratio = 0.37, 95% CI = 0.17 to 0.80, I2 = 54%). There were insufficient data for meta-analysis of bladder or bowel function.

CONCLUSION

Conservative pelvic floor muscle interventions may be beneficial for improving sexual function and health-related quality of life in survivors of gynecological cancer. Given the levels of evidence reported in this review, further high-quality studies are needed, especially to investigate effects on bladder and bowel function.

IMPACT

This review provides moderate-level evidence for the role of pelvic floor rehabilitation to improve health outcomes in the gynecological cancer survivorship journey. Clinicians and health service providers should consider how to provide cancer survivors the opportunity to participate in supervised pelvic floor rehabilitation programs.

Resilience-based alcohol education: developing an intervention, evaluating feasibility and barriers to implementation using mixed-methods

Alcohol education must ensure that young people have appropriate information, motivation and skills. This article describes the fifth phase in a program of intervention development based on principles of social marketing and intervention mapping. The aim was to enhance drink refusal self-efficacy (DRSE) and help develop skills for non-drinking or moderate drinking. We conducted a mixed-methods feasibility trial that measured intervention effects among 277 UK secondary school students aged 14-16, and used qualitative methods to explore four teachers' experiences of delivering the intervention. The intervention did not produce the desired changes in DRSE or alcohol use, but nor did it increase alcohol use. In the qualitative process evaluation, time constraints, pressure to prioritize other topics, awkwardness and embarrassment were identified as barriers to fidelitous delivery. A more intense and/or more prolonged intervention delivered with greater fidelity may have produced the desired changes in DRSE and alcohol use. This study illustrates how principles of social marketing and intervention mapping can aid development of resilience-based education designed to help students develop skills to drink moderately, or not drink. It also highlights the need to consider the constraints of micro-social (school) and macro-social (societal) cultures when designing alcohol education.

Strategies for balancing internal and external validity in evaluations of interventions

BACKGROUND

Evaluations of interventions should be carefully designed and conducted to maintain a balance between internal and external validity, with the dual goal of minimising the influence of potential confounders and improving the generalisability or applicability of any findings to practice.

AIM

To review strategies to promote balance between internal and external validity in an evaluation of a cognitive-behavioural intervention for chronic insomnia.

DISCUSSION

A pragmatic approach is needed to balance internal and external validity, and generate evidence relevant to practice. The authors present strategies to promote such a balance, including using strict eligibility criteria, subgroup analysis, random assignment of patients based on preferences, a no-treatment control condition, and standardised and consistent implementation of the intervention.

CONCLUSION

A balance between internal and external validity is essential to promote enrolment in the study and confidence in attributing its outcomes to an intervention, as well as to provide answers to clinically relevant questions such as who benefits most from which intervention.

IMPLICATIONS FOR PRACTICE

The authors recommend researchers conduct a pilot study in advance of an evaluation, to help decide which strategies to use and how to balance internal and external validity.

Developing a Survey Tool to Assess Implementation of Evidence-Based Chronic Disease Prevention in Public Health Settings Across Four Countries

Background: Understanding the contextual factors that influence the dissemination and implementation of evidence-based chronic disease prevention (EBCDP) interventions in public health settings across countries could inform strategies to support the dissemination and implementation of EBCDP interventions globally and more effectively prevent chronic diseases. A survey tool to use across diverse countries is lacking. This study describes the development and reliability testing of a survey tool to assess the stage of dissemination, multi-level contextual factors, and individual and agency characteristics that influence the dissemination and implementation of EBCDP interventions in Australia, Brazil, China, and the United States. Methods: Development of the 26-question survey included, a narrative literature review of extant measures in EBCDP; qualitative interviews with 50 chronic disease prevention practitioners in Australia, Brazil, China, and the United States; review by an expert panel of researchers in EBCDP; and test-retest reliability assessment. Results: A convenience sample of practitioners working in chronic disease prevention in each country completed the survey twice (N = 165). Overall, this tool produced good to moderately reliable responses. Generally, reliability of responses was higher among practitioners from Australia and the United States than China and Brazil. Conclusions: Reliability findings inform the adaptation and further development of this tool. Revisions to four questions are recommended before use in China and revisions to two questions before use in Brazil. This survey tool can contribute toward an improved understanding of the contextual factors that public health practitioners in Australia, Brazil, China, and the United States face in their daily chronic disease prevention work related to the dissemination and implementation of EBCDP interventions. This understanding is necessary for the creation of multi-level strategies and policies that promote evidence-based decision-making and effective prevention of chronic diseases on a more global scale.

Comparison of registered and published intervention fidelity assessment in cluster randomised trials of public health interventions in low- and middle-income countries: systematic review

BACKGROUND

Cluster randomised trials (CRTs) are a key instrument to evaluate public health interventions. Fidelity assessment examines study processes to gauge whether an intervention was delivered as initially planned. Evaluation of implementation fidelity (IF) is required to establish whether the measured effects of a trial are due to the intervention itself and may be particularly important for CRTs of complex interventions conducted in low- and middle-income countries (LMICs). However, current CRT reporting guidelines offer no guidance on IF assessment. The objective of this review was to study current practices concerning the assessment of IF in CRTs of public health interventions in LMICs.

METHODS

CRTs of public health interventions in LMICs that planned or reported IF assessment in either the trial protocol or the main trial report were included. The MEDLINE/PubMed, CINAHL and EMBASE databases were queried from January 2012 to May 2016. To ensure availability of a study protocol, CRTs reporting a registration number in the abstract were included. Relevant data were extracted from each study protocol and trial report by two researchers using a predefined screening sheet. Risk of bias for individual studies was assessed.

RESULTS

We identified 90 CRTs of public health interventions in LMICs with a study protocol in a publicly available trial registry published from January 2012 to May 2016. Among these 90 studies, 25 (28%) did not plan or report assessing IF; the remaining 65 studies (72%) addressed at least one IF dimension. IF assessment was planned in 40% (36/90) of trial protocols and reported in 71.1% (64/90) of trial reports. The proportion of overall agreement between the trial protocol and trial report concerning occurrence of IF assessment was 66.7% (60/90). Most studies had low to moderate risk of bias.

CONCLUSIONS

IF assessment is not currently a systematic practice in CRTs of public health interventions carried out in LMICs. In the absence of IF assessment, it may be difficult to determine if CRT results are due to the intervention design, to its implementation, or to unknown or external factors that may influence results. CRT reporting guidelines should promote IF assessment.

TRIAL REGISTRATION

Protocol published and available at: https://doi.org/10.1186/s13643-016-0351-0.

An updated protocol for a systematic review of implementation-related measures

BACKGROUND

Implementation science is the study of strategies used to integrate evidence-based practices into real-world settings (Eccles and Mittman, Implement Sci. 1(1):1, 2006). Central to the identification of replicable, feasible, and effective implementation strategies is the ability to assess the impact of contextual constructs and intervention characteristics that may influence implementation, but several measurement issues make this work quite difficult. For instance, it is unclear which constructs have no measures and which measures have any evidence of psychometric properties like reliability and validity. As part of a larger set of studies to advance implementation science measurement (Lewis et al., Implement Sci. 10:102, 2015), we will complete systematic reviews of measures that map onto the Consolidated Framework for Implementation Research (Damschroder et al., Implement Sci. 4:50, 2009) and the Implementation Outcomes Framework (Proctor et al., Adm Policy Ment Health. 38(2):65-76, 2011), the protocol for which is described in this manuscript.

METHODS

Our primary databases will be PubMed and Embase. Our search strings will be comprised of five levels: (1) the outcome or construct term; (2) terms for measure; (3) terms for evidence-based practice; (4) terms for implementation; and (5) terms for mental health. Two trained research specialists will independently review all titles and abstracts followed by full-text review for inclusion. The research specialists will then conduct measure-forward searches using the "cited by" function to identify all published empirical studies using each measure. The measure and associated publications will be compiled in a packet for data extraction. Data relevant to our Psychometric and Pragmatic Evidence Rating Scale (PAPERS) will be independently extracted and then rated using a worst score counts methodology reflecting "poor" to "excellent" evidence.

DISCUSSION

We will build a centralized, accessible, searchable repository through which researchers, practitioners, and other stakeholders can identify psychometrically and pragmatically strong measures of implementation contexts, processes, and outcomes. By facilitating the employment of psychometrically and pragmatically strong measures identified through this systematic review, the repository would enhance the cumulativeness, reproducibility, and applicability of research findings in the rapidly growing field of implementation science.

Comparison of registered and published intervention fidelity assessment in cluster randomised trials of public health interventions in low- and middle-income countries: systematic review protocol

BACKGROUND

Cluster randomised trials (CRTs) are a key instrument to evaluate public health interventions, particularly in low- and middle-income countries (LMICs). Fidelity assessment examines study processes to gauge whether an intervention was delivered as initially planned. Evaluation of implementation fidelity (IF) is required to establish whether the measured effects of a trial are due to the intervention itself and may be particularly important for CRTs of complex interventions. Current CRT reporting guidelines offer no guidance on IF assessment. We will systematically review the scientific literature to study current practices concerning the assessment of IF in CRTs of public health interventions in LMICs.

METHODS

We will include CRTs of public health interventions in LMICs that planned or assessed IF in either the trial protocol or the main trial report (or an associated document). Search strategies use Medical Subject Headings (MESH) and text words related to CRTs, developing countries, and public health interventions. The electronic database search was developed first for MEDLINE and adapted for the following databases: EMBASE, CINAHL, PubMed, and EMB Reviews, to identify CRT reports in English, Spanish, or French published on or after January 1, 2012. To ensure availability of a study protocol, we will include CRTs reporting a registration number in the abstract. For each included study, we will compare planned versus reported assessment of IF, and consider the dimensions of IF studied, and data collection methods used to evaluate each dimension. Data will be synthesised using quantitative and narrative techniques. Risk of bias for individual studies will be assessed using the Cochrane Collaboration Risk of Bias Tool criteria and additional criteria related to CRT methods. We will investigate possible sources of heterogeneity by performing subgroup analysis. This review was not eligible for inclusion in the PROSPERO registry.

DISCUSSION

Fidelity assessment may be a key tool for making studies more reliable, internally valid, and externally generalizable. This review will provide a portrait of current practices related to the assessment of intervention fidelity in CRTs and offer suggestions for improvement. Results will be relevant to researchers, those who finance health interventions, and for decision-makers who seek the best evidence on public health interventions.