The use of theory in process evaluations conducted alongside randomized trials of implementation interventions: A systematic review

Developing a theory of change - the importance of rich process data and authors' insights into context, implementation and mechanisms

BACKGROUND

Theories of change explaining how interventions work are increasingly important, yet the methods/data to develop these are less advanced than for evaluating effects.

METHODS

We conducted a systematic evidence synthesis to develop a theory of change for structural adolescent contraception interventions. We reflect on the utility of the information provided in evaluation reports.

FINDINGS/DISCUSSION

Few of the included evaluations presented their theory of change, or included rich, qualitative process data. Authors' descriptions of context and implementation, typically in introduction and discussion sections, were very useful. These helped to understand the intervention's context, how it was experienced and why or how it had the effect that it did. We recommend incorporating rich process evaluations into studies, and reporting contextual insights into the intervention's development, implementation and experience. We also recommend including these data and insights within syntheses that aim to develop theories of change.

A systematic review of process evaluations for psychosocial interventions designed to improve the wellbeing and quality of life of community-dwelling people with dementia and their carers

BACKGROUND

Psychosocial interventions improve the wellbeing and quality of life of People Living with Dementia (PLWD) and their family carers; but due to their complexity it can be challenging to identify mechanisms of action. We reviewed process evaluations that have sought to elucidate how these interventions work, to inform their implementation.

METHOD

We systematically reviewed process evaluations of studies evaluating psychosocial interventions for PLWD in their own home and/or their family carers. We rated study quality using the Mixed Methods Appraisal Tool. We described, with reference to Medical Research Council (2015) process evaluation guidance, how implementation, mechanisms of impact and contextual factors were investigated; and describe commonalities in the mechanisms of action identified across studies.

RESULTS

Twenty four included studies evaluated the processes of 22 interventions. These studies collectively applied five frameworks; almost all frameworks' advised evaluations were theory-based and used mixed-methods analyses, but only 5/24 evaluation designs were informed by the intervention theory and 8/24 used mixed methods. 8/24 evaluations considered contextual factors in their design, though 20/24 cited contextual factors in findings. Interventions were more successful where PLWD were motivated and aware of potential benefits, and when carers could support engagement and were themselves supported by the intervention. How the intervention aligned with participants' current needs and stage of dementia were key influencing factors.

CONCLUSION

Knowing how interventions can influence change for community-dwelling people with dementia and their family carer's will improve translation of trial findings into practice. Robust, theory-driven process evaluations can enable this.

Do the Expert Recommendations for Implementing Change (ERIC) strategies adequately address sustainment?

Background

Sustainability science is an emerging area within implementation science. There is limited evidence regarding strategies to best support the continued delivery and sustained impact of evidence-based interventions (EBIs). To build such evidence, clear definitions, and ways to operationalize strategies specific and/or relevant to sustainment are required. Taxonomies and compilations such as the Expert Recommendations for Implementing Change (ERIC) were developed to describe and organize implementation strategies. This study aimed to adapt, refine, and extend the ERIC compilation to incorporate an explicit focus on sustainment. We also sought to classify the specific phase(s) of implementation when the ERIC strategies could be considered and applied.

Methods

We used a two-phase iterative approach to adapt the ERIC. This involved: (1) adapting through consensus (ERIC strategies were mapped against barriers to sustainment as identified via the literature to identify if existing implementation strategies were sufficient to address sustainment, needed wording changes, or if new strategies were required) and; (2) preliminary application of this sustainment-explicit ERIC glossary (strategies described in published sustainment interventions were coded against the glossary to identify if any further amendments were needed). All team members independently reviewed changes and provided feedback for subsequent iterations until consensus was reached. Following this, and utilizing the same consensus process, the Exploration, Preparation, Implementation and Sustainment (EPIS) Framework was applied to identify when each strategy may be best employed across phases.

Results

Surface level changes were made to the definitions of 41 of the 73 ERIC strategies to explicitly address sustainment. Four additional strategies received deeper changes in their definitions. One new strategy was identified: Communicate with stakeholders the continued impact of the evidence-based practice. Application of the EPIS identified that at least three-quarters of strategies should be considered during preparation and implementation phases as they are likely to impact sustainment.

Conclusion

A sustainment-explicit ERIC glossary is provided to help researchers and practitioners develop, test, or apply strategies to improve the sustainment of EBIs in real-world settings. Whilst most ERIC strategies only needed minor changes, their impact on sustainment needs to be tested empirically which may require significant refinement or additions in the future.

Required sample size to detect mediation in 3-level implementation studies

Background

Statistical tests of mediation are important for advancing implementation science; however, little research has examined the sample sizes needed to detect mediation in 3-level designs (e.g., organization, provider, patient) that are common in implementation research. Using a generalizable Monte Carlo simulation method, this paper examines the sample sizes required to detect mediation in 3-level designs under a range of conditions plausible for implementation studies.

Method

Statistical power was estimated for 17,496 3-level mediation designs in which the independent variable (X) resided at the highest cluster level (e.g., organization), the mediator (M) resided at the intermediate nested level (e.g., provider), and the outcome (Y) resided at the lowest nested level (e.g., patient). Designs varied by sample size per level, intraclass correlation coefficients of M and Y, effect sizes of the two paths constituting the indirect (mediation) effect (i.e., X→M and M→Y), and size of the direct effect. Power estimates were generated for all designs using two statistical models—conventional linear multilevel modeling of manifest variables (MVM) and multilevel structural equation modeling (MSEM)—for both 1- and 2-sided hypothesis tests.

Results

For 2-sided tests, statistical power to detect mediation was sufficient (≥0.8) in only 463 designs (2.6%) estimated using MVM and 228 designs (1.3%) estimated using MSEM; the minimum number of highest-level units needed to achieve adequate power was 40; the minimum total sample size was 900 observations. For 1-sided tests, 808 designs (4.6%) estimated using MVM and 369 designs (2.1%) estimated using MSEM had adequate power; the minimum number of highest-level units was 20; the minimum total sample was 600. At least one large effect size for either the X→M or M→Y path was necessary to achieve adequate power across all conditions.

Conclusions

While our analysis has important limitations, results suggest many of the 3-level mediation designs that can realistically be conducted in implementation research lack statistical power to detect mediation of highest-level independent variables unless effect sizes are large and 40 or more highest-level units are enrolled. We suggest strategies to increase statistical power for multilevel mediation designs and innovations to improve the feasibility of mediation tests in implementation research.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13012-022-01235-2.

Impacts of two behavior change interventions on determinants of medication adherence: process evaluation applying the health action process approach and habit theory alongside a randomized controlled trial

Investigating the mechanisms of behavior change interventions provides a more fulsome understanding of how and why interventions work (or don't work). We assessed mechanisms of two interventions (mailouts alone, and mailouts plus telephone support, informed by the Health Action Process Approach (HAPA) and Habit Theory), designed to increase medication adherence after myocardial infarction. We conducted a process evaluation alongside a pragmatic trial. Medication adherence was assessed via self-report at 12-months in the trial, and participants in all trial groups were invited to contemporaneously complete an additional questionnaire assessing targeted mechanisms (HAPA constructs and automaticity). We used multiple regression-based mediation models to investigate indirect effects. Of 589 respondents, 497 were analyzed (92 excluded due to missing data). Mailouts plus telephone support had statistically significant but small effects on intention, social support, action planning, coping planning, and automaticity. There were no indirect effects of interventions on medication adherence via these constructs. Therefore, while this intervention led to changes in proposed mechanisms, these changes were not great enough to lead to behavior change. Refinements (and subsequent evaluation) of the interventions are warranted, and our findings indicate that this could involve offering more intensive support to form plans and identify cues for taking medications, in addition to providing physical supports to encourage self-monitoring, feedback, and habit formation. Trial registration: ClinicalTrials.gov: NCT02382731.

Relocation to Activity-Based Workplaces (ABW)-Importance of the Implementation Process

Activity-based workplaces (ABW) have been implemented in many organizations to offer office flexibility and decrease facility costs. Evaluations of the ABW implementation process are rare. The study aimed to examine the ABW relocation process of two offices in a Swedish governmental agency and to explore factors that influence the implementation process and satisfaction with it. Qualitative or quantitative data were collected on process variables (context, recruitment, reach, dose delivered, dose received, satisfaction), barriers and facilitators to the process were explored in focus group interviews, and immediate outcomes (perceived knowledge, understanding office rules, satisfying information and support) were measured by questionnaire before and after the relocation. The evaluation showed that recruitment was unsatisfactory and reach insufficient—and participation in activities was thus low for both offices. However, intended changes improved. Unclear aims of ABW, lack of manager support and, lack of communication were some of the reported barriers to participation, while a well-planned process, work groups, and program activities were facilitators. Thus, to increase satisfaction with the relocation, our results suggest that recruitment should be thoroughly planned, taking these factors into account to increase participation. This knowledge may be useful for planning and designing successful ABW relocations and evaluations.

Attitude theory and measurement in implementation science: a secondary review of empirical studies and opportunities for advancement

BACKGROUND

Implementation science studies often express interest in "attitudes," a term borrowed from psychology. In psychology, attitude research has an established methodological and theoretical base, which we briefly summarize here. We then review implementation studies designed to measure attitudes and compare their definitions and methods with those from psychology.

METHODS

A recent review identified 46 studies empirically examining factors associated with implementation. For each of these studies, we evaluated whether authors included attitudes as a construct of interest, and if so, whether and how the construct was defined, measured, and analyzed.

RESULTS

Most of the articles (29/46 [63%]) mention attitudes as an implementation factor. Six articles include a definition of the construct. Nineteen studies were designed to measure attitudes but lacked clarity in describing how attitudes were measured. Those that explained their measurement approach used methods that differed from one another and from validated methods in social psychology. Few articles described associated analyses or provided results specific to attitudes. Despite the lack of specificity regarding relevant measurement, analysis, and results, the articles often included causal conclusions about the role of attitudes.

CONCLUSIONS

Attitudes may be an important construct to implementation scientists, but studies to date are ambiguous in their definitions of attitudes and inconsistent in the methods used to measure and analyze attitudes. We discuss how implementation studies can apply psychology's standardized definitions, validated measurement approaches, and causal models that include attitudes. This application of attitude theory and methods could offer implementation research valuable scientific opportunities.

Process evaluation of complex interventions in chronic and neglected tropical diseases in low- and middle-income countries-a scoping review protocol

BACKGROUND

The use of process evaluations is a growing area of interest in research groups working on complex interventions. This methodology tries to understand how the intervention was implemented to inform policy and practice. A recent systematic review by Liu et al. on process evaluations of complex interventions addressing non-communicable diseases found few studies in low- and middle- income countries (LMIC) because it was restricted to randomized controlled trials, primary healthcare level and non-communicable diseases. Yet, LMICs face different barriers to implement interventions in comparison to high-income countries such as limited human resources, access to health care and skills of health workers to treat chronic conditions especially at primary health care level. Therefore, understanding the challenges of interventions for non-communicable diseases and neglected tropical diseases (diseases that affect poor populations and have chronic sequelae) will be important to improve how process evaluation is designed, conducted and used in research projects in LMICs. For these reasons, in comparison to the study of Liu et al., the current study will expand the search strategy to include different study designs, languages and settings.

OBJECTIVE

Map research using process evaluation in the areas of non-communicable diseases and neglected tropical diseases to inform the gaps in the design and conduct of this type of research in LMICs.

METHODS

Scoping review of process evaluation studies of randomized controlled trials (RCTs) and non-RCTs of complex interventions implemented in LMICs including participants with non-communicable diseases or neglected tropical diseases and their health care providers (physicians, nurses, technicians and others) related to achieve better health for all through reforms in universal coverage, public policy, service delivery and leadership. The aspects that will be evaluated are as follows: (i) available evidence of process evaluation in the areas of non-communicable diseases and neglected tropical diseases such as frameworks and theories, (ii) methods applied to conduct process evaluations and (iii) gaps between the design of the intervention and its implementation that were identified through the process evaluation. Studies published from January 2008. Exclusion criteria are as follows: not peer reviewed articles, not a report based on empirical research, not reported in English or Spanish or Portuguese or French, reviews and non-human research.

DISCUSSION

This scoping review will map the evidence of process evaluations conducted in LMICs. It will also identify the methods they used to collect and interpret data, how different theories and frameworks were used and lessons from the implementation of complex interventions. This information will allow researchers to conduct better process evaluations considering special characteristics from countries with limited human resources, scarce data available and limited access to health care.

Reflexive thematic analysis exploring stakeholder experiences of virtual pulmonary rehabilitation (VIPAR)

Introduction

To enable greater availability of pulmonary rehabilitation (PR), video-conferencing technology was employed to link a community site to a standard outpatient PR service to deliver the programme closer to those with chronic respiratory disease. The service was called virtual pulmonary rehabilitation (VIPAR). To understand the experiences of those involved in this service, this study aimed to answer the question: How do the different stakeholders that interact with VIPAR experience the programme?

Methods

Focus groups were conducted with people with chronic lung disease attending either the community or outpatient PR site, in addition to the staff involved in the running of the PR groups. A total of five focus groups were conducted. Reflexive thematic analysis was used to analyse the transcripts.

Results

Participants who received VIPAR were positive regarding the programme and described the benefits of the service. Additionally, participants discussed the trade-off between convenience and relatedness. Staff described barriers to the service and solutions that could be employed to mitigate these, including greater training, organisation and understanding of risk.

Discussion

Using technology to increase the availability of PR is acceptable to the stakeholders involved, providing problems and organisational issues are resolved with training.

A comparative assessment of two tools designed to support patient safety culture in UK general practice

BACKGROUND

The NHS has recognised the importance of a high quality patient safety culture in the delivery of primary health care in the rapidly evolving environment of general practice. Two tools, PC-SafeQuest and MapSaf, were developed with the intention of assessing and improving patient safety culture in this setting. Both have been made widely available through their inclusion in the Royal College of General Practitioners' Patient Safety Toolkit and our work offerss a timely exploration of the tools to inform practice staff as to how each might be usefully applied and in which circumstances. Here we present a comparative analysis of their content, and describe the perspectives of staff on their design, outputs and the feasibility of their sustained use.

METHODS

We have used a content analysis to provide the context for the qualitative study of staff experiences of using the tools at a representative range of practices recruited from across the Midlands (UK). Data was collected through moderated focus groups using an identical topic guide.

RESULTS

A total of nine practices used the PC-SafeQuest tool and four the MapSaf tool. A total of 159 staff completed the PC-SafeQuest tool 52 of whom took part in the subsequent focus group discussions, and 25 staff completed the MapSaf tool all of whom contributed to the focus group discussions. PC-SafeQuest was perceived as quick and easy to use with direct questions pertinent to the work of GP practices providing useful quantitative insight into important areas of safety culture. Though MaPSaF was more logistically challenging, it created a forum for synchronous cross- practice discussions raising awareness of perceptions of safety culture across the practice team.

CONCLUSIONS

Both tools were able to promote reflective and reflexive practice either in individual staff members or across the broader practice team and the oversight they granted provided useful direction for senior staff looking to improve patient safety. Because PC SafeQuest can be easily disseminated and independently completed it is logistically suited to larger practice organisations, whereas the MapSaf tool lends itself to smaller practices where assembling staff in a single workshop is more readily achieved.

Are we speaking the same language? Call for action to improve theory application and reporting in behaviour change research

Inconsistencies are evident in definitions and interpretations of theory, application of theory, and reporting of theory use within the behaviour change field impeding cumulative knowledge advancement. Standardised frameworks and methods are needed to support the definition, application, and reporting of theory, and to assist researchers in understanding how theory should be applied to build cumulative knowledge over time. Progress is being made with the development of ontologies, taxonomies, methods for mapping interventions, and coding schemes; however, consolidation is needed to improve levels and quality of theory use, and to facilitate the translation of theory-driven research in practice. This paper discusses the importance of rigorous theory application and reporting in health-related behaviour change research and outlines the need for a standardised framework that supports both researchers and practitioners in designing, implementing, and evaluating theory-driven interventions in a concrete and consistent manner. To this end, several recommendations are provided to facilitate the development of a standardised framework that supports theory application and reporting in the behaviour change field. Concrete and consistent theory application and reporting will permit critical appraisal within and across studies, thereby advancing cumulative knowledge of behaviour change over time.

Designing and undertaking randomised implementation trials: guide for researchers

Implementation science is the study of methods to promote the systematic uptake of evidence based interventions into practice and policy to improve health. Despite the need for high quality evidence from implementation research, randomised trials of implementation strategies often have serious limitations. These limitations include high risks of bias, limited use of theory, a lack of standard terminology to describe implementation strategies, narrowly focused implementation outcomes, and poor reporting. This paper aims to improve the evidence base in implementation science by providing guidance on the development, conduct, and reporting of randomised trials of implementation strategies. Established randomised trial methods from seminal texts and recent developments in implementation science were consolidated by an international group of researchers, health policy makers, and practitioners. This article provides guidance on the key components of randomised trials of implementation strategies, including articulation of trial aims, trial recruitment and retention strategies, randomised design selection, use of implementation science theory and frameworks, measures, sample size calculations, ethical review, and trial reporting. It also focuses on topics requiring special consideration or adaptation for implementation trials. We propose this guide as a resource for researchers, healthcare and public health policy makers or practitioners, research funders, and journal editors with the goal of advancing rigorous conduct and reporting of randomised trials of implementation strategies.

Identifying mechanisms of change in a magic-themed hand-arm bimanual intensive therapy programme for children with unilateral spastic cerebral palsy: a qualitative study using behaviour change theory

Background

There has been much research into how to promote upper-limb skills to achieve functional independence in children with unilateral spastic cerebral palsy (USCP). One researched intervention is the Breathe Magic programme, which follows the protocol of hand-arm bimanual intensive therapy (HABIT) whilst, incorporating magic tricks to develop children’s motor skills and bimanual skills. However, whilst research has found the programme to be effective, there has been little consideration of how the intervention leads to a positive outcome: what the psychological, social and physical mechanisms of action are.

Methods

Qualitative semi-structured interviews with 21 children with USCP who participated in the Breathe Magic HABIT intervention, and focus groups with 17 parents and/or carers were undertaken. Analysis was conducted through the lens of the COM-B behaviour change model using a combined deductive framework and inductive thematic analysis. Reliability of coding was confirmed through random extraction and double coding of a portion of responses and the calculation of inter-rater reliability.

Results

Breathe Magic brings about change and positive outcomes by increasing children’s psychological and physical capabilities, providing social opportunities, and enhancing reflective and automatic motivation. Additionally, a number of enablers to engaging in the intervention were identified, particularly under psychological capabilities, social opportunities and both reflective and automatic motivation. Very few barriers were raised; those that were raised were of relatively low frequency of reporting.

Conclusions

By conducting a theory-based qualitative process evaluation, this study demonstrated the mechanisms of change behind the Breathe Magic HABIT intervention for children with USCP. Breathe Magic was found to be a well-structured combination of intended and unintended mechanisms of change. Overall, the success of Breathe Magic was observed through not only its intended mechanisms to enhance hand skills, but also through unintended psychological improvements in children’s hand function, as well as social and motivational benefits resulting from interaction between children and parents.

Action, actor, context, target, time (AACTT): a framework for specifying behaviour

BACKGROUND

Designing implementation interventions to change the behaviour of healthcare providers and other professionals in the health system requires detailed specification of the behaviour(s) targeted for change to ensure alignment between intervention components and measured outcomes. Detailed behaviour specification can help to clarify evidence-practice gaps, clarify who needs to do what differently, identify modifiable barriers and enablers, design interventions to address these and ultimately provides an indicator of what to measure to evaluate an intervention's effect on behaviour change. An existing behaviour specification framework proposes four domains (Target, Action, Context, Time; TACT), but insufficiently clarifies who is performing the behaviour (i.e. the Actor). Specifying the Actor is especially important in healthcare settings characterised by multiple behaviours performed by multiple different people. We propose and describe an extension and re-ordering of TACT to enhance its utility to implementation intervention designers, practitioners and trialists: the Action, Actor, Context, Target, Time (AACTT) framework. We aim to demonstrate its application across key steps of implementation research and to provide tools for its use in practice to clarify the behaviours of stakeholders across multiple levels of the healthcare system.

METHODS AND RESULTS

We used French et al.'s four-step implementation process model to describe the potential applications of the AACTT framework for (a) clarifying who needs to do what differently, (b) identifying barriers and enablers, (c) selecting fit-for-purpose intervention strategies and components and (d) evaluating implementation interventions.

CONCLUSIONS

Describing and detailing behaviour using the AACTT framework may help to enhance measurement of theoretical constructs, inform development of topic guides and questionnaires, enhance the design of implementation interventions and clarify outcome measurement for evaluating implementation interventions.