The practice of 'doing' evaluation: lessons learned from nine complex intervention trials in action

A mixed-method approach to generate and deliver rapid-cycle evaluation feedback: lessons learned from a multicenter implementation trial in pediatric surgery

BACKGROUND

Rapid-cycle feedback loops provide timely information and actionable feedback to healthcare organizations to accelerate implementation of interventions. We aimed to (1) describe a mixed-method approach for generating and delivering rapid-cycle feedback and (2) explore key lessons learned while implementing an enhanced recovery protocol (ERP) across 18 pediatric surgery centers.

METHODS

All centers are members of the Pediatric Surgery Research Collaborative (PedSRC, www.pedsrc.org ), participating in the ENhanced Recovery In CHildren Undergoing Surgery (ENRICH-US) trial. To assess implementation efforts, we conducted a mixed-method sequential explanatory study, administering surveys and follow-up interviews with each center's implementation team 6 and 12 months following implementation. Along with detailed notetaking and iterative discussion within our team, we used these data to generate and deliver a center-specific implementation report card to each center. Report cards used a traffic light approach to quickly visualize implementation status (green = excellent; yellow = needs improvement; red = needs significant improvement) and summarized strengths and opportunities at each timepoint.

RESULTS

We identified several benefits, challenges, and practical considerations for assessing implementation and using rapid-cycle feedback among pediatric surgery centers. Regarding potential benefits, this approach enabled us to quickly understand variation in implementation and corresponding needs across centers. It allowed us to efficiently provide actionable feedback to centers about implementation. Engaging consistently with center-specific implementation teams also helped facilitate partnerships between centers and the research team. Regarding potential challenges, research teams must still allocate substantial resources to provide feedback rapidly. Additionally, discussions and consensus are needed across team members about the content of center-specific feedback. Practical considerations include carefully balancing timeliness and comprehensiveness when delivering rapid-cycle feedback. In pediatric surgery, moreover, it is essential to actively engage all key stakeholders (including physicians, nurses, patients, caregivers, etc.) and adopt an iterative, reflexive approach in providing feedback.

CONCLUSION

From a methodological perspective, we identified three key lessons: (1) using a rapid, mixed method evaluation approach is feasible in pediatric surgery and (2) can be beneficial, particularly in quickly understanding variation in implementation across centers; however, (3) there is a need to address several methodological challenges and considerations, particularly in balancing the timeliness and comprehensiveness of feedback.

TRIAL REGISTRATION

NIH National Library of Medicine Clinical Trials.

CLINICALTRIALS

gov Identifier: NCT04060303. Registered August 7, 2019, https://clinicaltrials.gov/ct2/show/NCT04060303.

Developing and testing a clinical care bundle incorporating caffeine citrate to manage apnoea of prematurity in a resource-constrained setting: a mixed methods clinical feasibility study protocol

BACKGROUND

Apnoea of prematurity (AOP) is a common condition among preterm infants. Methylxanthines, such as caffeine and aminophylline/theophylline, can help prevent and treat AOP. Due to its physiological benefits and fewer side effects, caffeine citrate is recommended for the prevention and treatment of AOP. However, caffeine citrate is not available in most resource-constrained settings (RCS) due to its high cost. Challenges in RCS using caffeine citrate to prevent AOP include identifying eligible preterm infants where gestational age is not always known and the capability for continuous monitoring of vital signs to readily identify apnoea. We aim to develop an evidence-based care bundle that includes caffeine citrate to prevent and manage AOP in tertiary healthcare facilities in Kenya.

METHODS

This protocol details a prospective mixed-methods clinical feasibility study on using caffeine citrate to manage apnoea of prematurity in a single facility tertiary-care newborn unit (NBU) in Nairobi, Kenya. This study will include a 4-month formative research phase followed by the development of an AOP clinical-care-bundle prototype over 2 months. In the subsequent 4 months, implementation and improvement of the clinical-care-bundle prototype will be undertaken. The baseline data will provide contextualised insights on care practices within the NBU that will inform the development of a context-sensitive AOP clinical-care-bundle prototype. The clinical care bundle will be tested and refined further during an implementation phase of the quality improvement initiative using a PDSA framework underpinned by quantitative and qualitative clinical audits and stakeholders' engagement. The quantitative component will include all neonates born at gestation age < 34 weeks and any neonate prescribed aminophylline or caffeine citrate admitted to the NBU during the study period.

DISCUSSION

There is a need to develop evidence-based and context-sensitive clinical practice guidelines to standardise and improve the management of AOP in RCS. Concerns requiring resolution in implementing such guidelines include diagnosis of apnoea, optimal timing, dosing and administration of caffeine citrate, standardisation of monitoring devices and alarm limits, and discharge protocols. We aim to provide a feasible standardised clinical care bundle for managing AOP in low and middle-income settings.

What do we want to get out of this? a critical interpretive synthesis of the value of process evaluations, with a practical planning framework

BACKGROUND

Process evaluations aim to understand how complex interventions bring about outcomes by examining intervention mechanisms, implementation, and context. While much attention has been paid to the methodology of process evaluations in health research, the value of process evaluations has received less critical attention. We aimed to unpack how value is conceptualised in process evaluations by identifying and critically analysing 1) how process evaluations may create value and 2) what kind of value they may create.

METHODS

We systematically searched for and identified published literature on process evaluation, including guidance, opinion pieces, primary research, reviews, and discussion of methodological and practical issues. We conducted a critical interpretive synthesis and developed a practical planning framework.

RESULTS

We identified and included 147 literature items. From these we determined three ways in which process evaluations may create value or negative consequences: 1) through the socio-technical processes of 'doing' the process evaluation, 2) through the features/qualities of process evaluation knowledge, and 3) through using process evaluation knowledge. We identified 15 value themes. We also found that value varies according to the characteristics of individual process evaluations, and is subjective and context dependent.

CONCLUSION

The concept of value in process evaluations is complex and multi-faceted. Stakeholders in different contexts may have very different expectations of process evaluations and the value that can and should be obtained from them. We propose a planning framework to support an open and transparent process to plan and create value from process evaluations and negotiate trade-offs. This will support the development of joint solutions and, ultimately, generate more value from process evaluations to all.

Antenatal couples' counselling in Uganda (ACCU): study protocol for a randomised controlled feasibility trial

BACKGROUND

Common avoidable factors leading to maternal, perinatal and neonatal deaths include lack of birth planning (and delivery in an inappropriate place) and unmet need for contraception. Progress has been slow because routine antenatal care has focused only on women. Yet, in Uganda, many women first want the approval of their husbands. The World Health Organization recommends postpartum family planning (PPFP) as a critical component of health care. The aim of this trial is to test the feasibility of recruiting and retaining participants in a trial of a complex community-based intervention to provide counselling to antenatal couples in Uganda.

METHODS

This is a two-group, non-blinded cluster-randomised controlled feasibility trial of a complex intervention. Primary health centres in Uganda will be randomised to receive the intervention or usual care provided by the Ministry of Health. The intervention consists of training village health teams to provide basic counselling to couples at home, encouraging men to accompany their wives to an antenatal clinic, and secondly of training health workers to provide information and counselling to couples at antenatal clinics, to facilitate shared decision-making on the most appropriate place of delivery, and postpartum contraception. We aim to recruit 2 health centres in each arm, each with 10 village health teams, each of whom will aim to recruit 35 pregnant women (a total of 700 women per arm). The village health teams will follow up and collect data on pregnant women in the community up to 12 months after delivery and will directly enter the data using the COSMOS software on a smartphone.

DISCUSSION

This intervention addresses two key avoidable factors in maternal, perinatal and neonatal deaths (lack of family planning and inappropriate place of delivery). Determining the acceptability and feasibility of antenatal couples' counselling in this study will inform the design of a fully randomised controlled clinical trial. If this trial demonstrates the feasibility of recruitment and delivery, we will seek funding to conduct a fully powered trial of the complex intervention for improving uptake of birth planning and postpartum family planning in Uganda.

TRIAL REGISTRATION

Pan African Clinical Trials Registry PACTR202102794681952 . Approved on 10 February 2021. ISRCTN Registry ISRCTN97229911. Registered on 23 September 2021.

Conducting a prospective evaluation of the development of a complex psycho-oncological care programme (isPO) in Germany

BACKGROUND

Evaluating the development phase of a complex intervention programme can be challenging. A prospective evaluation approach is presented based on the example of the new complex psycho-oncological care programme isPO (integrated, cross-sectoral Psycho-Oncology). Prior to programme implementation, we examined (1) if isPO was developed as intended, and (2) if it was relevant and transferable into the newly developed psycho-oncological care networks in North-Rhine Westphalia, Germany. Further, we investigated which implementation facilitators and barriers were anticipated and which implementation strategies were planned by the programme designers (multidisciplinary professionals and cancer supporting organizations who developed the isPO programme components and the networks).

METHODS

A mixed-methods approach was applied. Qualitative data were collected by quarterly progress reports, interviews and a focus group with the programme designers. Evaluation criteria for document analyses of the quarterly progress reports were developed and applied. Content analysis was applied for analysing interviews and focus group. Quantitative data were gained from evaluating the programme training for the isPO service providers by short written questionnaires that were analysed descriptively.

RESULTS

An implementable prototype of the isPO programme has been developed within 15 months, however no piloting was conducted. The programme's complexity proved to be challenging with regard to coordination and communication of the numerous programme designers. This was intensified by existing interdependencies between the designers. Further, there was little communication and participation between the programme designers and the prospective users (patients and service providers). Due to these challenges, only context-unspecific implementation strategies were planned.

CONCLUSION

The required resources for developing a new complex care programme and the need of a mature implementation strategy should be sufficiently addressed. Programmes may benefit from prospective evaluation by gaining insightful knowledge concerning the programme's maturity and anticipating implementation facilitators and barriers. A mixed-methods evaluation design was crucial for achieving profound insight into the development process.

TRIAL REGISTRATION

The study has been registered in the German Clinical Trials Register (No. DRKS00015326 ) on 30.10.2018.

Coverage of intermittent preventive treatment of malaria in pregnancy in four sub-Saharan countries: findings from household surveys

Background

Intermittent preventive treatment in pregnancy (IPTp) with sulphadoxine-pyrimethamine (SP) is a key malaria prevention strategy in areas with moderate to high transmission. As part of the TIPTOP (Transforming IPT for Optimal Pregnancy) project, baseline information about IPTp coverage was collected in eight districts from four sub-Saharan countries: Democratic Republic of Congo (DRC), Madagascar, Mozambique and Nigeria.

Methods

Cross-sectional household surveys were conducted using a multistage cluster sampling design to estimate the coverage of IPTp and antenatal care attendance. Eligible participants were women of reproductive age who had ended a pregnancy in the 12 months preceding the interview and who had resided in the selected household during at least the past 4 months of pregnancy. Coverage was calculated using percentages and 95% confidence intervals.

Results

A total of 3911 women were interviewed from March to October 2018. Coverage of at least three doses of IPTp (IPTp3+) was 22% and 24% in DRC project districts; 23% and 12% in Madagascar districts; 11% and 16% in Nigeria local government areas; and 63% and 34% in Mozambique districts. In DRC, Madagascar and Nigeria, more than two-thirds of women attending at least four antenatal care visits during pregnancy received less than three doses of IPTp.

Conclusions

The IPTp3+ uptake in the survey districts was far from the universal coverage. However, one of the study districts in Mozambique showed a much higher coverage of IPTp3+ than the other areas, which was also higher than the 2018 average national coverage of 41%. The reasons for the high IPTp3+ coverage in this Mozambican district are unclear and require further study.

When all else fails: The (mis)use of qualitative research in the evaluation of complex interventions

RATIONAL, AIMS, AND OBJECTIVES

Qualitative research has been promoted as an important component of the evaluation of complex interventions to support the scale up and spread of health service interventions, but is currently not being maximized in practice. We aim to identify and explore the sociocultural and structural factors that impact the uses (and misuses) of qualitative research in the evaluation of complex health services interventions.

METHODS

We conducted a qualitative analysis of data collected in a multiple case study of the evaluation and scale up and spread of three health service intervention.

RESULTS

Our findings demonstrate the challenges of meaningfully integrating qualitative research in evaluation programmes lead by clinicians with limited qualitative expertise and operating within an environment dominated by biomedical research, even with methodological support.

CONCLUSIONS

Based on these findings we encourage ongoing engagement of qualitative researchers in evaluation programmes to begin to refine our methodological understanding, while also suggesting changes to medical education and evaluation funding models to create fertile environments for interdisciplinary collaborations.

Process Evaluation of a Community-Based Microbial Larviciding Intervention for Malaria Control in Rural Tanzania

Microbial larviciding can be an effective component of integrated vector management malaria control schemes, although it is not commonly implemented. Moreover, quality control and evaluation of intervention activities are essential to evaluate the potential of community-based larviciding interventions. We conducted a process evaluation of a larval source management intervention in rural Tanzania where local staff were employed to apply microbial larvicide to mosquito breeding habitats with the aim of long-term reductions in malaria transmission. We developed a logic model to guide the process evaluation and then established quantitative indicators to measure intervention success. Quantitative analysis of intervention reach, exposure, and fidelity was performed to assess larvicide application, and interviews with larviciding staff were reviewed to provide context to quantitative results. Results indicate that the intervention was successful in terms of reach, as staff applied microbial larvicide at 80% of identified mosquito breeding habitats. However, the dosage of larvicide applied was sufficient to ensure larval elimination at only 26% of sites, which does not meet the standard set for intervention fidelity. We propose that insufficient training and protocol adaptation, environment and resource issues, and human error contributed to low larvicide application rates. This demonstrates how several small, context-specific details in sum can result in meaningful differences between intervention blueprint and execution. These findings may serve the design of other larval source management interventions by demonstrating the value of additional training, supervision, and measurement and evaluation of protocol adherence.

Context and the evidence-based paradigm: The potential for participatory research and systems thinking in oral health

The implementation of research evidence to promote oral health is critical, given the intransigent and emerging challenges for policymakers at a population level. Despite this, little attention has been paid to implementation research within the evidence-based paradigm. This is important as getting research evidence into clinical practice is not a linear path that consists of simple sequential steps. In this article, we argue that we need to consider a broader range of conceptual and methodological approaches to increase the value of information generated. This should be undertaken either in parallel with empirical and experimental designs, or in some cases, instead of. This is important if we are going to understand the complexity and contextual knowledge of the 'system', within which interventions are implemented. Involving key stakeholders alongside empirical and experimental designs is one helpful approach. Examples of these approaches include Patient and Public Involvement and the development of Core Outcome Sets, where the views of those that will be potentially affected by the research, are included. The use of theoretical frameworks and process evaluations alongside trials are also important, if they are fully integrated into the approach taken to address the research question. A more radical approach is using participatory designs and 'systems thinking'. Participatory approaches include subject matter 'experts by experience'. These include patients, their families, carers, healthcare professionals, services managers, policymakers, commissioners and researchers. Participatory approaches raise important questions about who facilitates the process, when it should happen and how the diverse actors become meaningfully engaged so that their involvement is active, democratic and ongoing. We argue that the issues of control, power and language are central to this and represent a paradigmatic shift to conventional approaches. Systems thinking captures the idea that public health problems commonly involve multiple interdependent and interconnected factors, which interact with each other dynamically. This approach challenges the simplicity of the hierarchy of evidence and linear sequential logic, when it does not account for context. In contrast, systems thinking accepts complexity de novo and emphasizes the need to understand the whole system rather than its individual component parts. We conclude with the idea that participatory and systems thinking help to unpack the diverse agents that are often involved in the generation and translation of evidence into clinical dental practice. It moves our conception of research away from a simple exchange between 'knowledge producers' and 'knowledge users' and raises both methodological and epistemological challenges.

Our Healthy Clarence: A Community-Driven Wellbeing Initiative

In 2015–2016, the Clarence Valley in Northern New South Wales, Australia, experienced an unexpectedly high number of deaths by suicide, and the resulting distress was exacerbated by unhelpful press coverage. The local response was to adopt a community-wide positive mental health and wellbeing initiative. This paper describes the process and achievements of the initiative called ‘Our Healthy Clarence’. Key stakeholders were interviewed at year two and relevant documents reviewed. Data were analysed using document and thematic analysis. Our Healthy Clarence was established following community consultation, including forums, interviews, surveys and workshops. It adopted a strengths-based approach to suicide prevention, encompassing positive health promotion, primary and secondary prevention activities, advocacy, and cross-sectoral collaboration. A stakeholder group formed to develop and enact a community mental health and wellbeing plan. Factors contributing to its successful implementation included a collective commitment to mental health and wellbeing, clarity of purpose, leadership support from key local partners, a paid independent coordinator, and inclusive and transparent governance. Stakeholders reported increased community agency, collaboration, optimism and willingness to discuss mental health, suicide and help-seeking. Our Healthy Clarence draws ideas from mental health care, community development and public health. This initiative could serve as a model for other communities to address suicide, self-harm and improve wellbeing on a whole-of-community scale.

Evaluating Complex Health Interventions With Randomized Controlled Trials: How Do We Improve the Use of Qualitative Methods?

Qualitative methods are underutilized in health intervention evaluation, and overshadowed by the importance placed on randomized controlled trials (RCTs). This Commentary describes how innovative qualitative methods are being used as part of RCTs, drawing on articles included in a special issue of Qualitative Health Research on this topic. The articles' insights and a review of innovative qualitative methods described in trial protocols highlights a lack of attention to structural inequalities as a causal mechanism for understanding human behavior. We situate this gap within some well-known constraints of RCT methodologies, and a discussion of alternative RCT approaches that hold promise for bringing qualitative methods center stage in intervention evaluation, including adaptive designs, pragmatic trials, and realist RCTs. To address the power hierarchies of health evaluation research, however, we argue that a fundamental shift needs to take place away from a focus on RCTs and toward studies of health interventions.

Intervention fidelity in postintensive care follow-up consultations at ten sites in the RAPIT-trial: A mixed-methods evaluation

AIM

The aim of the study was to evaluate intervention fidelity of nurses' delivery of the RAPIT recovery program for postintensive care patients.

BACKGROUND

Interventions addressing patient problems after intensive care lack description of the process of delivery and the evidence of their effectiveness. This is needed to understand how these interventions work.

DESIGN

Multistage intervention framework in a mixed-methods design. Intervention fidelity strategies were assessed for intervention design, training, delivery, receipt, and enactment with quantitative and qualitative methods inspired by the Medical Research Council and the National Institutes of Health Fidelity Framework.

METHODS

Data collection was embedded in a multicenter randomized controlled trial to explore intervention fidelity of a recovery program (December 2012-February 2017). Ten Danish intensive care units participated in the RAPIT-trial including 386 patients and 27 nurses. Quantitative data covered training and delivery. Qualitative data explored design, quality of delivery, receipt, and enactment seen from nurses' and patients' perspectives. Data were analysed statistically and by systematic deductive-inductive thematic analysis.

FINDINGS

A framework for participatory enactment of a complex intervention was developed and demonstrated delivery with high consistent fidelity across sites. Low delivery doses and variations were related to the program, patient, provider nurses and context.

CONCLUSION

Our study provides insight into the process of intervention fidelity of a nurse-led postintensive care recovery program and potentially enables professionals to understand key factors in cross-site implementation. Although we demonstrate consistent delivery and variations suggest that some patients may benefit more than others.

How context can impact clinical trials: a multi-country qualitative case study comparison of diagnostic biomarker test interventions

Background

Context matters for the successful implementation of medical interventions, but its role remains surprisingly understudied. Against the backdrop of antimicrobial resistance, a global health priority, we investigated the introduction of a rapid diagnostic biomarker test (C-reactive protein, or CRP) to guide antibiotic prescriptions in outpatient settings and asked, “Which factors account for cross-country variations in the effectiveness of CRP biomarker test interventions?”

Methods

We conducted a cross-case comparison of CRP point-of-care test trials across Yangon (Myanmar), Chiang Rai (Thailand), and Hanoi (Vietnam). Cross-sectional qualitative data were originally collected as part of each clinical trial to broaden their evidence base and help explain their respective results. We synthesised these data and developed a large qualitative data set comprising 130 interview and focus group participants (healthcare workers and patients) and nearly one million words worth of transcripts and interview notes. Inductive thematic analysis was used to identify contextual factors and compare them across the three case studies. As clinical trial outcomes, we considered patients’ and healthcare workers’ adherence to the biomarker test results, and patient exclusion to gauge the potential “impact” of CRP point-of-care testing on the population level.

Results

We identified three principal domains of contextual influences on intervention effectiveness. First, perceived risks from infectious diseases influenced the adherence of the clinical users (nurses, doctors). Second, the health system context related to all three intervention outcomes (via the health policy and antibiotic policy environment, and via health system structures and the ensuing utilisation patterns). Third, the demand-side context influenced the patient adherence to CRP point-of-care tests and exclusion from the intervention through variations in local healthcare-seeking behaviours, popular conceptions of illness and medicine, and the resulting utilisation of the health system.

Conclusions

Our study underscored the importance of contextual variation for the interpretation of clinical trial findings. Further research should investigate the range and magnitude of contextual effects on trial outcomes through meta-analyses of large sets of clinical trials. For this to be possible, clinical trials should collect qualitative and quantitative contextual information for instance on their disease, health system, and demand-side environment.

Trial registration

ClinicalTrials.gov, NCT02758821 registered on 3 May 2016 and NCT01918579 registered on 7 August 2013.

Electronic supplementary material

The online version of this article (10.1186/s13063-019-3215-9) contains supplementary material, which is available to authorized users.

Characterizing performance improvement in primary care systems in Mesoamerica: A realist evaluation protocol

Background. Evaluations of performance measurement and management interventions in public, primary care delivery systems of low- and middle-income countries are scarce. In such contexts, few studies to date have focused on characterizing how, why and under what contextual conditions do such complex, multifaceted arrangements lead to intended and unintended consequences for the healthcare workforce, the healthcare organizations involved, and the communities that are served. Methods. Case-study design with purposeful outlier sampling of high-performing primary care delivery systems in El Salvador and Honduras, as part of the Salud Mesoamerica Initiative. Case study design is suitable for characterizing individual, interpersonal and collective mechanisms of change in complex adaptive systems. The protocol design includes literature review, document review, non-participant observation, and qualitative analysis of in-depth interviews. Data analysis will use inductive and deductive approaches to identify causal patterns organized as ‘context-mechanism-outcome’ configurations. Findings will be triangulated with existing secondary data sources collected including country-specific performance measurement data, impact, and process evaluations conducted by the Salud Mesoamerica Initiative. Discussion. This realist evaluation protocol aims to characterize how, why and under what conditions do performance measurement and management arrangements contribute to the improvement of primary care system performance in two low-income countries.

Disruption as opportunity: Impacts of an organizational health equity intervention in primary care clinics

BACKGROUND

The health care sector has a significant role to play in fostering equity in the context of widening global social and health inequities. The purpose of this paper is to illustrate the process and impacts of implementing an organizational-level health equity intervention aimed at enhancing capacity to provide equity-oriented health care.

METHODS

The theoretically-informed and evidence-based intervention known as 'EQUIP' included educational components for staff, and the integration of three key dimensions of equity-oriented care: cultural safety, trauma- and violence-informed care, and tailoring to context. The intervention was implemented at four Canadian primary health care clinics committed to serving marginalized populations including people living in poverty, those facing homelessness, and people living with high levels of trauma, including Indigenous peoples, recent immigrants and refugees. A mixed methods design was used to examine the impacts of the intervention on the clinics' organizational processes and priorities, and on staff.

RESULTS

Engagement with the EQUIP intervention prompted increased awareness and confidence related to equity-oriented health care among staff. Importantly, the EQUIP intervention surfaced tensions that mirrored those in the wider community, including those related to racism, the impacts of violence and trauma, and substance use issues. Surfacing these tensions was disruptive but led to focused organizational strategies, for example: working to address structural and interpersonal racism; improving waiting room environments; and changing organizational policies and practices to support harm reduction. The impact of the intervention was enhanced by involving staff from all job categories, developing narratives about the socio-historical context of the communities and populations served, and feeding data back to the clinics about key health issues in the patient population (e.g., levels of depression, trauma symptoms, and chronic pain). However, in line with critiques of complex interventions, EQUIP may not have been maximally disruptive. Organizational characteristics (e.g., funding and leadership) and characteristics of intervention delivery (e.g., timeframe and who delivered the intervention components) shaped the process and impact.

CONCLUSIONS

This analysis suggests that organizations should anticipate and plan for various types of disruptions, while maximizing opportunities for ownership of the intervention by those within the organization. Our findings further suggest that equity-oriented interventions be paced for intense delivery over a relatively short time frame, be evaluated, particularly with data that can be made available on an ongoing basis, and explicitly include a harm reduction lens.

Assessing fidelity: balancing methodology and reality in jail interventions

BACKGROUND

While fidelity to research protocols is important to ensure generalizable outcomes, interventions in criminal justice settings present unique challenges to uniform implementation. The goal of this paper is to describe the fidelity methods and outcomes for a sexual health intervention implemented in three local county jails.

METHODS

As part of a longitudinal cohort study, four trained fidelity assessors observed 25 of the 230 sessions presented (including both intervention and comparison groups) at three separate times during the 29 months of the intervention. Assessment methods included the assessors' field notes, a nine-item facilitator quality scale and a content inclusion scale with 6-13 items specific for each of the five sessions.

RESULTS

Facilitator quality score ranged from 87.6 to 99.2%. Content inclusion scores ranged from 77.3 to 88%. Specific challenges to fidelity were found in two areas: the jail environment and the participants' response to content.

CONCLUSIONS

The realities of conducting research in jails and prisons must be addressed in real time by adjusting program content to fit both unexpected facility and participant situations. Skilled facilitators are essential to this effort.

Empowerment for healthy nutrition in German communities: a study framework

Empowerment is seen as a key strategy for sustainable health promotion efforts. However, there is only limited research on how to link the empowerment approach to the promotion of healthy eating, which is a major current public health issue. The article presents the development of a study framework for implementing and evaluating an empowerment intervention for healthy nutrition. This framework was created for a community intervention study meaning to involve elderly citizens in Bavaria, Germany. The study protocol was developed in an iterative process basing on (i) literature reviews on the topics empowerment in relation to healthy nutrition and mixed-methods evaluation, (ii) workshops with empowerment and public health experts and (iii) consultations with local community representatives. Through these measures we identified good practice criteria as well as specific challenges of integrating empowerment and healthy nutrition, e.g. engaging people in healthy nutrition, reconciling participants' nutrition preferences with public health nutrition priorities and evaluating bottom-up activities in the community. Consequences for the study design were deducted from the literature and the consultations, e.g. practical recommendations as to how power could be gradually assigned to group members. A qualitative mixed-method evaluation design was chosen to capture emergent empowerment processes. The study framework presented here is the first on empowerment and nutrition to provide explicit guidance on how empowerment may be applied to healthy nutrition and implemented and evaluated in the community context.

Community Health Workers in Diabetes Prevention and Management in Developing Countries

BACKGROUND

There is limited evidence regarding the effect of community health worker (CHW) interventions for prevention and management of the burgeoning epidemic of noncommunicable diseases (NCDs) in low- and middle-income countries (LMICs). The objective of this review was to critically appraise evidence regarding the effectiveness of CHW interventions for prevention and management of type 2 diabetes mellitus (T2DM) in LMICs.

METHODS

To identify studies that reported the effect of CHW interventions for prevention and management of T2DM in LMICs, Medline/PubMed, EMBASE, Web of Science (Science and Social Science Citation Indices), EBSCO (PsycINFO and CINAHL), POPLINE, the Cochrane Metabolic and Endocrine Disorders Group's Specialized Register, the Cochrane Central Register of Controlled Trials, the Grey literature (Google, Google Scholar), and reference lists of identified articles were searched from inception to May 31, 2017.

FINDINGS

Ten studies were included (4 pre- and post-studies, 2 randomized controlled trials, 2 cohort studies, 1 cross-sectional study, and 1 case-control study). The role of CHWs consisted of patient education, identification and referral of high-risk individuals to physicians, and provision of social support through home visits. Positive outcomes were reported in 7 of 10 studies. These outcomes included increased knowledge of T2DM symptoms and prevention measures; increased adoption of treatment-seeking and prevention measures; increased medication adherence; and improved fasting blood sugar, glycated hemoglobin, and body mass index. Three studies showed no significant outcomes.

CONCLUSIONS

CHWs have the potential to improve knowledge, health behavior, and health outcomes related to prevention and management of T2DM in LMICs. Given the limited number of studies included in this review, robust conclusions cannot be drawn at the present time.

Measures of fidelity of delivery of, and engagement with, complex, face-to-face health behaviour change interventions: A systematic review of measure quality

PURPOSE

Understanding the effectiveness of complex, face-to-face health behaviour change interventions requires high-quality measures to assess fidelity of delivery and engagement. This systematic review aimed to (1) identify the types of measures used to monitor fidelity of delivery of, and engagement with, complex, face-to-face health behaviour change interventions and (2) describe the reporting of psychometric and implementation qualities.

METHODS

Electronic databases were searched, systematic reviews and reference lists were hand-searched, and 21 experts were contacted to identify articles. Studies that quantitatively measured fidelity of delivery of, and/or engagement with, a complex, face-to-face health behaviour change intervention for adults were included. Data on interventions, measures, and psychometric and implementation qualities were extracted and synthesized using narrative analysis.

RESULTS

Sixty-six studies were included: 24 measured both fidelity of delivery and engagement, 20 measured fidelity of delivery, and 22 measured engagement. Measures of fidelity of delivery included observation (n = 17; 38.6%), self-report (n = 15; 34%), quantitatively rated qualitative interviews (n = 1; 2.3%), or multiple measures (n = 11; 25%). Measures of engagement included self-report (n = 18; 39.1%), intervention records (n = 11; 24%), or multiple measures (n = 17; 37%). Fifty-one studies (77%) reported at least one psychometric or implementation quality; 49 studies (74.2%) reported at least one psychometric quality, and 17 studies (25.8%) reported at least one implementation quality.

CONCLUSION

Fewer than half of the reviewed studies measured both fidelity of delivery of, and engagement with complex, face-to-face health behaviour change interventions. More studies reported psychometric qualities than implementation qualities. Interpretation of intervention outcomes from fidelity of delivery and engagement measurements may be limited due to a lack of reporting of psychometric and implementation qualities. Statement of contribution What is already known on this subject? Evidence of fidelity and engagement is needed to understand effectiveness of complex interventions Evidence of fidelity and engagement are rarely reported High-quality measures are needed to measure fidelity and engagement What does this study add? Evidence that indicators of quality of measures are reported in some studies Evidence that psychometric qualities are reported more frequently than implementation qualities A recommendation for intervention evaluations to report indicators of quality of fidelity and engagement measures.

Women's views and experiences of a mobile phone-based intervention to support post-abortion contraception in Cambodia

BACKGROUND

The MObile Technology for Improved Family Planning (MOTIF) trial assessed a mobile phone-based intervention comprising voice messages and counsellor support to increase post-abortion contraception at four Marie Stopes International clinics in Cambodia. The aim of this process evaluation was to assess women's views and experiences of receiving the MOTIF intervention, gain insights into the mechanism of action of the intervention and seek recommendations for improvements.

METHODS

We conducted a qualitative study comprising15 semi-structured interviews with women who had received the intervention and undertook a simple thematic analysis.

RESULTS

We identified themes relating to communication via mobile phone, supporting contraception use, broader post-abortion care, interaction with family and friends and suggestions for improvement. The majority of women were positive about the mobile phone-based intervention to support contraception use and reported it to be a convenient way to ask questions or get advice without going to a health centre, although a few women found the voice messages intrusive. The intervention supported contraception use by provision of information, encouragement, reminders to return to clinic, reassurance and advice for problems and had a positive effect on contraceptive uptake and continuation. Women reported a sense of being cared for and received support for additional physical and emotional issues. Most women thought that the duration of the intervention and frequency of messages were acceptable.

CONCLUSIONS

The majority of women were positive about the mobile phone-based intervention which provided support for contraception use as well as additional physical and emotional issues. The study provides some insights into how the intervention might have worked and considers how the intervention could be improved.

Process evaluation of a mobile phone-based intervention to support post-abortion contraception in Cambodia

BACKGROUND

The MObile Technology for Improved Family Planning (MOTIF) trial assessed a mobile phone-based intervention comprising voice messages and counsellor support to increase post-abortion contraception at four Marie Stopes International clinics in Cambodia. The aim of this process evaluation was to assess participants' interaction with the intervention from a service provider perspective.

METHODS

(1) We conducted a descriptive analysis to assess participants' interaction with the intervention. (2) In order to explore how the intervention might work, we assessed associations between interaction with the intervention and contraception use using logistic regression analysis. (3) We undertook a logistic regression analysis to assess associations between baseline socio-demographic factors and ever requesting to speak to a counsellor (pressing '1'), a variable found to be associated with contraception use.

RESULTS

Amongst 249 women that received six interactive voice messages +/- counsellor support for contraception, around half actively requested to speak to a counsellor (pressed '1') and over 90% spoke to a counsellor at some stage. Women who spoke to the counsellor having requested to (by pressing '1') were more than four times as likely to be using effective contraception at four months compared to women who didn't request or speak to the counsellor (Odds Ratio 4.39; 95% CI: 1.15-16.71). There was a small, non-statistically significant increase in contraception use amongst women that spoke to the counsellor without requesting a call. Increased parity, a history of >2 previous induced abortions, lower socio-economic status, and medical abortion (after adjusting for age, socio-economic status and residence) were associated with requesting to speak to a counsellor.

CONCLUSIONS

The interactive message can identify a subgroup of women in whom counselling will be more effective and appears to be equitable in terms of engaging those most in-need. The intervention could be adapted based on the findings of this study.

Examining Intervention Design: Lessons from the Development of Eight Related Malaria Health Care Intervention Studies

Abstract—Rigorous evidence of “what works” to improve health care is in demand, but methods for the development of interventions have not been scrutinized in the same ways as methods for evaluation. This article presents and examines intervention development processes of eight malaria health care interventions in East and West Africa. A case study approach was used to draw out experiences and insights from multidisciplinary teams who undertook to design and evaluate these studies. Four steps appeared necessary for intervention design: (1) definition of scope, with reference to evaluation possibilities; (2) research to inform design, including evidence and theory reviews and empirical formative research; (3) intervention design, including consideration and selection of approaches and development of activities and materials; and (4) refining and finalizing the intervention, incorporating piloting and pretesting. Alongside these steps, projects produced theories, explicitly or implicitly, about (1) intended pathways of change and (2) how their intervention would be implemented.The work required to design interventions that meet and contribute to current standards of evidence should not be underestimated. Furthermore, the process should be recognized not only as technical but as the result of micro and macro social, political, and economic contexts, which should be acknowledged and documented in order to infer generalizability. Reporting of interventions should go beyond descriptions of final intervention components or techniques to encompass the development process. The role that evaluation possibilities play in intervention design should be brought to the fore in debates over health care improvement.

"Because We All Trust and Care about Each Other": Exploring Tensions Translating a Theater-based HIV Prevention Intervention into a New Context

BACKGROUND

A theater-based human immunodeficiency virus (HIV) prevention intervention developed in urban California was piloted with a new partnership in North Carolina.

OBJECTIVES

This work describes the experience of translating a complex program with an enhanced partnership approach, barriers and facilitators of implementation in the new setting, and the challenges and benefits of interdisciplinary, collaborative interventions.

METHODS

We gathered perspectives of local stakeholders involved in program implementation through process evaluation interviews and focus groups with undergraduates, a college instructor, school district administrators, and high school teachers.

RESULTS

Implementing the intervention in a new setting proved feasible and successful; however, misunderstandings arose among stakeholder groups regarding teaching priorities, philosophies, and values, and were a limiting factor in partnership functioning.

CONCLUSIONS

Implementing a cross-disciplinary intervention in a new setting is best achieved through a local community-engaged process, with active involvement of relevant stakeholders. We suggest strategies to strengthen community partnerships cooperating in implementation of complex, context-tailored interventions.

Learning from a cluster randomized controlled trial to improve healthcare workers' access to prevention and care for tuberculosis and HIV in Free State, South Africa: the pivotal role of information systems

Background

Occupational tuberculosis (TB) continues to plague the healthcare workforce in South Africa. A 2-year cluster randomized controlled trial was therefore launched in 27 public hospitals in Free State province, to better understand how a combined workforce and workplace program can improve health of the healthcare workforce.

Objective

This mid-term evaluation aimed to analyze how well the intervention was being implemented, seek evidence of impact or harm, and draw lessons.

Methods

Both intervention and comparison sites had been instructed to conduct bi-annual and issue-based infection control assessments (when healthcare workers [HCW] are diagnosed with TB) and offer HCWs confidential TB and HIV counseling and testing, TB treatment and prophylaxis for HIV-positive HCWs. Intervention sites were additionally instructed to conduct quarterly workplace assessments, and also offer HCWs HIV treatment at their occupational health units (OHUs). Trends in HCW mortality, sick-time, and turnover rates (2005–2014) were analyzed from the personnel salary database (‘PERSAL’). Data submitted by the OHUs were also analyzed. Open-ended questionnaires were then distributed to OHU HCWs and in-depth interviews conducted at 17 of the sites to investigate challenges encountered.

Results

OHUs reported identifying and treating 23 new HCW cases of TB amongst the 1,372 workers who used the OHU for HIV and/or TB services; 39 new cases of HIV were also identified and 108 known-HIV-positive HCWs serviced. Although intervention-site workforces used these services significantly more than comparison-site healthcare staff (p<0.001), the data recorded were incomplete for both the intervention and comparison OHUs. An overall significant decline in mortality and turnover rates was documented over this period, but no significant differences between intervention and comparison sites; sick-time data proved unreliable. Severe OHU workload as well as residual confidentiality concerns prevented the proper implementation of protocols, especially workplace assessments and data recording. Particularly, the failure to implement computerized data collection required OHU staff to duplicate their operational data collection duties by also entering research paper forms. The study was therefore halted pending the implementation of a computerized system.

Conclusions

The significant differences in OHU use documented cannot be attributable to the intervention due to incomplete data reporting; unreliable sick-time data further precluded ascertaining the benefit potentially attributable to the intervention. Computerized data collection is essential to facilitate operational monitoring while conducting real-world intervention research. The digital divide still requires the attention of researchers along with overall infrastructural constraints.

Behind the scenes of the PRIME intervention: designing a complex intervention to improve malaria care at public health centres in Uganda

Background

In Uganda, health system challenges limit access to good quality healthcare and contribute to slow progress on malaria control. We developed a complex intervention (PRIME), which was designed to improve quality of care for malaria at public health centres.

Objective

Responding to calls for increased transparency, we describe the PRIME intervention's design process, rationale, and final content and reflect on the choices and challenges encountered during the design of this complex intervention.

Design

To develop the intervention, we followed a multistep approach, including the following: 1) formative research to identify intervention target areas and objectives; 2) prioritization of intervention components; 3) review of relevant evidence; 4) development of intervention components; 5) piloting and refinement of workshop modules; and 6) consolidation of the PRIME intervention theories of change to articulate why and how the intervention was hypothesized to produce desired outcomes. We aimed to develop an intervention that was evidence-based, grounded in theory, and appropriate for the study context; could be evaluated within a randomized controlled trial; and had the potential to be scaled up sustainably.

Results

The process of developing the PRIME intervention package was lengthy and dynamic. The final intervention package consisted of four components: 1) training in fever case management and use of rapid diagnostic tests for malaria (mRDTs); 2) workshops in health centre management; 3) workshops in patient-centred services; and 4) provision of mRDTs and antimalarials when stocks ran low.

Conclusions

The slow and iterative process of intervention design contrasted with the continually shifting study context. We highlight the considerations and choices made at each design stage, discussing elements we included and why, as well as those that were ultimately excluded. Reflection on and reporting of ‘behind the scenes’ accounts of intervention design may improve the design, assessment, and generalizability of complex interventions and their evaluations.

Tracking funded health intervention research

OBJECTIVE

To describe the research publication outputs from intervention research funded by Australia's National Health and Medical Research Council (NHMRC).

DESIGN AND SETTING

Analysis of descriptive data and data on publication outputs collected between 23 July 2012 and 10 December 2013 relating to health intervention research project grants funded between 1 January 2003 and 31 December 2007.

MAIN OUTCOME MEASURES

Stages of development of intervention studies (efficacy, effectiveness, replication, adaptation or dissemination of intervention); types of interventions studied; publication output per NHMRC grant; and whether interventions produced statistically significant changes in primary outcome variables.

RESULTS

Most of the identified studies tested intervention efficacy or effectiveness in clinical or community settings, with few testing the later stages of intervention development, such as replication, adaptation or dissemination. Studies focused largely on chronic disease treatment and management, and encompassed various medical and allied health disciplines. Equal numbers of studies had interventions that produced statistically significant results on primary outcomes, (27) and those that did not (27). The mean number of total published articles per grant was 3.3, with 2.0 articles per grant focusing on results, and the remainder covering descriptive, exploratory or methodological aspects of intervention research.

CONCLUSIONS

Our study provides a benchmark for the publication outputs of NHMRC-funded health intervention research in Australia. Research productivity is particularly important for intervention research, where findings are likely to have more immediate and direct applicability to health policy and practice. Tracking research outputs in this way provides information on whether current research investment patterns match the need for evidence about health care interventions.

Logistic, ethical, and political dimensions of stepped wedge trials: critical review and case studies

Background

Three arguments are usually invoked in favour of stepped wedge cluster randomised controlled trials: the logistic convenience of implementing an intervention in phases, the ethical benefit of providing the intervention to all clusters, and the potential to enhance the social acceptability of cluster randomised controlled trials. Are these alleged benefits real? We explored the logistic, ethical, and political dimensions of stepped wedge trials using case studies of six recent evaluations.

Methods

We identified completed or ongoing stepped wedge evaluations using two systematic reviews. We then purposively selected six with a focus on public health in high, middle, and low-income settings. We interviewed their authors about the logistic, ethical, and social issues faced by their teams. Two authors reviewed interview transcripts, identified emerging issues through qualitative thematic analysis, reflected upon them in the context of the literature, and invited all participants to co-author the manuscript.

Results

Our analysis raises three main points. First, the phased implementation of interventions can alleviate problems linked to simultaneous roll-out, but also brings new challenges. Issues to consider include the feasibility of organising intervention activities according to a randomised sequence, estimating time lags in implementation and effects, and accommodating policy changes during the trial period. Second, stepped wedge trials, like parallel cluster trials, require equipoise: without it, randomising participants to a control condition, even for a short time, remains problematic. In stepped wedge trials, equipoise is likely to lie in the degree of effect, effectiveness in a specific operational milieu, and the balance of benefit and harm, including the social value of better evaluation. Third, the strongest arguments for a stepped wedge design are logistic and political rather than ethical. The design is advantageous when simultaneous roll-out is impractical and when it increases the acceptability of using counterfactuals.

Conclusions

The logistic convenience of phased implementation is context-dependent, and may be vitiated by the additional requirements of phasing. The potential for stepped wedge trials to enhance the social acceptability of cluster randomised trials is real, but their ethical legitimacy still rests on demonstrating equipoise and its configuration for each research question and setting.

Improving treatment adherence for blood pressure lowering via mobile phone SMS-messages in South Africa: a qualitative evaluation of the SMS-text Adherence SuppoRt (StAR) trial

Background

Effective use of proven treatments for high blood pressure, a preventable health risk, is challenging for many patients. Prompts via mobile phone SMS-text messaging may improve adherence to clinic visits and treatment, though more research is needed on impact and patient perceptions of such support interventions, especially in low-resource settings.

Method

An individually-randomised controlled trial in a primary care clinic in Cape Town (2012–14), tested the effect of an adherence support intervention delivered via SMS-texts, on blood pressure control and adherence to medication, for hypertensive patients. (Trial registration: ClinicalTrials.gov NCT02019823). We report on a qualitative evaluation that explored the trial participants’ experiences and responses to the SMS-text messages, and identified barriers and facilitators to delivering adherence support via patients’ own mobile phones. Two focus groups and fifteen individual interviews were conducted. We used comparative and thematic analysis approaches to identify themes and triangulated our analysis amongst three researchers.

Results

Most participants were comfortable with the technology of using SMS-text messages. Messages were experienced as acceptable, relevant and useful to a broad range of participants. The SMS-content, the respectful tone and the delivery (timing of reminders and frequency) and the relational aspect of trial participation (feeling cared for) were all highly valued. A subgroup who benefitted the most, were those who had been struggling with adherence due to high levels of personal stress. The intervention appeared to coincide with their readiness for change, and provided practical and emotional support for improving adherence behaviour. Change may have been facilitated through increased acknowledgement of their health status and attitudinal change towards greater self-responsibility. Complex interaction of psycho-social stressors and health service problems were reported as broader challenges to adherence behaviours.

Conclusion

Adherence support for treatment of raised blood pressure, delivered via SMS-text message on the patient’s own phone, was found to be acceptable, relevant and helpful, even for those who already had their own reminder systems in place. Our findings begin to identify for whom and what core elements of the SMS-text message intervention appear to work best in a low-resource operational setting, issues that future research should explore in greater depth.

Electronic supplementary material

The online version of this article (doi:10.1186/s12875-015-0289-7) contains supplementary material, which is available to authorized users.

Minimal, negligible and negligent interventions

Many interventions are not disruptive enough of the patterns that entrench poor health and health inequities. Ways forward may require a break with tradition to embrace system-focussed theory, complex logic modelling, and ways of funding and responding to problems that address the competition of ideas and needs.