A realist evaluation of a multi-component program with disengaged students

Periods spent in the absence of education, employment, or training (NEET) are associated with adverse psychological wellbeing, social marginalisation, and premature mortality. Implementing effective programs to re-engage young people who are classified, or are at risk of becoming NEET, is of importance to these individuals, family, and society. We conducted a realist evaluation to understand how, and under which circumstances a multi-component program may impact the engagement, behavioural, and psychosocial outcomes of disengaged students at risk of becoming NEET. During the early project phase, a narrative review of the literature and key stakeholder discussions were conducted to develop our initial program theories regarding how the program was expected to achieve its outcomes. Participant observations, video footage, and forty-two interviews were then conducted with teachers and students to form context-mechanism-outcome configurations and to refine these theories. Overall, refined program theories relating to positions of authority, the power of collective experience, exploration of possible life directions, constructivist pedagogies and active learning, and the endorsement of an ethic of caring and strengths-based orientation were developed. Collectively, our findings provide a detailed understanding of the architecture of programs that may benefit disengaged students and help inform the design of future programs aimed at reducing disaffection.

Facilitating social coping-'seeking emotional and practical support from others'-as a critical strategy in maintaining the family care of people with dementia

The aim of this study was to identify how the family care of people living with dementia could be supported to make reliance on family care sustainable in the long term despite the impact of stress. A Realist Evaluation (Pawson & Tilley, 1997) was conducted to investigate this aim. An initial review established 'coping' as a primary means of mediating stressors associated with caregiving. However, there was a need to specify which coping approaches/strategies are most effective. In-depth interviews were conducted with a purposive sample of family carers (n = 18) in a suburb in North East England from 2016 to 2017. Analysis of the data revealed 'social coping' (SC) that included an emotional support component as a critical mediator of family carer stress. Several key hindrances to the utilisation of SC, including underpinning causal factors, are explicated. Ways in which these hindrances might be overcome are discussed and guidelines introduced for how family carers, formal providers and practitioners can facilitate SC as a critical coping strategy in sustaining the family care of people with dementia over the long term.

How and why are Theory of Change and Realist Evaluation used in food security contexts? A scoping review

The complex ways in which food security actions lead to nutrition and other health outcomes make it important to clarify what programs work and how, with theory-driven evaluation emerging as a promising approach to evaluate complex programs. However, it is unclear how and why theory-driven evaluation is applied in food security contexts. Our objective is to examine the development and use of Theory of Change and Realist Evaluation to support food security programs globally. Using a systematic search and screening process, we included studies that described a food security program, used a Theory of Change or Realist Evaluation, and presented original research or evaluations. We found a total of 59 relevant Theory of Change studies and eight Realist Evaluation studies. Based on our analysis, Theories of Change arose in response to three main problems: 1) the need to evaluate under complexity; 2) challenges with evaluation; and, 3) information gaps surrounding a program. In contrast, Realist Evaluation was reported to be developed primarily to understand a program's outcomes. Reflecting on the problem to be addressed in the evaluation would help improve understandings of the evaluation context, which would then inform the choice and design of an evaluation approach.

A realist evaluation exploring simulated patient role-play in pharmacist undergraduate communication training

BACKGROUND

Effective communication between pharmacists and patients is essential and improves health outcomes. Simulated patients (SPs) are trained to reproduce real-life situations and can help pharmacy students to develop and adapt their communication skills in a safe, learner-centred environment. The aim of this research was to explore how SP and pharmacy student role-play supports communication training.

METHODS

A mixed methods realist evaluation approach was adopted to test an initial theory relating to SP role-play for pharmacy students. The intervention tested involved complex communication cases in a men's and women's health module in year three of a new MPharm programme. This SP session was the first such session, of the programme which exclusively focused on complex communication skills for the students. Data collected comprised video-recordings of both training and mock OSCE sessions, and from student focus groups. Communication videos were scored using the Explanation and Planning Scale (EPSCALE) tool. Scores from SP and mock OSCE sessions were compared using the Wilcoxon-signed rank test. Focus groups were conducted with students about their experience of the training and analysed thematically, through a realist lens. Data was analysed for Context-Mechanism-Outcome configurations to produce modified programme theories.

RESULTS

Forty-six students (n = 46/59, 78 %) consented to their video-recorded interactions to be used. Students identified contextual factors relating to the timing within the course and the setting of the intervention, the debrief and student individual contexts. Mechanisms included authenticity, feedback, reflection, self-awareness and confidence. Negative responses included embarrassment and nervousness. They distinguished outcomes including increased awareness of communication style, more structured communication and increased comfort. However quantitative data showed a decrease (p < 0.001) in communication scores in the mock OSCE compared with scores from training sessions. Modified programme theories relating to SP training for pharmacy students were generated.

CONCLUSIONS

SP role-play is a valuable communication skills training approach. Emphasis should be placed on multiple stakeholder feedback and promotion of reflection. Time limits need to be considered in this context and adjusted to meet student needs, especially for students with lower levels of communication comfort and those communicating in languages different to their first language.

Strengthening Community Health Systems Through Novel eHealth Initiatives? Commencing a Realist Study of the Virtual Health Rooms in Rural Northern Sweden

Background: Unlike the large body of research that has examined the ‘success’ or ‘failure’ of eHealth in terms of patient and provider perceptions or cost- and clinical effectiveness, the current study teases out ways through which a novel eHealth initiative in rural northern Sweden might result in more distal or systemic beneficial outcomes. More specifically, this paper aims to explore how and under what circumstances the so-called virtual health rooms (VHRs) are expected to improve access to person-centred care and strengthen community health systems, especially for elderly residents of rural areas.

Methods: The first phase of the realist evaluation methodology was conducted, involving qualitative interviews with 8 key stakeholders working with eHealth, business development, digitalisation, and process management. Using thematic analysis and following an abductive-retroductive analytical process, an intervention-context-actor-mechanism-outcome(ICAMO) configuration was developed and elicited into an initial programme theory.

Results: The findings indicate that a novel eHealth initiative, which provides reliable technologies in a customized facility that connects communities and providers, might improve access to person-centred care and strengthen community health systems for rural populations. This is theorized to occur if mechanisms acting at individual (such as knowledge, skills and trust) and collective (like a common vision and shared responsibilities) levels are triggered in contexts characterised by supportive societal transitions, sufficient organisational readiness and the harnessing of rural cohesiveness and creativity.

Conclusion: The elicited initial programme theory describes and explains how a novel eHealth initiative in rural northern Sweden is presumed to operate and under what circumstances. Further testing, refinements and continued gradual building of theory following the realist evaluation methodology is now needed to ascertain if the ‘VHRs’ work as intended, for whom, in what conditions and why.

'At this [adherence] club, we are a family now': A realist theory-testing case study of the antiretroviral treatment adherence club, South Africa

Background

An estimated 7.9 million people were living with HIV in South Africa in 2017, with 63.3% of them remaining in antiretroviral therapy (ART) care and 62.9% accessing ART. Poor retention in care and suboptimal adherence to ART undermine the successful efforts of initiating people living with HIV on ART. To address these challenges, the antiretroviral adherence club intervention was designed to streamline ART services to 'stable' patients. Nevertheless, it is poorly understood exactly how and why and under what health system conditions the adherence club intervention works.

Objectives

The aim of this study was to test a theory on how and why the adherence club intervention works and in what health system context(s) in a primary healthcare facility in the Western Cape Province.

Method

Within the realist evaluation framework, we applied a confirmatory theory-testing case study approach. Kaplan-Meier descriptions were used to estimate the rates of dropout from the adherence club intervention and virological failure as the principal outcomes of the adherence club intervention. Qualitative interviews and non-participant observations were used to explore the context and identify the mechanisms that perpetuate the observed outcomes or behaviours of the actors. Following the retroduction logic of making inferences, we configured information obtained from quantitative and qualitative approaches using the intervention-context-actor-mechanism-outcome heuristic tool to formulate generative theories.

Results

We confirmed that patients on ART in adherence clubs will continue to adhere to their medication and remain in care because their self-efficacy is improved; they are motivated or are being nudged.

Conclusion

A theory-based understanding provides valuable lessons towards the adaptive implementation of the adherence club intervention.

Realist evaluation of a complex integrated care programme: protocol for a mixed methods study

INTRODUCTION

The lack of understanding of how complex integrated care programmes achieve their outcomes due to the lack of acceptable methods leads to difficulties in the development, implementation, adaptation and scaling up of similar interventions. In this study, we evaluate an integrated care network, the National University Health System (NUHS) Regional Health System (RHS), consisting of acute hospitals, step down care, primary care providers, social services and community partners using a theory-driven realist evaluation approach. This study aims to examine how and for whom the NUHS-RHS works to improve healthcare utilisations, outcomes, care experiences and reduce healthcare costs. By using a realist approach that balances the needs of context-specific evaluation with international comparability, this study carries the potential to address current research gaps.

METHODS AND ANALYSIS

This evaluation will be conducted in three research phases: (1) development of initial programme theory (IPT) underlying the NUHS-RHS; (2) testing of programme theory using empirical data; and (3) refinement of IPT. IPT was elicited and developed through reviews of programme documents, informal discussions and in-depth interviews with relevant stakeholders. Then, a convergent parallel mixed method study will be conducted to assess context (C), mechanisms (M) and outcomes (O) to test the IPT. Findings will then be analysed according to the realist evaluation formula of CMO in which findings on the context, mechanisms will be used to explain the outcomes. Finally, based on findings gathered, IPT will be refined to highlight how to improve the NUHS-RHS by detailing what works (outcome), as well as how (mechanisms) and under what conditions (context).

ETHICS AND DISSEMINATION

The National Healthcare Group, Singapore, Domain Specific Review Board reviewed and approved this study protocol. Study results will be published in international peer-reviewed journals and presented at conferences and internally to NUHS-RHS and Ministry of Health, Singapore.

Characterizing performance improvement in primary care systems in Mesoamerica: A realist evaluation protocol

Background. Improving performance of primary care systems in low- and middle-income countries (LMICs) may be a necessary condition for achievement of universal health coverage in the age of Sustainable Development Goals. The Salud Mesoamerica Initiative (SMI), a large-scale, multi-country program that uses supply-side financial incentives directed at the central-level of governments, and continuous, external evaluation of public, health sector performance to induce improvements in primary care performance in eight LMICs. This study protocol seeks to explain whether and how these interventions generate program effects in El Salvador and Honduras. Methods. This study presents the protocol for a study that uses a realist evaluation approach to develop a preliminary program theory that hypothesizes the interactions between context, interventions and the mechanisms that trigger outcomes. The program theory was completed through a scoping review of relevant empirical, peer-reviewed and grey literature; a sense-making workshop with program stakeholders; and content analysis of key SMI documents. The study will use a multiple case-study design with embedded units with contrasting cases. We define as a case the two primary care systems of Honduras and El Salvador, each with different context characteristics. Data will be collected through in-depth interviews with program actors and stakeholders, documentary review, and non-participatory observation. Data analysis will use inductive and deductive approaches to identify causal patterns organized as 'context, mechanism, outcome' configurations. The findings will be triangulated with existing secondary, qualitative and quantitative data sources, and contrasted against relevant theoretical literature. The study will end with a refined program theory. Findings will be published following the guidelines generated by the Realist and Meta-narrative Evidence Syntheses study (RAMESES II). This study will be performed contemporaneously with SMI's mid-term stage of implementation. Of the methods described, the preliminary program theory has been completed. Data collection, analysis and synthesis remain to be completed.

Gender equity programmes in academic medicine: a realist evaluation approach to Athena SWAN processes

OBJECTIVES

Gender inequity has persisted in academic medicine. Yet equity is vital for countries to achieve their full potential in terms of translational research and patient benefit. This study sought to understand how the gender equity programme, Athena SWAN, can be enabled and constrained by interactions between the programme and the context it is implemented into, and whether these interactions might produce unintended consequences.

DESIGN

Multimethod qualitative case studies using a realist evaluation approach.

SETTING

5 departments from a university medical school hosting a Translational Research Organisation.

PARTICIPANTS

25 hours of observations of gender equality committee meetings, 16 in-depth interviews with Heads of Departments, Committee Leads and key personnel involved in the initiative. 4 focus groups with 15 postdoctoral researchers, lecturers and senior lecturers.

RESULTS

The implementation of Athena SWAN principles was reported to have created social space to address gender inequity and to have highlighted problematic practices to staff. However, a number of factors reduced the programme's potential to impact gender inequity. Gender inequity was reproduced in the programme's enactment as female staff was undertaking a disproportionate amount of Athena SWAN work, with potential negative impacts on individual women's career progression. Early career researchers experienced problems accessing Athena SWAN initiatives. Furthermore, the impact of the programme was perceived to be undermined by wider institutional practices, national policies and societal norms, which are beyond the programme's remit.

CONCLUSIONS

Gender equity programmes have the potential to address inequity. However, paradoxically, they can also unintentionally reproduce and reinforce gender inequity through their enactment. Potential programme impacts may be undermined by barriers to staff availing of career development and training initiatives, and by wider institutional practices, national policies and societal norms.

Improving the performance of community health workers in humanitarian emergencies: a realist evaluation protocol for the PIECES programme

INTRODUCTION

Understanding what enhances the motivation and performance of community health workers (CHWs) in humanitarian emergencies represents a key research gap within the field of human resources for health. This paper presents the research protocol for the Performance ImprovEment of CHWs in Emergency Settings (PIECES) research programme. Enhancing Learning and Research in Humanitarian Action (ELRHA) funded the development of this protocol as part of their Health in Humanitarian Crises (R2HC) call (No.19839). PIECES aims to understand what factors improve the performance of CHWs in level III humanitarian emergencies.

METHODS AND ANALYSIS

The suggested protocol uses a realist evaluation with multiple cases across the 3 country sites: Turkey, Iraq and Lebanon. Working with International Medical Corps (IMC), an initial programme theory was elicited through literature and document reviews, semistructured interviews and focus groups with IMC programme managers and CHWs. Based on this initial theory, this protocol proposes a combination of semistructured interviews, life histories and critical incident narratives, surveys and latent variable modelling of key constructs to explain how contextual factors work to trigger mechanisms for specific outcomes relating to IMC's 300+ CHWs' performance. Participants will also include programme staff, CHWs and programme beneficiaries. Realist approaches will be used to better understand 'what works, for whom and under what conditions' for improving CHW performance within humanitarian contexts.

ETHICS AND DISSEMINATION

Trinity College Dublin's Health Policy and Management/Centre for Global Health Research Ethics Committee gave ethical approval for the protocol development phase. For the full research project, additional ethical approval will be sought from: Université St. Joseph (Lebanon), the Ethics Committee of the Ministry of Health in Baghdad (Iraq) and the Middle East Technical University (Turkey). Dissemination activities will involve a mixture of research feedback, policy briefs, guidelines and recommendations, as well as open source academic articles.

The Well Organised Working Environment: A mixed methods study

BACKGROUND

The English National Health Service Institute for Innovation and Improvement designed a series of programmes called The Productive Series. These are innovations designed to help healthcare staff reduce inefficiency and improve quality, and have been implemented in healthcare organisations in at least 14 different countries. This paper examines an implementation of the first module of the Productive Community Services programme called 'The Well Organised Working Environment'.

OBJECTIVE

The quantitative component aims to identify the quantitative outcomes and impact of the implementation of the Well Organised Working Environment module. The qualitative component aims to describe the contexts, mechanisms and outcomes evident during the implementation, and to consider the implication of these findings for healthcare staff, commissioners and implementation teams.

DESIGN

Mixed methods explanatory sequential design.

SETTINGS

Community Healthcare Organisation in East Anglia, England.

PARTICIPANTS

For the quantitative data, participants were 73 staff members that completed End of Module Assessments. Data from 25 services that carried out an inventory of stock items stored were also analysed. For the qualitative element, participants were 45 staff members working in the organisation during the implementation, and four members of the Productive Community Services Implementation Team.

METHODS

Staff completed assessments at the end of the module implementation, and the value of items stored by clinical services was recorded. After the programme concluded, semi-structured interviews with staff and a focus group with members of the Productive Community Services implementation team were analysed using Framework Analysis employing the principles of Realist Evaluation.

RESULTS

62.5% respondents (n=45) to the module assessment reported an improvement in their working environment, 37.5% (n=27) reported that their working environment stayed the same or deteriorated. The reduction of the value of items stored by services ranged from £4 to £5039 across different services. Results of the qualitative analysis suggests explanations for why the programme worked in some contexts and not others, for instance due to varying levels of management support, and varying levels of resources allocated to carrying out or sustaining the improvement work.

CONCLUSIONS

Quantitative analysis of data generated during healthcare improvement initiatives can give an impression of the benefits realised, but additional qualitative analysis also provides opportunity for learning to improve future implementations. Targets set by commissioners for innovation should focus on sustaining improvement rather demonstrating one-off benefits, and implementation teams should not let their preconceptions of what will and what will not work prevent them from trying interventions that may benefit staff.