Understanding integrated care pathways in palliative care using realist evaluation: a mixed methods study protocol

Development of palliative care clinical practice guidelines and referral care pathways for primary care practitioners in Pakistan

BACKGROUND

Despite a large burden of life-limitingillness, there exists a dearth of services of palliative care in Pakistan. International guidelines have questionable applicability in Pakistan due to the socioeconomic differences. We generated a protocol describing the process of developing comprehensive palliative care guidelines and palliative care referral pathways for primary care practitioners to adopt in Pakistan.

METHODS

A GRADE-ADOLOPMENT approach with modification has been employed to create guidelines for a Pakistani context. The "National Comprehensive Cancer Network Guidelines Insights: Palliative Care, Version 2.2021" was used as the source guideline. Recommendations from the source guideline were reviewed by two local palliative care specialists to either "Adopt," "Adapt" or "Exclude". The finalized recommendations were incorporated into the local palliative care guideline. Clinical diagnosis and referral pathways were made from the finalized guideline. Any gaps in management found in the pathways were filled by taking existing recommendations from other credible guidelines.

RESULTS

Twenty-seven recommendations were adopted without modification. No recommendations were deemed to be adapted and 15 were excluded. The referral care pathways created were reflective of the local guideline and included elements of initial assessment, preliminary management, reassessment, and referral. 6 additional recommendations were made.

CONCLUSION

The described clinical practice guidelines and primary care clinical referral pathways will aid to standardize palliative care provision in Pakistan. These can be used by other resource constrained settings to develop guidelines within their own local context.

Supporting stroke survivors to meet their personal rehabilitation needs in community-based arm rehabilitation: development of initial programme theories to explore what may work for whom, how and under what circumstances

Objective

This study explored what worked for whom, how and under what circumstances in a community-based augmented arm rehabilitation programme that was designed to enable stroke survivors to meet their personal rehabilitation needs.

Design

A mixed methods realist-informed study of data from a randomised controlled feasibility trial, comparing augmented arm rehabilitation after stroke with usual care. The analysis was designed to develop initial programme theories and refine these through triangulation of qualitative and quantitative trial data. Participants with a confirmed stroke diagnosis and stroke-related arm impairment were recruited from five health boards in Scotland. Only data from participants in the augmented group were analysed. The augmented intervention comprised evidence-based arm rehabilitation (27 additional hours over 6 weeks) including self-managed practice, and focused on individual rehabilitation needs identified through the Canadian Occupational Performance Measure (COPM). The COPM indicated to which extent rehabilitation needs were met following the intervention, the Action Research Arm Test provided data on changes in arm function, and qualitative interviews provided information about the context and potential mechanisms of action.

Findings

Seventeen stroke survivors (11 males, age range 40-84 years, NIHSS median (IQR) 6 (8)) were included. Median (IQR) COPM Performance and Satisfaction scores (min.1-max.10) improved from pre-intervention 2 (5) to post-intervention 5 (7). Findings suggested that meeting rehabilitation needs was facilitated by strengthening participants' sense of intrinsic motivation (through grounding exercises in everyday activities linked to valued life roles, and enabling them to overcome barriers to self-managed practice), and via therapeutic relationships (through trust and expertise, shared decision-making, encouragement and emotional support). Collectively, these mechanisms enabled stroke survivors to build confidence and gain mastery experience necessary to engage in new self-managed practice routines.

Conclusion

This realist-informed study enabled the development of initial programme theories to explain how and in what circumstances the augmented arm rehabilitation intervention may have enabled participants to meet their personal rehabilitation needs. Encouraging participants' sense of intrinsic motivation and building therapeutic relationships appeared instrumental. These initial programme theories require further testing, refinement, and integration with the wider literature.

Health literacy among self-help leprosy group members reduces stereotype endorsement and stigma-related harm in rural Nepal

There is increasing appreciation that group memberships can have both beneficial and damaging impacts on health. In collaboration with Nepal Leprosy Trust (NLT), this longitudinal study explores a group-based approach to stigma reduction among people affected by leprosy in rural Nepal (N = 71)-a hard to reach and underrepresented non-WEIRD population. Informed by the 'social cure' literature, and the progressive model of self-stigma, we use a longitudinal design. We found that a sense of belonging to a self-help group can facilitate education in terms of health literacy, and over time these two factors also have impacts on participants stigma. Specifically, self-help group belonging predicted improvements in health literacy, leading to reduced endorsement of negative stereotypes and thus less stigma-related harm among people affected by leprosy. The study offers promising evidence that group-based interventions, which support health education, can reduce the harmful impact of stigma in very challenging contexts.

Evaluating Large-Scale Integrated Care Projects: The Development of a Protocol for a Mixed Methods Realist Evaluation Study in Belgium

BACKGROUND

The twelve Integrated Care Program pilot projects (ICPs) created by the government plan 'Integrated Care for Better Health' aim to achieve four outcome types (the Quadruple Aim) for people with chronic diseases in Belgium: improved population health, improved patient and provider experiences and improved cost efficiency. The aim of this article is to present the development of a mixed methods realist evaluation of this large-scale, whole system change programme.

METHODS

A scientific team was commissioned to co-design and implement an evaluation protocol in close collaboration with the government, the ICPs and several other involved stakeholders.

RESULTS

A protocol for a mixed methods realist evaluation was developed to gain insights into the mechanisms that foster successful results in ICPs. The qualitative evaluation proposed will be based on the document analysis of yearly ICP progress reports, selected case studies and focus group interviews with stakeholders. Processes and outcomes of all the projects will be monitored using indicators based on administrative data on population health and the quality and costs of care. A yearly survey will be organized to collect data on patient-reported outcomes and experiences and on provider-reported measures of inter-professional collaboration and proper wellbeing. Using both quantitative and qualitative data, we will develop theories about the mechanisms and the associated contextual factors that lead to integrated care and the Quadruple Aim outcomes.

DISCUSSION

The objective of this study is to deliver policy recommendations on strategies and best practices to improve care integration in Belgium and to implement a sustainable monitoring system that serves both policy makers and the stakeholders within the ICPs. Some challenges due to the large scale of the project and the multiple stakeholders involved may impede the successful implementation of this proposal.

Integrated palliative care definition and constitutive elements: scoping review

Purpose

The purpose of this paper is to propose a unified definition of integrated palliative care (IPC), and to identify the elements that facilitate or hinder implementation of an integrated palliative care system (IPCS).

Design/methodology/approach

A scoping review of the conceptualization and essential elements of IPC was undertaken, based on a search of the PubMed, Scopus and ISI Web of Science databases. The search identified 79 unduplicated articles; 43 articles were selected for content analysis.

Findings

IPC is coordinated and collaborative across different health organizations, levels of care and types of providers. Eight key elements facilitate implementation of an IPCS: coordination, early patient identification, patient-centered services, care continuity, provider education and training, a standard implementation model and screening tool, shared information technology system, and supportive policies and funding. These elements were plotted as a “Circle of Integrated Palliative Care System Elements.”

Practical implications

This paper offers researchers an inclusive definition of IPC and describes the essential elements of its successful implementation.

Originality/value

This study provides evidence from researchers on five continents, offering insights from multiple countries and cultures on the topic of IPC. The findings of this thematic analysis could assist international researchers aiming to develop a standard evaluative model or assess the level of integration in a health care system’s delivery of palliative care.

Motivation as a mechanism underpinning exercise-based falls prevention programmes for older adults with cognitive impairment: a realist review

OBJECTIVES

This review aimed to identify mechanisms underlying participation in falls prevention interventions, in older adults with cognitive impairment. In particular we studied the role of motivation.

DESIGN

A realist review of the literature.

DATA SOURCES

EMBASE, MEDLINE, CINAHL, the Cochrane Library, PsycINFO and PEDRO.

ELIGIBILITY CRITERIA

Publications reporting exercise-based interventions for people with cognitive impairment, including dementia, living in the community.

DATA EXTRACTION AND SYNTHESIS

A 'rough programme theory' (a preliminary model of how an intervention works) was developed, tested against findings from the published literature and refined. Data were collected according to elements of the programme theory and not isolated to outcomes. Motivation emerged as a key element, and was prioritised for further study.

RESULTS

An individual will access mechanisms to support participation when they think that exercise will be beneficial to them. Supportive mechanisms include having a 'gate-keeper', such as a carer or therapist, who shares responsibility for the perception of exercise as beneficial. Lack of access to support decreases adherence and participation in exercise. Motivational mechanisms were particularly relevant for older adults with mild-to-moderate dementia, where the exercise intervention was multicomponent, in a preferred setting, at the correct intensity and level of progression, correctly supported and considered, and flexibly delivered.

CONCLUSION

Motivation is a key element enabling participation in exercise-based interventions for people with cognitive impairment. Many of the mechanisms identified in this review have parallels in motivational theory. Clinically relevant recommendations were derived and will be used to further develop and test a motivationally considered exercise-based falls intervention for people with mild dementia.

PROSPERO REGISTRATION NUMBER

CRD42015030169.

Realist evaluation of cancer rehabilitation services in South Wales (REEACaRS): a mixed methods study protocol

INTRODUCTION

Long-term and late effects of cancer treatments can cause functional limitations and reduce quality of life. Cancer rehabilitation services, which can comprise physical exercise, psychological support and educational interventions depending on the individual's needs, have been found to have a positive effect on health-related quality of life worldwide. However, accessibility or the lack of awareness on available help can act as barriers and influence the uptake of services, resulting in people having unmet rehabilitation needs. In Wales, UK, 41% of people, who have had health and social care needs resulting from cancer and its treatments, reported that they did not receive care when needed. The reason for this lack of support has not yet been fully investigated. The aim of this study is to investigate the conditions in which cancer rehabilitation services work and their underpinning mechanisms in South Wales, UK, specifically addressing barriers, facilitators and costs.

METHODS AND ANALYSIS

Realist evaluation, which explains for whom a service works in what circumstances and how through context-mechanism-outcome pattern conjunctions, will be used in three phases to investigate the conditions in which cancer rehabilitation services work and their underpinning mechanisms. Phase 1 will be secondary analysis of a cancer rehabilitation database from a local Health Board to give context to who are accessing rehabilitation. Phase 2 will be thematic analysis of face-to-face, semistructured rehabilitation participant (n=20) and healthcare professional (n=20) interviews to explore the mechanisms of how cancer rehabilitation works. Phase 3 will be two case studies and cost-consequences analysis of cancer rehabilitation services.

ETHICS AND DISSEMINATION

This study received favourable ethical opinion from London South-East Research Ethics Committee (17/LO/2123) in December 2017. This project is part of the author's PhD thesis and it is expected that the findings will be disseminated in academic journals and at local and international conferences.

The Development of Pathways in Palliative Medicine: Definition, Models, Cost and Quality Impact

Palliative Care and its medical subspecialty, known as Palliative Medicine, is the care of anyone with a serious illness. This emerging field includes Hospice and comfort care, however, it is not limited to end-of-life care. Examples of the types of serious illness that Palliative Medicine clinicians care for include and are not limited to hematologic and oncologic diseases, such as cancer, advanced heart and lung diseases (e.g., congestive heart failure and chronic obstructive pulmonary disorder), advanced liver and kidney diseases, and advanced neurologic illnesses (e.g., Alzheimer’s and Parkinson’s disease). In the past decade, there has been tremendous growth of Palliative Medicine programs across the country. As the population of patients with serious illnesses increases, there is growing concentration on quality of care, including symptom management, meeting patients’ goals regarding their medical care and providing various types of support, all of which are provided by Palliative Medicine. In this review article we define Palliative Medicine, describe care pathways and their applicability to Palliative Medicine, identify different models for Palliative Care and provide evidence for its impact on cost and quality of care.

Open communication strategies between a triad of 'experts' facilitates death in usual place of residence: A realist evaluation

BACKGROUND

In order to meet policy drivers on death in usual place of residence, it is key to understand how shared decision-making can be facilitated in practice. An integrated care pathway was implemented in primary care in the North East of England to facilitate death in usual place of residence.

AIM

To understand how, for whom and in which circumstances death in usual place of residence is facilitated.

DESIGN

A mixed method realist evaluation was employed. Local primary care practice death audit data were analysed to identify outcomes using a mixed effects logistic regression model. Focus groups and interviews with staff of the integrated care pathway and bereaved relatives were analysed to identify the related contexts and mechanisms.

SETTING/PARTICIPANTS

Death audit data of 4182 patients were readily available from 14 general practitioner practices. Three focus groups were conducted with primary and secondary care staff, voluntary sector organisations and care home representatives. Interviews with bereaved relatives were carried out in participants' homes ( n = 5).

RESULTS

A mixed effects logistic regression model indicated a significant effect of year on death in usual place of residence when compared to a model without year using an analysis of deviance ( p = 0.016). Qualitative analysis suggested that this outcome was achieved when a triad of 'experts' (comprising patient, family members/family carers/formal carers and healthcare professionals) used open communication strategies.

CONCLUSION

An empirically supported theory of how, for whom and in which circumstances death in usual place of residence happens is provided, which has important implications for both policy and practice.

Understanding the implementation and adoption of a technological intervention to improve medication safety in primary care: a realist evaluation

Background

Monitoring for potentially hazardous prescribing is increasingly important to improve medication safety. Healthcare information technology can be used to achieve this aim, for example by providing access to prescribing data through surveillance of patients’ electronic health records. The aim of our study was to examine the implementation and adoption of an electronic medicines optimisation system that was intended to facilitate clinical audit in primary care by identifying patients at risk of an adverse drug event. We adopted a sociotechnical approach that focuses on how complex social, organisational and institutional factors may impact upon the use of technology within work settings.

Methods

We undertook a qualitative realist evaluation of the use of an electronic medicines optimisation system in one Clinical Commissioning Group in England. Five semi-structured interviews, four focus groups and one observation were conducted with a range of stakeholders. Consistent with a realist evaluation methodology, the analysis focused on exploring the links between context, mechanism and outcome to explain the ways the intervention might work, for whom and in what circumstances.

Results

Using the electronic medicines optimisation system could lead to a number of improved patient safety outcomes including pre-emptively reviewing patients at risk of adverse drug events. The effective use of the system depended upon engagement with the system, the flow of information between different health professionals centrally placed at the Clinical Commissioning Group and those locally placed at individual general practices, and upon variably adapting work practices to facilitate the use of the system. The use of the system was undermined by perceptions of ownership, lack of access, and lack of knowledge and awareness.

Conclusions

The use of an electronic medicines optimisation system may improve medication safety in primary care settings by identifying those patients at risk of an adverse drug event. To fully realise the potential benefits for medication safety there needs to be better utilisation across primary care and with a wider range of stakeholders. Engaging with all potential stakeholders and users prior to implementation of such systems might allay perceptions that the system is owned centrally and increase knowledge of the potential benefits.

Evaluating the implementation of a national clinical programme for diabetes to standardise and improve services: a realist evaluation protocol

BACKGROUND

Over the last three decades in response to the growing burden of diabetes, countries worldwide have developed national and regional multifaceted programmes to improve the monitoring and management of diabetes and to enhance the coordination of care within and across settings. In Ireland in 2010, against a backdrop of limited dedicated strategic planning and engrained variation in the type and level of diabetes care, a national programme was established to standardise and improve care for people with diabetes in Ireland, known as the National Diabetes Programme (NDP). The NDP comprises a range of organisational and service delivery changes to support evidence-based practices and policies. This realist evaluation protocol sets out the approach that will be used to identify and explain which aspects of the programme are working, for whom and in what circumstances to produce the outcomes intended.

METHODS/DESIGN

This mixed method realist evaluation will develop theories about the relationship between the context, mechanisms and outcomes of the diabetes programme. In stage 1, to identify the official programme theories, documentary analysis and qualitative interviews were conducted with national stakeholders involved in the design, development and management of the programme. In stage 2, as part of a multiple case study design with one case per administrative region in the health system, qualitative interviews are being conducted with frontline staff and service users to explore their responses to, and reasoning about, the programme's resources (mechanisms). Finally, administrative data will be used to examine intermediate implementation outcomes such as service uptake, acceptability, and fidelity to models of care.

DISCUSSION

This evaluation is using the principles of realist evaluation to examine the implementation of a national programme to standardise and improve services for people with diabetes in Ireland. The concurrence of implementation and evaluation has enabled us to produce formative feedback for the NDP while also supporting the refinement and revision of initial theories about how the programme is being implemented in the dynamic and unstable context of the Irish healthcare system.

Reducing inequalities in care for patients with non-malignant diseases: Insights from a realist evaluation of an integrated palliative care pathway

BACKGROUND

The need for palliative care is growing internationally with an increasing prevalence of non-malignant diseases. The integrated care pathway was implemented in primary care by multidisciplinary teams from 2009 in a locality in the North East of England. Fourteen general practitioner practices provided data for the study.

AIM

To find whether, how, and under what circumstances palliative care registrations are made for patients with non-malignant diseases in primary care.

DESIGN

General practitioner practice data were analysed statistically and qualitative data were collected from health care professionals and members of relevant organisations.

FINDINGS

A mixed-effects logistic model indicated a significant difference beyond the 0.1% level (p < 0.001) in registrations between the malignant and non-malignant groups in 2011, with an odds ratio of 0.09 (=exp(-2.4266)), indicating that patients in the non-malignant group are around 11 times (1/0.09) less likely to be registered than patients in the malignant group. However, patients with non-malignant diseases were significantly more likely to be registered in 2012 than in 2011 with an odds ratio of 1.46, significant beyond the 1% level. Qualitative analyses indicate that health care professionals find registering patients with non-malignant diseases stressful, yet feel that their confidence in treating this population is increasing.

CONCLUSION

The integrated care pathway began to enable the reduction in inequalities in care by identifying, registering and managing an increasing number of palliative patients with non-malignant diseases. Consensual and inclusive definitions of palliative care were developed in order to legitimise the registration of such patients.

Do-not-attempt-cardiopulmonary-resuscitation decisions: an evidence synthesis

Background

Cardiac arrest is the final common step in the dying process. In the right context, resuscitation can reverse the dying process, yet success rates are low. However, cardiopulmonary resuscitation (CPR) is a highly invasive medical treatment, which, if applied in the wrong setting, can deprive the patient of dignified death. Do-not-attempt-cardiopulmonary-resuscitation (DNACPR) decisions provide a mechanism to withhold CPR. Recent scientific and lay press reports suggest that the implementation of DNACPR decisions in NHS practice is problematic.

Aims and objectives

This project sought to identify reasons why conflict and complaints arise, identify inconsistencies in NHS trusts’ implementation of national guidelines, understand health professionals’ experience in relation to DNACPR, its process and ethical challenges, and explore the literature for evidence to improve DNACPR policy and practice.

Methods

A systematic review synthesised evidence of processes, barriers and facilitators related to DNACPR decision-making and implementation. Reports from NHS trusts, the National Reporting and Learning System, the Parliamentary and Health Service Ombudsman, the Office of the Chief Coroner, trust resuscitation policies and telephone calls to a patient information line were reviewed. Multiple focus groups explored service-provider perspectives on DNACPR decisions. A stakeholder group discussed the research findings and identified priorities for future research.

Results

The literature review found evidence that structured discussions at admission to hospital or following deterioration improved patient involvement and decision-making. Linking DNACPR to overall treatment plans improved clarity about goals of care, aided communication and reduced harms. Standardised documentation improved the frequency and quality of recording decisions. Approximately 1500 DNACPR incidents are reported annually. One-third of these report harms, including some instances of death. Problems with communication and variation in trusts’ implementation of national guidelines were common. Members of the public were concerned that their wishes with regard to resuscitation would not be respected. Clinicians felt that DNACPR decisions should be considered within the overall care of individual patients. Some clinicians avoid raising discussions about CPR for fear of conflict or complaint. A key theme across all focus groups, and reinforced by the literature review, was the negative impact on overall patient care of having a DNACPR decision and the conflation of ‘do not resuscitate’ with ‘do not provide active treatment’.

Limitations

The variable quality of some data sources allows potential overstatement or understatement of findings. However, data source triangulation identified common issues.

Conclusion

There is evidence of variation and suboptimal practice in relation to DNACPR decisions across health-care settings. There were deficiencies in considering, discussing and implementing the decision, as well as unintended consequences of DNACPR decisions being made on other aspects of patient care.

Future work

Recommendations supported by the stakeholder group are standardising NHS policies and forms, ensuring cross-boundary recognition of DNACPR decisions, integrating decisions with overall treatment plans and developing tools and training strategies to support clinician and patient decision-making, including improving communication.

Study registration

This study is registered as PROSPERO CRD42012002669.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Exposing the impact of Citizens Advice Bureau services on health: a realist evaluation protocol

INTRODUCTION

Welfare advice services can be used to address health inequalities, for example, through Citizens Advice Bureau (CAB). Recent reviews highlight evidence for the impact of advice services in improving people's financial position and improving mental health and well-being, daily living and social relationships. There is also some evidence for the impact of advice services in increasing accessibility of health services, and reducing general practitioner appointments and prescriptions. However, direct evidence for the impact of advice services on lifestyle behaviour and physical health is currently much less well established. There is a need for greater empirical testing of theories around the specific mechanisms through which advice services and associated financial or non-financial benefits may generate health improvements.

METHODS AND ANALYSIS

A realist evaluation will be conducted, operationalised in 5 phases: building the explanatory framework; refining the explanatory framework; testing the explanatory framework through empirical data (mixed methods); development of a bespoke data recording template to capture longer term impact; and verification of findings with a range of CAB services. This research will therefore aim to build, refine and test an explanatory framework about how CAB services can be optimally implemented to achieve health improvement.

ETHICS AND DISSEMINATION

The study was approved by the ethics committee at Northumbria University, UK. Project-related ethical issues are described and quality control aspects of the study are considered. A stakeholder mapping exercise will inform the dissemination of results in order to ensure all relevant institutions and organisations are targeted.

Characteristics of a palliative care consultation service with a focus on pain in a German university hospital

Background

A minority of patients with incurable and advanced disease receive specialised palliative care. Specialised palliative care services that complement the care of difficult and complex cases ought to be integrated with services that deliver general care for most patients. A typical setting in which this integrative concept takes place is the hospital setting, where patients suffering from incurable and advanced disease are treated in many different departments.

The aim of the study is to investigate the profile and spectrum of a palliative care consultation service (PCCS) at a German university hospital with special reference to pain therapy.

Methods

We retrospectively analysed the PCCS documentation of three years.

Results

Most patients were referred from non-surgical departments, 72% were inpatients, and 28% were outpatients. 98% of the patients suffered from cancer. Counselling in pain therapy was one of the key aspects of the consultation: For 76% of all consulted patients, modifications of the analgesic regimen were recommended, which involved opioids in 96%. Recommendations on breakthrough-pain medication were made for 70% of the patients; this was an opioid in most cases (68%). The most commonly used opioid was morphine. For 17% of the patients, additional diagnostic procedures were recommended. Besides pain management palliative care consultation implied a wide range of recommendations and services: In addition to organising home care infrastructure, palliative care services supported patients and their families in understanding the life-limiting diseases. They also coordinated physical therapy and social and legal advice.

Conclusion

This survey clearly shows that for a consultation service to support patients with incurable or advanced disease, a multi-disciplinary approach is necessary to meet the complex requirements of a needs-adapted palliative care in inpatient or outpatient settings. Timely integration of palliative expertise may support symptom control and may give the required advice to patients, their carers, and their families.

Twelve tips for getting started using mixed methods in medical education research

BACKGROUND

Mixed methods research, which is gaining popularity in medical education, provides a new and comprehensive approach for addressing teaching, learning, and evaluation issues in the field.

AIM

The aim of this article is to provide medical education researchers with 12 tips, based on consideration of current literature in the health professions and in educational research, for conducting and disseminating mixed methods research.

CONCLUSION

Engaging in mixed methods research requires consideration of several major components: the mixed methods paradigm, types of problems, mixed method designs, collaboration, and developing or extending theory. Mixed methods is an ideal tool for addressing a full range of problems in medical education to include development of theory and improving practice.

Rethinking programme evaluation in health professions education: beyond 'did it work?'

CONTEXT

For nearly 40 years, outcome-based models have dominated programme evaluation in health professions education. However, there is increasing recognition that these models cannot address the complexities of the health professions context and studies employing alternative evaluation approaches that are appearing in the literature. A similar paradigm shift occurred over 50 years ago in the broader discipline of programme evaluation. Understanding the development of contemporary paradigms within this field provides important insights to support the evolution of programme evaluation in the health professions.

METHODS

In this discussion paper, we review the historical roots of programme evaluation as a discipline, demonstrating parallels with the dominant approach to evaluation in the health professions. In tracing the evolution of contemporary paradigms within this field, we demonstrate how their aim is not only to judge a programme's merit or worth, but also to generate information for curriculum designers seeking to adapt programmes to evolving contexts, and researchers seeking to generate knowledge to inform the work of others.

DISCUSSION

From this evolution, we distil seven essential elements of educational programmes that should be evaluated to achieve the stated goals. Our formulation is not a prescriptive method for conducting programme evaluation; rather, we use these elements as a guide for the development of a holistic 'programme of evaluation' that involves multiple stakeholders, uses a combination of available models and methods, and occurs throughout the life of a programme. Thus, these elements provide a roadmap for the programme evaluation process, which allows evaluators to move beyond asking whether a programme worked, to establishing how it worked, why it worked and what else happened. By engaging in this process, evaluators will generate a sound understanding of the relationships among programmes, the contexts in which they operate, and the outcomes that result from them.