Combining Theory-Driven Evaluation and Causal Loop Diagramming for Opening the 'Black Box' of an Intervention in the Health Sector: A Case of Performance-Based Financing in Western Uganda

Systems thinking methods: a worked example of supporting emergency medical services decision-makers to prioritize and contextually analyse potential interventions and their implementation

BACKGROUND

Systems thinking can be used as a participatory data collection and analysis tool to understand complex implementation contexts and their dynamics with interventions, and it can support the selection of tailored and effective implementation actions. A few previous studies have applied systems thinking methods, mainly causal loop diagrams, to prioritize interventions and to illustrate the respective implementation context. The present study aimed to explore how systems thinking methods can help decision-makers (1) understand locally specific causes and effects of a key issue and how they are interlinked, (2) identify the most relevant interventions and best fit in the system, and (3) prioritize potential interventions and contextually analyse the system and potential interventions.

METHODS

A case study approach was adopted in a regional emergency medical services (EMS) system in Germany. We applied systems thinking methods following three steps: (1) a causal loop diagram (CLD) with causes and effects (variables) of the key issue "rising EMS demand" was developed together with local decision-makers; (2) targeted interventions addressing the key issue were determined, and impacts and delays were used to identify best intervention variables to determine the system's best fit for implementation; (3) based on steps 1 and 2, interventions were prioritized and, based on a pathway analysis related to a sample intervention, contextually analysed.

RESULTS

Thirty-seven variables were identified in the CLD. All of them, except for the key issue, relate to one of five interlinked subsystems. Five variables were identified as best fit for implementing three potential interventions. Based on predicted implementation difficulty and effect, as well as delays and best intervention variables, interventions were prioritized. The pathway analysis on the example of implementing a standardized structured triage tool highlighted certain contextual factors (e.g. relevant stakeholders, organizations), delays and related feedback loops (e.g. staff resource finiteness) that help decision-makers to tailor the implementation.

CONCLUSIONS

Systems thinking methods can be used by local decision-makers to understand their local implementation context and assess its influence and dynamic connections to the implementation of a particular intervention, allowing them to develop tailored implementation and monitoring strategies.

Theory-based evaluation and programme theories in nursing: A discussion on the occasion of the updated Medical Research Council (MRC) Framework

Developing and evaluating health interventions for the benefit of patients is notoriously difficult. This also applies to the discipline of nursing, owing to the complexity of nursing interventions. Following significant revision, the updated guidance of the Medical Research Council (MRC) adopts a pluralistic view to intervention development and evaluation, including a theory-based perspective. This perspective promotes the use of program theory, aiming to understand how and under what circumstances interventions lead to change. In this discussion paper, we reflect the recommended use of program theory in the context of evaluation studies addressing complex nursing interventions. First, we review the literature by investigating the question whether and how evaluation studies targeting complex interventions used theory and to what extent program theories may contribute to enhance the theoretical foundations of intervention studies in nursing. Second, we illustrate the nature of theory-based evaluation and program theories. Third, we argue how this may impact theory building in nursing in general. We finish by discussing which resources, skills and competencies are necessary to fulfill the demanding task of undertaking theory-based evaluations. We caution against an oversimplified interpretation of the updated MRC guidance regarding the theory-based perspective, e.g. by using simple linear logic models, rather than articulating program theories. Instead, we encourage researchers to embrace the corresponding methodology, i.e. theory-based evaluation. With the prevailing perspective of knowledge production in crisis, we might be on the verge of a paradigm shift in health intervention research. Viewed through this lens, the updated MRC guidance could lead to a renewed understanding of what constitutes useful knowledge in nursing. This may facilitate knowledge production and, thereby, contribute to improve nursing practice for the benefit of the patient.

Long drives and red tape: mapping rural veteran access to primary care using causal-loop diagramming

Background

Rural veterans experience more challenges than their urban peers in accessing primary care services, which can negatively impact their health and wellbeing. The factors driving this disparity are complex and involve patient, clinic, health system, community and policy influences. Federal policies over the last decade have relaxed requirements for some veterans to receive primary care services from community providers through their VA benefits, known as community care.

Methods

We used a participatory systems mapping approach involving causal-loop diagramming to identify interrelationships between variables underlying challenges to veteran access to primary care and potential opportunities for change—known as leverage points in systems science. Our methods involved a secondary analysis of semi-structured qualitative interviews with rural veterans, VA staff, non-VA clinic staff and providers who serve rural veterans, and veteran service officers (VSOs) in the Northwest region of the US, followed by a two-part participatory modeling session with a study advisory board. We then applied Meadows’s leverage point framework to identify and categorize potential interventions to improve rural veteran access to primary care.

Results

The final model illustrated challenges at the veteran, clinic, and system levels as experienced by stakeholders. Main components of the diagram pertained to the choice of VA or non-VA primary care, veteran satisfaction with the VA, enrollment in VA benefits and other insurance, community care authorization, reimbursement of non-VA care, referrals to specialty care, record sharing and communication between VA and non-VA providers, institutional stability of the VA, and staffing challenges. Fourteen interventions, including administrative and communications changes, were identified by analyzing the model using the leverage points framework.

Conclusions

Our findings illustrate how challenges rural veterans face accessing health care are interconnected and persist despite recent changes to federal law pertaining to the VA health care system in recent years. Systems mapping and modeling approaches such as causal-loop diagramming have potential for engaging stakeholders and supporting intervention and implementation planning.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-022-08318-2.

How to do (or not to do)…Using Causal Loop Diagrams for Health System Research in Low- and Middle-Income Settings

Causal loop diagrams (CLDs) are a systems thinking method that can be used to visualize and unpack complex health system behaviour. They can be employed prospectively or retrospectively to identify the mechanisms and consequences of policies or interventions designed to strengthen health systems and inform discussion with policymakers and stakeholders on actions that may alleviate sub-optimal outcomes. Whilst the use of CLDs in health systems research has generally increased, there is still limited use in low- and middle-income settings. In addition to their suitability for evaluating complex systems, CLDs can be developed where opportunities for primary data collection may be limited (such as in humanitarian or conflict settings) and instead be formulated using secondary data, published or grey literature, health surveys/reports and policy documents. The purpose of this paper is to provide a step-by-step guide for designing a health system research study that uses CLDs as their chosen research method, with particular attention to issues of relevance to research in low- and middle-income countries (LMICs). The guidance draws on examples from the LMIC literature and authors’ own experience of using CLDs in this research area. This paper guides researchers in addressing the following four questions in the study design process; (1) What is the scope of this research? (2) What data do I need to collect or source? (3) What is my chosen method for CLD development? (4) How will I validate the CLD? In providing supporting information to readers on avenues for addressing these key design questions, authors hope to promote CLDs for wider use by health system researchers working in LMICs.

Participatory Causal Loop Mapping of the Adoption of Organic Farming in Nigeria

Contrary to the expectations of promoters of organic agriculture, the adoption of the technology by smallholder farmers in Africa has been low and slow, for reasons not well understood. Existing studies on the topic mostly estimated the effect of some variables on the adoption of the technology. But adoption is characterized by complex and dynamic interactions of many interconnected factors, which existing studies overlooked. The underlying causal structures and feedback mechanisms that dynamically interact to affect the adoption of organic farming in urban and rural Africa are also not well known. To bridge these gaps, we used a system dynamics tool called participatory causal loop diagraming to map the underlying causal factors and feedback mechanisms driving the adoption of organic farming in rural and urban Nigeria. We conducted loop and network analyses of the group causal loop diagrams, which were created during the participatory system dynamics modeling workshops with the organic farmers in our study areas. Our findings underscore the importance of the knowledge of organic farming, demand- and supply-side-oriented awareness creation, and the economic viability of organic farming for widespread adoption of the technology. We suggested the potential leverages around which interventions can be built to boost the adoption rates of the technology.

Advancing sustainable development goals through immunization: a literature review

BACKGROUND

Immunization directly impacts health (SDG3) and brings a contribution to 14 out of the 17 Sustainable Development Goals (SDGs), such as ending poverty, reducing hunger, and reducing inequalities. Therefore, immunization is recognized to play a central role in reaching the SDGs, especially in low- and middle-income countries (LMICs). Despite continuous interventions to strengthen immunization systems and to adequately respond to emergency immunization during epidemics, the immunization-related indicators for SDG3 lag behind in sub-Saharan Africa. Especially taking into account the current Covid19 pandemic, the current performance on the connected SDGs is both a cause and a result of this.

METHODS

We conduct a literature review through a keyword search strategy complemented with handpicking and snowballing from earlier reviews. After title and abstract screening, we conducted a qualitative analysis of key insights and categorized them according to showing the impact of immunization on SDGs, sustainability challenges, and model-based solutions to these challenges.

RESULTS

We reveal the leveraging mechanisms triggered by immunization and position them vis-à-vis the SDGs, within the framework of Public Health and Planetary Health. Several challenges for sustainable control of vaccine-preventable diseases are identified: access to immunization services, global vaccine availability to LMICs, context-dependent vaccine effectiveness, safe and affordable vaccines, local/regional vaccine production, public-private partnerships, and immunization capacity/capability building. Model-based approaches that support SDG-promoting interventions concerning immunization systems are analyzed in light of the strategic priorities of the Immunization Agenda 2030.

CONCLUSIONS

In general terms, it can be concluded that relevant future research requires (i) design for system resilience, (ii) transdisciplinary modeling, (iii) connecting interventions in immunization with SDG outcomes, (iv) designing interventions and their implementation simultaneously, (v) offering tailored solutions, and (vi) model coordination and integration of services and partnerships. The research and health community is called upon to join forces to activate existing knowledge, generate new insights and develop decision-supporting tools for Low-and Middle-Income Countries' health authorities and communities to leverage immunization in its transformational role toward successfully meeting the SDGs in 2030.

Understanding the maternal and child health system response to payment for performance in Tanzania using a causal loop diagram approach

Payment for performance (P4P) has been employed in low and middle-income (LMIC) countries to improve quality and coverage of maternal and child health (MCH) services. However, there is a lack of consensus on how P4P affects health systems. There is a need to evaluate P4P effects on health systems using methods suitable for evaluating complex systems. We developed a causal loop diagram (CLD) to further understand the pathways to impact of P4P on delivery and uptake of MCH services in Tanzania. The CLD was developed and validated using qualitative data from a process evaluation of a P4P scheme in Tanzania, with additional stakeholder dialogue sought to strengthen confidence in the diagram. The CLD maps the interacting mechanisms involved in provider achievement of targets, reporting of health information, and population care seeking, and identifies those mechanisms affected by P4P. For example, the availability of drugs and medical commodities impacts not only provider achievement of P4P targets but also demand of services and is impacted by P4P through the availability of additional facility resources and the incentivisation of district managers to reduce drug stock outs. The CLD also identifies mechanisms key to facility achievement of targets but are not within the scope of the programme; the activities of health facility governing committees and community health workers, for example, are key to demand stimulation and effective resource use at the facility level but both groups were omitted from the incentive system. P4P design considerations generated from this work include appropriately incentivising the availability of drugs and staffing in facilities and those responsible for demand creation in communities. Further research using CLDs to study heath systems in LMIC is urgently needed to further our understanding of how systems respond to interventions and how to strengthen systems to deliver better coverage and quality of care.

•

Holistic analysis key to avoiding suboptimal P4P performance.

•

Availability of drugs is influenced by P4P and affects success of P4P.

•

Need to incentivise groups outside facility that support service coverage.

How do participatory methods shape policy? Applying a realist approach to the formulation of a new tuberculosis policy in Georgia

OBJECTIVES

This paper presents the iterative process of participatory multistakeholder engagement that informed the development of a new national tuberculosis (TB) policy in Georgia, and the lessons learnt.

METHODS

Guided by realist evaluation methods, a multistakeholder dialogue was organised to elicit stakeholders' assumptions on challenges and possible solutions for better TB control. Two participatory workshops were conducted with key actors, interspersed by reflection meetings within the research team and discussions with policymakers. Using concept mapping and causal mapping techniques, and drawing causal loop diagrams, we visualised how actors understood TB service provision challenges and the potential means by which a results-based financing (RBF) policy could address these.

SETTING

The study was conducted in Tbilisi, Georgia.

PARTICIPANTS

A total of 64 key actors from the Ministry of Labour, Health and Social Affairs, staff of the Global Fund to Fight AIDS, TB and Malaria Georgia Project, the National Centre for Disease Control and Public Health, the National TB programme, TB service providers and members of the research team were involved in the workshops.

RESULTS

Findings showed that beyond provider incentives, additional policy components were necessary. These included broadening the incentive package to include institutional and organisational incentives, retraining service providers, clear redistribution of roles to support an integrated care model, and refinement of monitoring tools. Health system elements, such as effective referral systems and health information systems were highlighted as necessary for service improvement.

CONCLUSIONS

Developing policies that address complex issues requires methods that facilitate linkages between multiple stakeholders and between theory and practice. Such participatory approaches can be informed by realist evaluation principles and visually facilitated by causal loop diagrams. This approach allowed us leverage stakeholders' knowledge and expertise on TB service delivery and RBF to codesign a new policy.

Does decentralization of health systems translate into decentralization of authority? A decision space analysis of Ugandan healthcare facilities

Since the 1990s, following similar reforms to its general politico-administrative systems, Uganda has decentralized its public healthcare system by shifting decision-making power away from its central Ministry of Health and towards more distal administrative levels. Previous research has used decision space—the decision-making autonomy demonstrated by entities in an administrative hierarchy—to measure overall health system decentralization. This study aimed to determine how the decision-making autonomy reported by managers of Ugandan healthcare facilities (de facto decision space) differs from that which they are allocated by official policies (de jure decision space). Additionally, it sought to determine associations between decision space and indicators of managerial performance. Using quantitative primary healthcare data from Ugandan healthcare facilities, our study determined the decision space expressed by facility managers and the performance of their facilities on measures of essential drug availability, quality improvement and performance management. We found managers reported greater facility-level autonomy than expected in disciplining staff compared with recruitment and promotion, suggesting that managerial functions that require less financial or logistical investment (i.e. discipline) may be more susceptible to differences in de jure and de facto decision space than those that necessitate greater investment (i.e. recruitment and promotion). Additionally, we found larger public health facilities expressed significantly greater facility-level autonomy in drug ordering compared with smaller facilities, which indicates ongoing changes in the Ugandan medical supply chain to a hybrid ‘push-pull’ system. Finally, we found increased decision space was significantly positively associated with some managerial performance indicators, such as essential drug availability, but not others, such as our performance management and quality improvement measures. We conclude that increasing managerial autonomy alone is not sufficient for improving overall health facility performance and that many factors, specific to individual managerial functions, mediate relationships between decision space and performance.

Environment, Business, and Health Care Prevail: A Comprehensive, Systematic Review of System Dynamics Application Domains

System dynamics, as a methodology for analyzing and understanding various types of systems, has been applied in research for several decades. We undertook a review to identify the latest application domains and map the realm of system dynamics. The systematic review was conducted according to the PRISMA methodology. We analyzed and categorized 212 articles and found that the vast majority of studies belong to the fields of business administration, health, and environmental research. Altogether, 20 groups of modeling and simulation topics can be recognized. System dynamics is occasionally supported by other modeling methodologies such as the agent-based modeling approach. There are issues related to published studies mostly associated with testing of validity and reasonability of models, leading to the development of predictions that are not grounded in verified models. This study contributes to the development of system dynamics as a methodology that can offer new ideas, highlight limitations, or provide analogies for further research in various research disciplines.

Mechanisms of scaling up: combining a realist perspective and systems analysis to understand successfully scaled interventions

Background

Sustainable shifts in population behaviours require system-level implementation and embeddedness of large-scale health interventions. This paper aims to understand how different contexts of scaling up interventions affect mechanisms to produce intended and unintended scale up outcomes.

Methods

A mixed method study combining a realist perspective and systems analysis (causal loop diagrams) of scaled-up physical activity and/or nutrition interventions implemented at a state/national level in Australia (2010–18). The study involved four distinct phases: Phase 1 expert consultation, database and grey literature searches to identify scaled-up interventions; Phase 2 generating initial Context-Mechanism-Outcome configurations (CMOs) from the WHO ExpandNet framework for scaling up; Phase 3 testing and refining CMOs via online surveys and realist interviews with academics, government and non-government organisations (NGOs) involved in scale up of selected interventions (Phase 1); and Phase 4 generating cross-case mid-range theories represented in systems models of scaling up; validated by member checking. Descriptive statistics were reported for online survey data and realist analysis for interview data.

Results

Seven interventions were analysed, targeting nutrition (n = 1), physical activity (n = 1), or a combination (n = 5). Twenty-six participants completed surveys; 19 completed interviews. Sixty-three CMO pathways underpinned successful scale up, reflecting 36 scale up contexts, 8 key outcomes; linked via 53 commonly occurring mechanisms. All five WHO framework domains were represented in the systems models. Most CMO pathways included ‘intervention attributes’ and led to outcomes ‘community sustainability/embeddedness’ and ‘stakeholder buy-in/perceived value’. Irrespective of interventions being scaled in similar contexts (e.g., having political favourability); mechanisms still led to both intended and unintended scale up outcomes (e.g., increased or reduced sustainability).

Conclusion

This paper provides the first evidence for mechanisms underpinning outcomes required for successful scale up of state or nationally delivered interventions. Our findings challenge current prerequisites for effective scaling suggesting other conditions may be necessary. Future scale up approaches that plan for complexity and encourage iterative adaptation throughout, may enhance scale up outcomes. Current linear, context-to-outcome depictions of scale up oversimplify what is a clearly a complex interaction between perceptions, worldviews and goals of those involved. Mechanisms identified in this study could potentially be leveraged during future scale up efforts, to positively influence intervention scalability and sustainability.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12966-021-01103-0.

What theories underpin performance-based financing? A scoping review

PURPOSE

The study aims to explore the theoretical bases justifying the use of performance-based financing (PBF) in the health sector in low- and middle-income countries (LMICs).

DESIGN/METHODOLOGY/APPROACH

The authors conducted a scoping review of the literature on PBF so as to identify the theories utilized to underpin it and analyzed its theoretical justifications.

FINDINGS

Sixty-four studies met the inclusion criteria. Economic theories were predominant, with the principal-agent theory being the most commonly-used theory, explicitly referred to by two-thirds of included studies. Psychological theories were also common, with a wide array of motivation theories. Other disciplines in the form of management or organizational science, political and social science and systems approaches also contributed. However, some of the theories referred to contradicted each other. Many of the studies included only casually alluded to one or more theories, and very few used these theories to justify or support PBF. No theory emerged as a dominant, consistent and credible justification of PBF, perhaps except for the principal-agent theory, which was often inappropriately applied in the included studies, and when it included additional assumptions reflecting the contexts of the health sector in LMICs, might actually warn against adopting PBF.

PRACTICAL IMPLICATIONS

Overall, this review has not been able to identify a comprehensive, credible, consistent, theoretical justification for using PBF rather than alternative approaches to health system reforms and healthcare providers' motivation in LMICs.

ORIGINALITY/VALUE

The theoretical justifications of PBF in the health sector in LMICs are under-documented. This review is the first of this kind and should encourage further debate and theoretical exploration of the justifications of PBF.

Using implementation science theories and frameworks in global health

In global health, researchers and decision makers, many of whom have medical, epidemiology or biostatistics background, are increasingly interested in evaluating the implementation of health interventions. Implementation science, particularly for the study of public policies, has existed since at least the 1930s. This science makes compelling use of explicit theories and analytic frameworks that ensure research quality and rigour. Our objective is to inform researchers and decision makers who are not familiar with this research branch about these theories and analytic frameworks. We define four models of causation used in implementation science: intervention theory, frameworks, middle-range theory and grand theory. We then explain how scientists apply these models for three main implementation studies: fidelity assessment, process evaluation and complex evaluation. For each study, we provide concrete examples from research in Cuba and Africa to better understand the implementation of health interventions in global health context. Global health researchers and decision makers with a quantitative background will not become implementation scientists after reading this article. However, we believe they will be more aware of the need for rigorous implementation evaluations of global health interventions, alongside impact evaluations, and in collaboration with social scientists.

At what cost is performance-based financing implemented? Novel evidence from Malawi

Our study estimated the full economic cost of implementing performance-based financing [PBF, the Support for Service Delivery Integration Performance-Based Incentives (SSDI-PBI) programme], as a means of first introducing strategic purchasing in a low-income setting, Malawi. Our analysis distinguished design from implementation costs and traces costs across personnel and non-personnel cost categories over the 2012-15 period. The full cost of the SSDI-PBI programme amounted to USD 3 402 187, equivalent to USD 6.46 per targeted beneficiary. The design phase accounted for about one-third (USD 1 161 332) of the total costs, while the incentives (USD 1 140 436) represented about one-third of the total cost of the intervention and about half the cost of the implementation phase. With a cost of USD 1 605 178, personnel costs represented the dominant cost category. Our study indicated that the introduction of PBF entailed consumption of a substantial amount of resources, hence representing an important opportunity cost for the health system.

Effects in spite of tough constraints - A theory of change based investigation of contextual and implementation factors affecting the results of a performance based financing scheme extended to malnutrition in Burundi

BACKGROUND

From January 2015 to December 2016, the health authorities in Burundi piloted the inclusion of child nutrition services into the pre-existing performance-based financing free health care policy (PBF-FHC). An impact evaluation, focused on health centres, found positive effects both in terms of volume of services and quality of care. To some extent, this result is puzzling given the harshness of the contextual constraints related to the fragile setting.

METHODS

With a multi-methods approach, we explored how contextual and implementation constraints interacted with the pre-identified tracks of effect transmission embodied in the intervention. For our analysis, we used a hypothetical Theory of Change (ToC) that mapped a set of seven tracks through which the intervention might develop positive effects for children suffering from malnutrition. We built our analysis on (1) findings from the facility surveys and (2) extra qualitative data (logbooks, interviews and operational document reviews).

FINDINGS

Our results suggest that six constraints have weighted upon the intervention: (1) initial low skills of health workers; (2) unavailability of resources (including nutritional dietary inputs and equipment); (3) payment delays; (4) suboptimal information; (5) restrictions on autonomy; and (6) low intensity of supervision. Together, they have affected the intensity of the intervention, especially during its first year. From our analysis of the ToC, we noted that the positive effects largely occurred as a result of the incentive and information tracks. Qualitative data suggests that health centres have circumvented the many constraints by relying on a community-based recruitment strategy and a better management of inputs at the level of the facility and the patient himself.

CONCLUSION

Frontline actors have agency: when incentives are right, they take the initiative and find solutions. However, they cannot perform miracles: Burundi needs a holistic societal strategy to resolve the structural problem of child malnutrition.

TRIAL REGISTRATION

Clinical Trials.gov Identifier: NCT02721160; March 2016 (retrospectively registered).

Determinants of effective organisational capacity training: lessons from a training programme on health workforce development with participants from three African countries

BACKGROUND

Health systems in sub-Saharan Africa face multifaceted capacity challenges to fulfil their mandates of service provision and governance of their resources. Four academic institutions in Africa implemented a World Health Organisation-funded collaborative project encompassing training, curriculum development, and partnership to strengthen national leadership and training capacity for health workforce development. This paper looks into the training component of the project, a blended Masters programme in public health that sought to improve the capacity of personnel involved in teaching or management/development of human resources for health. The paper aims to explore factors influencing contribution of training to organisational capacity development.

METHODS

We chose a case study design. Semi-structured interviews were held with 18 trainees that were enrolled in the training programme, and who were affiliated to health ministries or public health training institutions. We gathered additional data through document reviews, observation, and interviews with 14 key informants associated with the programme and/or working in the collaborating institutions. The evidence gathered were analysed thematically.

RESULTS

Thirteen of the 18 training participants stayed in the target institutions and contributed to improved capacity of their institutions in the fields of management, policy, planning, research, training, or curriculum development. Five left for private and international agencies due to dissatisfaction with payment, work conditions, or career prospect. Factors that were associated with the training, trainees, and the institutional and broader context, determine contribution of training to organisational capacity development. These include relevance of newly acquired knowledge and skills set of trainees to the role/position they assume in the organisation; recognition of trainees by employing organisations in terms of promotion or assignment of challenging tasks; and motivation and retention of trained staff.

CONCLUSION

Training, even if relevant and applicable, makes no more than a 'latent' contribution, one which is activated and realised through alignment of clusters of interacting contextual and relational factors related to the target institutions and trained personnel. While not predictable, implementers need to focus more deliberately on the likely interaction and best possible alignments between training relevance, student selection for potential to contribute, recognition and career advancement potential.

Is quality affordable for community health systems? Costs of integrating quality improvement into close-to-community health programmes in five low-income and middle-income countries

Introduction

Countries aspiring to universal health coverage view close-to-community (CTC) providers as a low-cost means of increasing coverage. However, due to lack of coordination and unreliable funding, the quality of large-scale CTC healthcare provision is highly variable and routine data about service quality are not trustworthy. Quality improvement (QI) approaches are a means of addressing these issues, yet neither the costs nor the budget impact of integrating QI approaches into CTC programme costs have been assessed.

Methods

This paper examines the costs and budget impact of integrating QI into existing CTC health programmes in five countries (Ethiopia, Indonesia, Kenya, Malawi, Mozambique) between 2015 and 2017. The intervention involved: (1) QI team formation; (2) Phased training interspersed with supportive supervision; which resulted in (3) QI teams independently collecting and analysing data to conduct QI interventions. Project costs were collected using an ingredients approach from a health systems perspective. Based on project costs, costs of local adoption of the intervention were modelled under three implementation scenarios.

Results

Annualised economic unit costs ranged from $62 in Mozambique to $254 in Ethiopia per CTC provider supervised, driven by the context, type of community health model and the intensity of the intervention. The budget impact of Ministry-led QI for community health is estimated at 0.53% or less of the general government expenditure on health in all countries (and below 0.03% in three of the five countries).

Conclusion

CTC provision is a key component of healthcare delivery in many settings, so QI has huge potential impact. The impact is difficult to establish conclusively, but as a first step we have provided evidence to assess affordability of QI for community health. Further research is needed to assess whether QI can achieve the level of benefits that would justify the required investment.

Effect of results-based financing on facility-based maternal mortality at birth: an interrupted time-series analysis with independent controls in Malawi

INTRODUCTION

The aim of this study was to assess the impact of a results-based financing (RBF) programme on the reduction of facility-based maternal mortality at birth. Malawi is a low-income country with high maternal mortality. The Results-Based Financing For Maternal and Newborn Health (RBF4MNH) Initiative was introduced at obstetric care facilities in four districts to improve quality and utilisation of maternal and newborn health services. The RBF4MNH Initiative was launched in April 2013 as a combined supply-side and demand-side RBF. Programme expansion occurred in October 2014.

METHODS

Controlled interrupted time series was used to estimate the effect of the RBF4MNH on reducing facility-based maternal mortality at birth. The study sample consisted of all obstetric care facilities in 4 intervention and 19 control districts, which constituted all non-urban mainland districts in Malawi. Data for obstetric care facilities were extracted from the Malawi Health Management Information System. Facility-based maternal mortality at birth was calculated as the number of maternal deaths per all deliveries at a facility in a given time period.

RESULTS

The RBF4MNH effectively reduced facility-based maternal mortality by 4.8 (-10.3 to 0.7, p<0.1) maternal deaths/100 000 facility-based deliveries/month after reaching full operational capacity in October 2014. Immediate effects (changes in level rather than slope) attributable to the RBF4MNH were not statistically significant.

CONCLUSION

This is the first study evaluating the effect of a combined supply-side and demand-side RBF on maternal mortality outcomes and demonstrates the positive role financial incentives can play in improving health outcomes. This study further shows that timeframes spanning several years might be necessary to fully evaluate the impact of health-financing programmes on health outcomes. Further research is needed to assess the extent to which the observed reduction in facility-based mortality at birth contributes to all-cause maternal mortality in the country.

The Importance of Leadership and Organizational Capacity in Shaping Health Workers' Motivational Reactions to Performance-Based Financing: A Multiple Case Study in Burkina Faso

BACKGROUND

Performance-based financing (PBF) is currently tested in many low- and middle-income countries as a health system strengthening strategy. One of the main mechanisms through which PBF is assumed to effect change is by motivating health workers to improve their service delivery performance. This article aims at a better understanding of such motivational effects of PBF. In particular, the study focused on organizational context factors and health workers' perceptions thereof as moderators of the motivational effects of PBF, which to date has been little explored.

METHODS

We conducted a multiple case study in 2 district hospitals and 16 primary health facilities across three districts. Health facilities were purposely sampled according to pre-PBF performance levels. Within sampled facilities, 82 clinical skilled healthcare workers were in-depth interviewed one year after the start of the PBF intervention. Data were analyzed using a blended deductive and inductive process, using self-determination theory (SDT) as an analytical framework.

RESULTS

Results show that the extent to which PBF contributed to positive, sustainable forms of motivation depended on the "ground upon which PBF fell," beyond health workers' individual personalities and disposition. In particular, health workers described three aspects of the organizational context in which PBF was implemented: the extent to which existing hierarchies fostered as opposed to hindered participation and transparency; managers' handling of the increased performance feedback inherent in PBF; and facility's pre-PBF levels in regards to infrastructure, equipment, and human resources.

CONCLUSION

Our results underline the importance of leadership styles and pre-implementation performance levels in shaping health workers' motivational reactions to PBF. Ancillary interventions aimed at fostering participatory as opposed to directional leadership or start-up support to low-performing health facilities will likely boost PBF effects in regards to the development of valuable motivational capacities.

Methods to improve quality performance at scale in lower- and middle-income countries

Universal Health Coverage is one of the Sustainable Development Goal targets. But coverage without quality health services limits benefits to populations. Performance-based financing programs (PBF) use strategic purchasing of services to expand coverage and promote quality by measuring quality and rewarding good performance. The widespread presence of PBF programs in lower and middle-income countries provide an opportunity to introduce and test new approaches for measuring and improving quality at scale. This article describes four approaches to improve quality of health services at scale in PBF programs. These approaches looked at structural and process measures of quality as well as outcome measures like patient satisfaction. Three types of tools were used in these approaches: clinical vignettes, competency tests and patient satisfaction surveys. Specific tools within each of the approaches are used in Kyrgyzstan, Cambodia, Democratic Republic of Congo and the Republic of Congo.

Performance-based financing in low-income and middle-income countries: isn't it time for a rethink?

This paper questions the view that performance-based financing (PBF) in the health sector is an effective, efficient and equitable approach to improving the performance of health systems in low-income and middle-income countries (LMICs). PBF was conceived as an open approach adapted to specific country needs, having the potential to foster system-wide reforms. However, as with many strategies and tools, there is a gap between what was planned and what is actually implemented. This paper argues that PBF as it is currently implemented in many contexts does not satisfy the promises. First, since the start of PBF implementation in LMICs, concerns have been raised on the basis of empirical evidence from different settings and disciplines that indicated the risks, cost and perverse effects. However, PBF implementation was rushed despite insufficient evidence of its effectiveness. Second, there is a lack of domestic ownership of PBF. Considering the amounts of time and money it now absorbs, and the lack of evidence of effectiveness and efficiency, PBF can be characterised as a donor fad. Third, by presenting itself as a comprehensive approach that makes it possible to address all aspects of the health system in any context, PBF monopolises attention and focuses policy dialogue on the short-term results of PBF programmes while diverting attention and resources from broader processes of change and necessary reforms. Too little care is given to system-wide and long-term effects, so that PBF can actually damage health services and systems. This paper ends by proposing entry points for alternative approaches.