Crossing the digital divide: The workload of manual data entry for integration between mobile health applications and eHealth infrastructure

Background

Many digital health interventions (DHIs), including mobile health (mHealth) apps, aim to improve both client outcomes and efficiency like electronic medical record systems (EMRS). Although interoperability is the gold standard, it is also complex and costly, requiring technical expertise, stakeholder permissions, and sustained funding. Manual data linkage processes are commonly used to "integrate" across systems and allow for assessment of DHI impact, a best practice, before further investment. For mHealth, the manual data linkage workload, including related monitoring and evaluation (M&E) activities, remains poorly understood.

Methodology

As a baseline study for an open-source app to mirror EMRS and reduce healthcare worker (HCW) workload while improving care in the Nurse-led Community-based Antiretroviral therapy Program (NCAP) in Lilongwe, Malawi, we conducted a time-motion study observing HCWs completing data management activities, including routine M&E and manual data linkage of individual-level app data to EMRS. Data management tasks should reduce or end with successful app implementation and EMRS integration. Data was analysed in Excel.

Results

We observed 69:53:00 of HCWs performing routine NCAP service delivery tasks: 39:52:00 (57%) was spent completing M&E data related tasks of which 15:57:00 (23%) was spent on manual data linkage workload, alone.

Conclusion

Understanding the workload to ensure quality M&E data, including to complete manual data linkage of mHealth apps to EMRS, provides stakeholders with inputs to drive DHI innovations and integration decision making. Quantifying potential mHealth benefits on more efficient, high-quality M&E data may trigger new innovations to reduce workloads and strengthen evidence to spur continuous improvement.

Kenya's Experience: Factors Enabling and Impeding the COVID-19 Response

This case study describes the country-level response to the COVID-19 pandemic in Kenya between February 2020 and May 2021. We organize the presentation of COVID-19 response strategies across the five stages of (a) engagement, (b) assessment, (c) planning, (d) action/implementation, and (e) evaluation. We describe the participatory monitoring and evaluation (M&E) process implemented in collaboration with the WHO Regional Office for Africa Monitoring and Evaluation Team. The M&E system was used to organize and make sense of emerging data regarding specific response activities and changing COVID incidence. We share the results of that collaborative sensemaking, with particular attention to our analysis of the factors that facilitated and those that impeded our pandemic response. We conclude with lessons learned and practical implications from Kenya's experience to help guide future country-level responses to rapidly changing public health crises.

Lessons Learned From Applying a Monitoring and Evaluation Framework to Economic, Social, and Other Health Impacts of the COVID-19 Pandemic

Individual and community-level COVID-19 mitigation policies can have effects beyond direct COVID-19 health outcomes, including social, behavioral, and economic outcomes. These social, behavioral, and economic outcomes can extend beyond the pandemic period and have disparate effects on populations. Public Health-Seattle & King County (PHSKC) built on the Centers for Disease Control and Prevention's community mitigation strategy framework to create a local project tracking near-real-time data to understand factors affected by mitigation approaches, inform decision-making, and monitor and evaluate community-level disparities during the pandemic. This case study describes the framework and lessons learned from PHSKC's collation, use, and dissemination of local data from 20 data sources to guide community and public health decision-making. Social, behavioral, economic, and health indicators were regularly updated and disseminated through interactive dashboards and products that examined data in the context of applicable policies. Data disaggregated by demographic characteristics and geography highlighted inequities, but not all datasets contained the same details; local surveys or qualitative data were used to fill gaps. Project outcomes included informing city and county emergency response planning related to implementation of financial and food assistance programs. Key lessons learned included the need to (1) build on existing processes and use automated processes and (2) partner with other sectors to use nontraditional public health data for active dissemination and data disaggregation and for real-time data contextualized by policy changes. This project provided programs and communities with timely, reliable data to understand where to invest recovery funding. A similar framework could position other health departments to examine social and economic effects during future public health emergencies.

A Community-Based, Mobile Electronic Medical Record System App for High-Quality, Integrated Antiretroviral Therapy in Lilongwe, Malawi: Design Process and Pilot Implementation

BACKGROUND

Differentiated service delivery (DSD) increases antiretroviral therapy (ART) access in sub-Saharan Africa by moving clients out of congested ART clinics to communities for care. However, DSD settings challenge provider adherence to complex, chronic care treatment guidelines and have burdensome systems for client monitoring and evaluation (M&E), reducing data for decision-making. Electronic medical record systems (EMRS) improve client outcomes and reduce M&E workload. Traditional EMRS cannot operate in most DSD settings with unreliable power and poor connectivity.

OBJECTIVE

This study aims to detail the human-centered design (HCD) process of developing a mobile EMRS for community-based DSD services in Lilongwe, Malawi.

METHODS

Lighthouse Trust (LT) operates 2 Ministry of Health (MoH) clinics in Lilongwe, Malawi, with a combined total of >35,000 ART clients. LT's real-time, point-of-care EMRS collects complex client M&E data and provides decision-making support, ensuring adherence to integrated HIV and tuberculosis guidelines that optimize client and program outcomes. LT's EMRS scaled to all large MoH ART clinics. LT also implements a nurse-led community-based ART program (NCAP), a DSD model to provide ART and rapid assessment for 2400 stable LT clients in the community. LT, alongside collaborators, from the University of Washington's International Training and Education Center for Health and technology partner, Medic, used the open-source Community Health Toolkit (CHT) and HCD to develop an open-source, offline-first, mobile EMRS-like app, "community-based ART retention and suppression" (CARES). CARES aims to bring EMRS-like provider benefits to NCAP's DSD clients.

RESULTS

CARES design took approximately 12 months and used an iterative process of highly participatory feedback sessions with provider, data manager, and M&E team inputs to ensure CARES optimization for the NCAP and LT settings. The CARES mobile EMRS prototype supports NCAP providers with embedded prompts and alerts to ensure adherence to integrated MoH ART guidelines, aiming to improve the quality of client care. CARES facilitates improved data quality and flow for NCAP M&E, aiming to reduce data gaps between community and clinic settings. The CARES pilot demonstrates the potential of a mobile, point-of-care EMRS-like app that could benefit NCAP clients, providers, and program teams with integrated client care and complete M&E data for decision-making. CARES challenges include app speed, search features to align longitudinal records, and CARES to EMRS integration that supports timely care alerts.

CONCLUSIONS

Leveraging the CHT and HCD processes facilitated the design of a locally specified and optimized mobile app with the promise to bring EMRS-like benefits to DSD settings. Moving from the CARES prototype to routine NCAP implementation should result in improved client care and strengthened M&E while reducing workload. Our transparent and descriptive process shares the progress and pitfalls of the CARES design and development, helping others in this digital innovation area to learn from our experiences at this stage.

Monitoring healthy ageing for the next decade: South Korea's perspective

South Korea is the fastest ageing country among OECD countries. Unlike the older generation growing up in the aftermath of the Korean war, the first and second baby boomer generations have heightened expectations regarding public services. In addition to the demand in higher quality of both social and health services by these newer older population, there is a concomitant increased quantitative demand. It is imperative that Korea reimagines their health, social welfare and economic policies to reflect the rapidly changing needs of such generations. One way to do this is to mainstream and continually monitor healthy ageing in all aspects of future policies. In 2021, the Korean Longitudinal Healthy Aging Study was launched in this context, to better understand the needs of the new-older age generation and to produce evidence to support formulation of better tailored policies that could promote healthy ageing. However, Korea is only in its early stage in developing a monitoring system that looks into the performance level of policies that support healthy ageing. As a country that is preparing for such rapid demographic transition and has already commenced developing its healthy ageing indicators, it will be important to assess and monitor uniformly the level of healthy ageing from the framework perspective of WHO. Korea welcomes WHO's development of an internationally applicable M&E framework for healthy ageing. We hope that WHO's M&E framework on healthy ageing will help Korea align to the international standards in its journey through the UN Decade of Healthy Ageing 2021-2030 and beyond.

Reflections on COVID-19 and internal evaluation in a humanitarian non-profit

This article uses reflective practice and social interdependence theory to unpack the ways in which our practice as internal evaluators was impacted by the COVID-19 pandemic, using the development of a monitoring and evaluation framework as a case study. The reflections are separated into six stages: description, feelings, evaluation, analysis, conclusion and action plan. Initial reflections on the impact of COVID-19 drew out the negative effects of the pandemic and associated restrictions, which limited our ability to build rapport and stretched our capacity to balance work and home commitments. Deeper investigation revealed that the disaster pushed us to develop new ways of working that will augment and improve our future efforts. We foresee that these learnings will enable a future with greater ability to offer hybrid online/face-to-face collaborative opportunities that will enhance inclusion and active participation, thereby promoting monitoring and evaluation with greater collective ownership and relevance to a wider audience.

Rapid-cycle evaluation and learning for the effective delivery of integrated interventions in early childhood in rural India

Background

Effective and real-time data analytics plays an essential role in understanding gaps and improving the quality and coverage of complex public health interventions. Studies of public health information systems identify problems with data quality, such as incomplete records and untimely reporting. Effective data collection and real-time analysis systems for rapid-cycle learning are necessary to monitor public health programs and take timely evidence-based decisions. Early childhood development (ECD) programs are very diverse. Rapid-cycle evaluation and learning (REAL) guides the implementation process of such complex interventions in real time. Stepping stones was one such early childhood development program implemented in Central India.

Objective

The objective was to improve the delivery of complex, integrated public health interventions for early childhood development in remote areas of rural India.

Methodology

The program was developed according to the principles of inclusion and community-centeredness, which can be tested quickly and iteratively. To enhance the decision-making process and improve delivery and coverage, the core team implemented an information system for rapid-cycle learning. We developed performance indicators and a performance measurement matrix after defining the specific needs. Following that, we trained staff to collect complete data using electronic data collection tools and transfer it the same day to the server for quality review and further analysis. A variety of data/information was triangulated to address the gaps in intervention delivery, and those decisions were subsequently implemented.

Results

We observed that the quality of data collection improved, and errors were reduced by 50% in the third quarter. The quality of the narrative was also enhanced; it became more elaborate and reflective. Sharing their field output in meetings and improving the quality of the narrative enhanced the self-reflection skills of field staff and consequently improved the quality of the intervention delivery. Refresher training and mentoring by supervisors helped to improve the data quality over time.

Conclusion

Rapid-cycle evaluation and learning (REAL) can be implemented in resource-limited settings to improve the quality and coverage of integrated intervention in early childhood. It nurtures a reinforcing ecosystem that integrates providers, community, and family perspectives and guides interactions among stakeholders by integrating data from all available sources.

Development of Monitoring and Evaluation Systems in Four National Programs Addressing Mother and Child Health in Cote d'Ivoire: Qualitative Analysis of the Emergence and Formulation Process

Background

Monitoring and evaluation were introduced into the management of national health programs to ensure that results were attained, and that donors' funds were used transparently. This study aims to describe the process of the emergence and formulation of monitoring and evaluation (M&E) systems in national programs addressing maternal and child health in Cote d'Ivoire.

Methods

We conducted a multilevel case study combining a qualitative investigation and a literature review. This study took place in the city of Abidjan, where in-depth interviews were conducted with twenty-four (24) former officials who served at the central level of the health system and with six (06) employees from the technical and financial partners' agencies. A total of 31 interviews were conducted from January 10 to April 20, 2020. Data analysis was conducted according to the Kingdon conceptual framework modified by Lemieux and adapted by Ridde.

Results

The introduction of M&E in national health programs was due to the will of the technical and financial partners and the political and technical decision-makers at the central level of the national health system, who were concerned with accountability and convincing results in these programs. However, its formulation through a top-down approach was sketchy and lacked content to guide its implementation and future evaluation in the absence of national expertise in M&E.

Conclusion

The emergence of M&E systems in national health programs was originally endogenous and exogenous but strongly recommended by donors. Its formulation in the context of limited national expertise was marked by the absence of standards and guidelines that could codify the development of robust M&E systems.

HIV self-test reporting using mHealth platforms: A pilot study in Johannesburg, South Africa

Background

The main impediment to operational scale-up of HIV self-testing (HIVST) and counselling, is a dearth of information on utilisation, reporting, and linkage to care for HIV-positive individuals. To inform solutions to this issue, this study investigated the utility of self-testers reporting their results using a mobile-health (mHealth) platform, and whether seropositive users linked into care.

Method

Candidates who met the recruitment criteria across multiple sites within inner-city Johannesburg each received an HIVST kit. Using short message service (SMS) reminders (50% standard and 50% behavioural science), participants were prompted to self-report results on provided platforms. On the seventh day, users who did not make contact, were called, and surveyed via an interactive voice response system (IVRS). Multivariable regression was used in reporting by age and sex.

Results

Of the 9,505 participants, 2,467 (25.9%) participants answered any survey question, and of those, 1,933 (78.4%) were willing to self-report their HIV status. Men were more likely than women to make an inbound call (10.2% vs. 9.1%, p = 0.06) however, women were significantly more likely to self-report their test result (AOR = 1.12, 95%CI = 1.01-1.24, p = 0.025). Overall, self-reporting a test result was predicted by being younger and female. In addition, reporting HIV results was associated with age, 25-35 (AOR = 1.58, 95% CI = 1.24-2.02) and above 35 years (AOR = 2.12, 95% CI = 1.61-2.80). Out of 1,933 participants willing to report their HIV status, 314 reported a positive test, indicating a HIV prevalence of 16.2% (95% CI: 14.6%-18.0%) and of those 204 (65.0%) reported inclination to link to care.

Conclusion

While self-reporting HIVST results via an IVRS system yielded a higher response rate, behavioural SMSs were ineffective in increasing self-reporting.

Capacity building among nursing and midwifery professional associations in East Africa

Background

Despite improvements in educational opportunities, policy changes, and pay raises in the nursing and midwifery professions in East Africa, poor working conditions, few professional development opportunities, and a general lack of respect for these professions predominate. These issues contribute to a low quality of care among a population with a high burden of communicable diseases. Health professional associations may help to address these challenges by providing a voice for nurses and midwives.

Objective

This study evaluated the impact of a 5-year programme focused on strengthening nurses’ and midwives’ professional associations in East Africa.

Methods

This study used a mixed methods design. Quantitative, cross-sectional descriptive data were captured via surveys (n = 1,266) distributed to association members. In-depth interviews (n = 65) were used to obtain qualitative data and complement the survey responses. Quantitative and qualitative data collection occurred concurrently. The results were compared to assess the impact of the programme across Uganda, Kenya, and Tanzania.

Results

The programme successfully built capacity in four of five organisational capacity areas: resource mobilisation, financial management, strategy, and monitoring and evaluation. Marketing and communications, the fifth targeted area, did not show improvement. Capacity in both research and service delivery was also improved, despite the programme not providing training in these areas. In addition, collaboration among associations and their members was improved.

Conclusion

These results support existing evidence on the impact of capacity building among professional nursing associations and coincide with the World Health Organization’s objectives for nursing. Future capacity building programmes should consider replicating the successful activities from this programme and investigate ways to reach more rural branches and provide tailored content. This study contributes to a small but growing body of knowledge that supports capacity building among the African health workforce.

Partnering with Aboriginal and Torres Strait Islander Peoples: An Evaluation Study Protocol to Strengthen a Comprehensive Multi-Scale Evaluation Framework for Participatory Systems Modelling through Indigenous Paradigms and Methodologies

The social and emotional wellbeing of young Aboriginal and Torres Strait Islander peoples should be supported through an Indigenous-led and community empowering approach. Applying systems thinking via participatory approaches is aligned with Aboriginal and Torres Strait Islander research paradigms and can be an effective method to deliver a decision support tool for mental health systems planning for Indigenous communities. Evaluations are necessary to understand the effectiveness and value of such methods, uncover protective and healing factors of social and emotional wellbeing, as well as to promote Aboriginal and Torres Strait Islander self-determination over allocation of funding and resources. This paper presents modifications to a published evaluation protocol for participatory systems modelling to align with critical Aboriginal and Torres Strait Islander guidelines and recommendations to support the social and emotional wellbeing of young people. This paper also presents a culturally relevant participatory systems modelling evaluation framework. Recognizing the reciprocity, strengths, and expertise Aboriginal and Torres Strait Islander methodologies can offer to broader research and evaluation practices, the amended framework presented in this paper facilitates empowering evaluation practices that should be adopted when working with Aboriginal and Torres Strait Islander peoples as well as when working with other diverse, non-Indigenous communities.

Participatory Monitoring and Evaluation of the COVID-19 Response in the Africa Region

As the COVID-19 pandemic swept throughout the world, it created a demand for information to help understand the public health response and its effects. Limited capacity to see and interpret data-"sensemaking" with measures of progress-affects the use of data for quality improvement. The World Health Organization Regional Office for Africa (WHO AFRO) supported partners from the Member States in using a participatory monitoring and evaluation system to document and systematically reflect on the COVID-19 response at the country level. The WHO AFRO's COVID-19 Response Monitoring and Evaluation (M&E) team captured and communicated response activities based on available reports from 35 of the 47 member countries. By reviewing reports and communications, the M&E team documented nearly 8,000 COVID-19 response activities during the study period (January 2020 through July 2021). A "sensemaking" protocol was used to support country partners in identifying factors associated with increases or decreases in both new cases and response activities. This report describes this participatory M&E approach and process of shared sensemaking. We illustrate with a country-level case study of the COVID-19 response in the Africa Region.

A mixed methodology for evaluating use of evidence in conservation planning

Conservation practitioners widely recognize the importance of making decisions based on the best available evidence. However, the effectiveness of evidence use in conservation planning is rarely assessed, which limits opportunities to improve evidence-based practice. We devised a mixed methodology for empirically evaluating use of evidence that applies social science tools to systematically appraise what kinds of evidence are used in conservation planning, to what effect, and under what limitations. We applied our approach in a case study of the Nature Conservancy of Canada (NCC), a leading land conservation organization. We conducted qualitative and quantitative analyses of 65 NCC planning documents (n = 13 in-depth) to identify patterns in evidence use, and surveyed 35 conservation planners to examine experiences of and barriers to using evidence. Although claims in plans contained a wide range of evidence types, 26% of claims were not referenced or associated with an identifiable source. Lack of evidence use was particularly apparent in claims associated with direct threats, particularly those identified as low (71% coded as insufficient or lacking evidence) or medium (45%) threats. Survey participants described relying heavily on practitioner experience and highlighted capacity limitations and disciplinary gaps in expertise among planning teams as barriers to using evidence effectively. We found that although time-intensive, this approach yielded actionable recommendations for improving evidence use in NCC conservation plans. Similar mixed-method assessments may streamline the process by including interviews and refining the document analysis frames to target issues or sections of concern. We suggest our method provides an accessible and robust point of departure for conservation practitioners to evaluate whether the use of conservation planning reflects in-house standards and more broadly recognized best practices.

The UN Decade of healthy ageing: strengthening measurement for monitoring health and wellbeing of older people

Over the past 100 years, life expectancy has increased dramatically in nearly all nations. Yet, these extra years of life gained have not all been healthy, particularly for older people aged 60 years and over. In 2020, the World Health Organisation (WHO) and United Nations (UN) member states embraced a sweeping 10-year global plan of action to ensure all older people can live long and healthy lives, formally known as the UN Decade of Healthy Ageing (2021–2030). With the adoption of the UN Decade of Healthy Ageing resolution, countries are committed to implementing collaborative actions to improve the lives of older people, their families and the communities in which they reside. The Decade addresses four interconnected areas of action. Adopting the UN's resolution on the Decade of Healthy Ageing has caused excitement, but a question that has weighed on everyone's mind is how governments will be held accountable? Besides, there have been no goals or targets set for the UN Decade of Healthy Ageing from a programmatic perspective for the action areas, and guidance on measures, data collection, analysis and reporting are urgently needed to support global, regional and national monitoring of the national strategies, programmes and policies. To this end, WHO in collaboration with UN agencies and international agencies established a Technical Advisory Group for Measurement of Healthy Ageing (TAG4MHA) to provide advice on the measurement, monitoring and evaluation of the UN Decade of Healthy Ageing at the global, regional and national levels.

Building coherent monitoring and evaluation plans with the Evaluation Planning Tool for global health

Practitioners in global health are called to monitor and evaluate their projects. This keeps projects on track, it meets donor and public demand, and it is a key mechanism by which global health organizations hold themselves accountable and improve their community of practice. However, monitoring and evaluation (M&E) is time- and resource-consuming, bringing into question whether the effort expended on M&E is worth it. While there has been a shift towards emphasizing the learning aspect of M&E, non-governmental organizations (NGOs) and other actors still struggle to get value from their efforts. One reason for this is that M&E plans are often not coherent or employed to their full potential. Theories of change, indicator lists, and data collection become a series of disjointed efforts that do not tie together. They become tick-the-box exercises to satisfy donors rather than a logical approach to draw meaningful findings for stakeholders, governments, and local communities. In this paper, we propose a step-by-step approach to utilizing M&E tools to their fullest potential, including: (1) a clearly defined theory of change that captures all program pathways and shows all intermediate objectives needed to achieve impact, (2) indicators which directly reflect the intermediate and ultimate objectives in the theory of change, and (3) a data collection plan which includes appropriate methods to measure indicators and address the questions stakeholders want answered. We make the case for a simpler, more coherent approach to M&E and propose a new tool to help practitioners more easily develop evaluation plans that are rigorous, practical, and worth the effort.

Assessing factors that support the abandonment of female genital mutilation in Ethiopia

This study assessed both demographic and Social Ecological Model (SEM) variables associated with supporting the abandonment of female genital mutilation (FGM) in three regions of Ethiopia. Data were collected through structured quantitative questionnaires with study participants (n=1146), including adolescent girls, caregivers, social network contacts, and community influentials. Data were analyzed using the statistical software STATA/SE. Chi-square, binary logistic regression, and multivariate logistic regression were used to analyze the data. Results found the one significant association between support for FGM abandonment and demographic variables was having an education above secondary level (AOR=0.11, CI=0.01-0.92). For SEM variables, attitudes regarding identity (AOR=5.74, CI=1.42-23.11), expectation to abandon (AOR=56.88, CI=14.99-215.90), sanctions (AOR=23.00, CI=4.77-110.95), and social networks (AOR=4.61, CI=1.21-17.51) were found to be significant factors in supporting FGM abandonment. Programs should intervene on multiple levels of the SEM and focus on social norms to empower and mobilize communities toward the elimination of FGM.

Development and Validation of a Novel Food-Based Global Diet Quality Score (GDQS)

BACKGROUND

Poor diet quality is a major driver of both classical malnutrition and noncommunicable disease (NCD) and was responsible for 22% of adult deaths in 2017. Most countries face dual burdens of undernutrition and NCDs, yet no simple global standard metric exists for monitoring diet quality in populations and population subgroups.

OBJECTIVES

We aimed to develop an easy-to-use metric for nutrient adequacy and diet related NCD risk in diverse settings.

METHODS

Using cross-sectional and cohort data from nonpregnant, nonlactating women of reproductive age in 10 African countries as well as China, India, Mexico, and the United States, we undertook secondary analyses to develop novel metrics of diet quality and to evaluate associations between metrics and nutrient intakes and adequacy, anthropometry, biomarkers, type 2 diabetes, and iteratively modified metric design to improve performance and to compare novel metric performance to that of existing metrics.

RESULTS

We developed the Global Diet Quality Score (GDQS), a food-based metric incorporating a more comprehensive list of food groups than most existing diet metrics, and a simple means of scoring consumed amounts. In secondary analyses, the GDQS performed comparably with the Minimum Dietary Diversity - Women indicator in predicting an energy-adjusted aggregate measure of dietary protein, fiber, calcium, iron, zinc, vitamin A, folate, and vitamin B12 adequacy and with anthropometric and biochemical indicators of undernutrition (including underweight, anemia, and serum folate deficiency), and the GDQS also performed comparably or better than the Alternative Healthy Eating Index - 2010 in capturing NCD-related outcomes (including metabolic syndrome, change in weight and waist circumference, and incident type 2 diabetes).

CONCLUSIONS

The simplicity of the GDQS and its ability to capture both nutrient adequacy and diet-related NCD risk render it a promising candidate for global monitoring platforms. Research is warranted to validate methods to operationalize GDQS assessment in population surveys, including a novel application-based 24-h recall system developed as part of this project.

Participatory Monitoring and Evaluation of the COVID-19 Response in a Local Public Health System

The coronavirus disease 2019 (COVID-19) pandemic tested the capacity of local health systems to understand and respond to changing conditions. Although data on new cases of COVID-19 were widely shared in communities, there was less information on the multisector response activities and factors associated with implementation. To address this gap, this empirical case study examined (a) the pattern of implementation of COVID-19 response activities and (b) the factors and critical events associated with both the pattern of new cases and the implementation of the local COVID-19 response. We used a participatory monitoring and evaluation system to capture, code, characterize, and communicate 580 COVID-19 response activities implemented in the city of Lawrence and Douglas County, Kansas. Collaboration across sectors including public health, medical services, city/county government, businesses, social services, public schools, and universities enabled the local public health system’s response effort. Documentation results showed the varying pattern of new COVID-19 cases and response activities over time and the factors identified as enabling or impeding the response and related new cases. Similar participatory monitoring and evaluation methods can be used by local health systems to help understand and respond to the changing conditions of COVID-19 response and recovery.

Making programmes worth their salt: Assessing the context, fidelity and outcomes of implementation of the double fortified salt programme in Uttar Pradesh, India

Double fortified salt (DFS) has proven efficacy in addressing iron deficiency and anaemia, thus improving maternal and child nutrition outcomes. However, DFS delivery in large‐scale settings is less understood, with limited documentation of its fidelity of implementation (FOI). We assessed the FOI of the DFS intervention in Uttar Pradesh, India, to improve the design and implementation of such programmes that aim to reduce the anaemia burden, especially in women of reproductive age (WRA). We conducted in‐depth interviews with DFS programme staff (n = 25) and end‐user WRAs (23), guided by a programme impact pathway. We transcribed and thematically analysed the interviews and used an adapted analytic framework to document FOI across four domains—objects of intervention, implementation staff, implementation context and target of implementation. DFS utilisation remained low due to a combination of factors including poor product quality, distribution challenges, ineffective promotion and low awareness amongst end‐user WRAs. Motivation levels were higher amongst district‐level staff compared to frontline staff, who lacked supervisory support and effective incentives to promote DFS. Three typologies of DFS users emerged—‘believers’, ‘thrifters’ and ‘naysayers’—who indicated differing reasons for DFS purchase and its use or nonuse. The implementation of the DFS programme varied significantly from its theorised programme impact pathway. The adapted analytic framework helped document FOI and assess the programme's readiness for impact assessments and subsequent scale‐up. The programme needs product quality improvements, incentivised distribution and stronger promotion to effectively deliver and improve the realisation of its potential as an anaemia prevention strategy.

Discrepancy in the evaluation of explicit and implicit nutrition care outcomes for patients at risk of malnutrition: A qualitative study

BACKGROUND

Nutrition care plays a significant role in the prevention and treatment of malnutrition, although the challenge to establish the precise impact of a nutrition intervention on patient outcomes remains. Malnutrition can be associated with diverse underlying diseases and an increased risk of complications, which increases the difficulty of monitoring and evaluating the nutrition intervention. The aim is to gain an understanding of dietitians' reflections concerning nutrition care outcomes of interventions in patients at risk of malnutrition.

METHODS

Six semi-structured audio-recorded focus group discussions with registered dietitians from primary healthcare and hospitals (n = 29) in Sweden were held at the dietitians' place of work or at the University. Focus group transcripts were analysed thematically to reveal patterns in the data and identify themes and subthemes.

RESULTS

The dietitians described an approach to nutrition monitoring and evaluation of patients at risk of malnutrition that was categorised into three themes: (i) quantitative explicit outcomes, based on objective measures and described as rigorous; (ii) quantitative estimated outcomes, based on estimates and described as less rigorous and (iii) qualitative implicit outcomes, based on patients' subjective perceptions and experiences of their health and described as difficult to measure.

CONCLUSIONS

Findings indicate the need for new strategies to promote systematic and comprehensive nutrition monitoring and evaluation.

Evolution of the monitoring and evaluation strategies to support the World Health Organization's Global Programme to Eliminate Lymphatic Filariasis

The Global Programme to Eliminate Lymphatic Filariasis (GPELF) was established with the ambitious goal of eliminating LF as a public health problem. The remarkable success of the GPELF over the past 2 decades in carrying out its principal strategy of scaling up and scaling down mass drug administration has relied first on the development of a rigorous monitoring and evaluation (M&E) framework and then the willingness of the World Health Organization and its community of partners to modify this framework in response to the practical experiences of national programmes. This flexibility was facilitated by the strong partnership that developed among researchers, LF programme managers and donors willing to support the necessary research agenda. This brief review summarizes the historical evolution of the GPELF M&E strategies and highlights current research needed to achieve the elimination goal.

Collect Once, Use Many Times: Attaining Unified Metrics for Tuberculosis Preventive Treatment for People Living With HIV

The World Health Organization (WHO) recommends providing tuberculosis preventive treatment (TPT) to all persons living with HIV and to all household contacts of persons with bacteriologically confirmed pulmonary tuberculosis disease. Regrettably, the absence of a harmonized data collection and management approach to TPT indicators has contributed to programmatic challenges at local, national, and global levels. However, in April 2020, the WHO launched the Consolidated HIV Strategic Information Guidelines, with an updated set of priority indicators. These guidelines recommend that Ministries of Health collect, report, and use data on TPT completion in addition to TPT initiation. Both indicators are reflected in the WHO's list of 15 core indicators for program management and are also required by the US President's Emergency Plan for AIDS Relief's Monitoring, Evaluation, and Reporting (MER) guidance. Although not perfectly harmonized, both frameworks now share essential indicator characteristics. Aligned indicators are necessary for robust strategic and operational planning, resource allocation, and data communication. "Collect once, use many times" is a best practice for strategic information management. Building harmonized and sustainable health systems will enable countries to successfully maintain essential HIV, tuberculosis, and other health services while combatting new health threats.

Evaluation of patient engagement in medicine development: A multi-stakeholder framework with metrics

BACKGROUND

Patient engagement is becoming more customary in medicine development. However, embedding it in organizational decision-making remains challenging, partly due to lack of agreement on its value and the means to evaluate it. The objective of this project was to develop a monitoring and evaluation framework, with metrics, to demonstrate impact and enhance learning.

METHODS

A consortium of five patient groups, 15 biopharmaceutical companies and two academic groups iteratively created a framework in a multi-phase participatory process, including analysis of its application in 24 cases.

RESULTS

The framework includes six components, with 87 metrics and 15 context factors distributed among (sub)components: (a) Input: expectations, preparations, resources, representativeness of stakeholders; (b) Activities/process: structure, management, interactions, satisfaction; (c) Learnings and changes; (d) Impacts: research relevance, study ethics and inclusiveness, study quality and efficiency, quality of evidence and uptake of products, empowerment, reputation and trust, embedding of patient engagement; (e) Context: policy, institutional, community, decision-making contextual factors. Case study findings show a wide variation in use of metrics. There is no 'one size fits all' set of metrics appropriate for every initiative or organization. Presented sample sets of metrics can be tailored to individual situations.

CONCLUSION

Introducing change into any process is best done when the value of that change is clear. This framework allows participants to select what metrics they value and assess to what extent patient engagement has contributed.

PATIENT CONTRIBUTION

Five patient groups were involved in all phases of the study (design, conduct, interpretation of data) and in writing the manuscript.

Insights from mathematical modelling and quantitative analysis on the proposed 2030 goals for trachoma

Trachoma is a neglected tropical disease and the leading infectious cause of blindness worldwide. The current World Health Organization goal for trachoma is elimination as a public health problem, defined as reaching a prevalence of trachomatous inflammation-follicular below 5% in children (1-9 years) and a prevalence of trachomatous trichiasis in adults below 0.2%. Current targets to achieve elimination were set to 2020 but are being extended to 2030. Mathematical and statistical models suggest that 2030 is a realistic timeline for elimination as a public health problem in most trachoma endemic areas. Although the goal can be achieved, it is important to develop appropriate monitoring tools for surveillance after having achieved the elimination target to check for the possibility of resurgence. For this purpose, a standardized serological approach or the use of multiple diagnostics in complement would likely be required.

Insights from mathematical modelling and quantitative analysis on the proposed 2030 goals for trachoma

Trachoma is a neglected tropical disease and the leading infectious cause of blindness worldwide. The current World Health Organization goal for trachoma is elimination as a public health problem, defined as reaching a prevalence of trachomatous inflammation-follicular below 5% in children (1-9 years) and a prevalence of trachomatous trichiasis in adults below 0.2%. Current targets to achieve elimination were set to 2020 but are being extended to 2030. Mathematical and statistical models suggest that 2030 is a realistic timeline for elimination as a public health problem in most trachoma endemic areas. Although the goal can be achieved, it is important to develop appropriate monitoring tools for surveillance after having achieved the elimination target to check for the possibility of resurgence. For this purpose, a standardized serological approach or the use of multiple diagnostics in complement would likely be required.

Uptake of the Ithaka mobile application in Johannesburg, South Africa, for human immunodeficiency virus self-testing result reporting

Background

Human immunodeficiency virus self-testing (HIVST) can reduce facility-based HIV testing barriers; however, no proven applications exist with widespread uptake for self-reporting or linkage to care. Mobile health (mHealth) applications (apps) have shown high usability and feasibility scores, so Ithaka was developed for South Africans to self-report HIVST results outside clinical settings.

Objectives

This study investigated the use of Ithaka as a support tool for HIVST users, specifically the ability to self-report results.

Method

This cross-sectional study was conducted from November 2018 to June 2019. At existing HIVST distribution sites, individuals were given HIVST kits and then invited to use Ithaka. Participants could test at home and report their results through the app anytime. Ithaka tracked when people logged-on, registered, received counselling and reported results. Post-study surveys on user experience were also conducted.

Results

Of 751 participants, 531 (70.7%) logged onto the app, 412 (54.9%) registered, 295 (39.3%) received counselling and 168 (22.4%) self-reported results. Participants strongly agreed that Ithaka was useful and that it was easy to upload results. Forty-one participants completed a post-test survey, and 39/41 (95.1%) completed the app journey. Most participants (36/41;87.8%) had no challenges, although 2/41 (4.9%) cited perceived data costs, 2/41 (4.9%) difficulty uploading results and 1/41 (2.4%) language, as challenges.

Conclusion

Despite the small sample size, this study has shown that HIVST participants under pragmatic conditions were willing and able to self-report results via the app, whilst also identifying areas of improvement for scaling up.

Impact, theory of change, and the horizons of scientific practice

Demands for research to generate impact, along with proliferating institutional regimes for evaluating impact, are a ubiquitous aspect of contemporary scientific practice. Based on participant observation at an agro-environmental research institute in southwest China, this article explores three iterations of a tool for planning and evaluating impactful science called 'theory of change' (TOC). Despite their ostensible common grounding in TOC, I show how an impact scientist's framework, a donor's monitoring and evaluation regime, and a communication consultant's branding strategy each suggest very different normative structures for scientific practice. These structures entail: particular horizons towards which scientific research is to be practiced, precise points in time at which the future effects of research are to be anticipated, and specific assumptions about how scientists' agency should play out across time. Taking the peculiar sensibilities of TOC as a comparative framework, I illuminate IFF scientists' implicit imaginations of how contemporary science does and should generate effects in the world.

Driving the usage of tuberculosis diagnostic data through capacity building in low- and middle-income countries

BACKGROUND

Connectivity platforms collect a wealth of data from connected GeneXpert instruments, with the potential to provide valuable insights into the burden of disease and effectiveness of tuberculosis programmes. The challenge faced by many countries is a lack of training, analytical skills, and resources required to understand and translate this data into patient management and programme improvement.

OBJECTIVE

We describe a novel training programme, the tuberculosis Data Fellowship, designed to build capacity in low- and middle- income countries for tuberculosis data analytics.

METHODS

The programme consisted of classroom and remote training plus mentorship over a 12-month period. The focus was on skills development in Tableau software, followed by training in exploration, analysis, and interpretation of GeneXpert tuberculosis data across five key programme areas: patient services, programme monitoring, quality of testing, inventory management, and disease burden.

RESULTS

The programme was piloted in six countries (Bangladesh, Ethiopia, Ghana, Malawi, Mozambique) in July 2018 and Nigeria in September 2018; 20 participants completed the training. A number of key outputs have been achieved, such as improved instrument utilisation rates, decreased error rates, and improved instrument management.

CONCLUSION

The training programme empowers local tuberculosis programme staff to discover and fix critical inefficiencies, provides high-level technical and operational support to the tuberculosis programme, and provides a platform for continued sharing of insights and best practices between countries. It supports the notion that connectivity can increase efficiencies and clinical benefits with better data for decision making, if coupled with commensurate capacity building in data analysis and interpretation.

High variability in the measurement of HIV primary prevention activities and outcomes

INTRODUCTION

While there is a global consensus on monitoring Human Immunodeficiency Virus (HIV) treatment progress, there has been less attention to the degree of consistency of the measurement of HIV prevention programmes-and the global prevention response is not on-track to achieve 2020 goals. In this paper, we assess the degree of variability in primary prevention indicators selected by national strategic plans (NSPs) and global stakeholder monitoring and evaluation (M&E) strategies.

METHODS

We obtained the most recent NSPs from low and middle income Joint United Nations Programme on HIV/AIDS (UNAIDS) Fast-Track countries, and M&E documents from The Global Fund to Fight AIDS, Tuberculosis and Malaria (The Global Fund), President's Emergency Plan for AIDS Relief (PEPFAR), UNAIDS, the Global HIV Prevention Coalition and the World Health Organization (WHO). We extracted HIV primary prevention indicators from each document, standardized and aggregated them by age/ sex, categorized indicators by topic, and evaluated the frequency of matched indicators between countries and stakeholders. Data were collected between February and April of 2019.

RESULTS

Twenty-one NSPs and five global stakeholder documents were assessed; 736 primary prevention indicators were identified; 284 remained following standardization and aggregation. NSPs contained from 3 to 48 primary prevention indicators, with an average of 23; categories included: HIV education and outreach (17.6%), testing (17.3%) and condom use (16.2%). Of unique national indicators, only 34% was shared between two or more countries. Sixty-nine per cent was applied in a single country only. 56% of NSP indicators did not appear in any global stakeholder document. Conversely, 42% of global indicators did not appear in any surveyed NSPs. Within global indicators, 63% was only measured by one global body, and no single indicator was measured by all five.

CONCLUSIONS

These analyses reveal a lack of consensus both between and within countries' and global stakeholders' measurement of HIV prevention. Though some variability is expected, these findings point to a need to refocus attention on achieving greater consensus on a global measurement framework for HIV prevention.

Sustained nutrition impact of a multisectoral intervention program two years after completion

Progress on the nutrition Sustainable Development Goals has been slow. More attention is needed on the 'sustainable' part, focused on impact lasting beyond programme implementation. To determine sustained impact of a multisectoral nutrition intervention that provided water, sanitation, hygiene, livelihood, health and nutrition support (2013-2015) in eastern Chad, we utilize longitudinal household data collected 2 years (2017) after the intervention ended. Between 2013 and 2015, children (6-59 months) in the multisectoral intervention were less likely to be severely wasted, underweight and had a higher weight-for-height z-score (WHZ) compared with the control. To measure sustained programme impact, we use data on six nutrition indicators from 517 children between 2015 and 2017. We ran three models: a generalized linear model on cross-sectional child cohorts; a mixed-effects model on household panel data; and a mixed-effects model on child panel data. For children who were born during the programme, we saw significant improvement in underweight, weight for age z-scores (WAZs) and height-for-age z-scores (HAZs). Boys 6-23 months born after the end of the programme, on the other hand, were significantly more likely to be underweight or wasted and had lower WHZ and WAZ compared with boys born during the programme and girls born during and after the programme. Corresponding to the literature from sub-Saharan Africa, boys appear to be more vulnerable to malnutrition, which might be why they are more sensitive to programme cessation. Future monitoring, evaluations and research need to consider impact sustainability and that it might not be homogeneous across age and gender.

Evaluation of a mobile application to support HIV self-testing in Johannesburg, South Africa

Background

Human immunodeficiency virus self-testing (HIVST) reduces barriers associated with facility-based testing; however, no formal mechanism exists for users to self-report results or link to care. The Aspect^TM HIVST mobile application (app) was developed for use in South Africa.

Objectives

This study evaluated the acceptability and feasibility of the Aspect^TM HIVST app for individuals from the inner city of Johannesburg.

Method

This cross-sectional pilot, with a convenience sample of 300 adults, was conducted in July 2018. Participants were provided an OraQuick HIVST kit and a smartphone preloaded with the app, then asked to follow the in-app instructions for use (IFU) to complete the HIVST and upload results. Trained healthcare workers (HCWs) observed and recorded any deviations from the IFU, and conducted a post-test survey to assess acceptability. Feasibility was evaluated by the number of participants who agreed to participate, completed the self-test, and uploaded all information onto the app correctly.

Results

Most participants (98.7%) found the app easy to use. To reduce difficulties related to the IFU (26; 8.7%), participants suggested multimedia supplements (4; 1.3%), additional languages (4; 1.3%) and simplified instructions (5; 1.7%). All individuals approached, agreed to participate, 267 (89.0%) correctly completed all steps and 210 (78.7%) successfully captured all information on the app. Most errors (26; 8.7%) were testing errors and 1 (0.3%) was from the app sequence. Twelve (4.5%) errors were with test strip imaging and 72 (27.0%) discordances were with demographic information.

Conclusion

Despite some challenges with IFU interpretation and data capture via the app, this pilot showed that the Aspect^TM HIVST app is an acceptable way to upload mobile HIVST results and demographic information to a central database.

Community Health Impacts of the Trident Copper Mine Project in Northwestern Zambia: Results from Repeated Cross-Sectional Surveys

The application of a health impact assessment (HIA) for a large-scale copper mining project in rural Zambia triggered the long-term monitoring and evaluation of determinants of health and health outcomes in communities living in proximity to the mine. Three consecutive cross-sectional surveys were conducted at intervals of four years; thus, at baseline (2011), four (2015) and eight (2019) years into the project’s development. Using the same field and laboratory procedures, the surveys allowed for determining changes in health indicators at the household level, in young children (<5 years), school attendees (9–14 years) and women (15–49 years). Results were compared between communities considered impacted by the project and communities outside the project area (comparison communities). The prevalence of Plasmodium falciparum infection increased in both the impacted and comparison communities between 2011 and 2019 but remained consistently lower in the impacted communities. Stunting in children < 5 years and the prevalence of intestinal parasite infections in children aged 9–14 years mostly decreased. In women of reproductive age, selected health indicators (i.e., anaemia, syphilis, underweight and place of delivery) either remained stable or improved. Impacted communities generally showed better health outcomes than comparison communities, suggesting that the health interventions implemented by the project as a consequence of the HIA have mitigated potential negative effects and enhanced positive effects. Caution is indicated to avoid promotion of health inequalities within and beyond the project area.

The multi-sectorial emergency response to a cholera outbreak in Internally Displaced Persons camps in Borno State, Nigeria, 2017

Introduction

In August 2017, a cholera outbreak started in Muna Garage Internally Displaced Persons camp, Borno state, Nigeria and >5000 cases occurred in six local government areas. This qualitative study evaluated perspectives about the emergency response to this outbreak.

Methods

We conducted 39 key informant interviews and focus group discussions, and reviewed 21 documents with participants involved with surveillance, water, sanitation, hygiene, case management, oral cholera vaccine (OCV), communications, logistics and coordination. Qualitative data analysis used thematic techniques comprising key words in context, word repetition and key sector terms.

Results

Authorities were alerted quickly, but outbreak declaration took 12 days due to a 10-day delay waiting for culture confirmation. Outbreak investigation revealed several potential transmission channels, but a leaking latrine around the index cases’ house was not repaired for more than 7 days. Chlorine was initially not accepted by the community due to rumours that it would sterilise women. Key messages were in Hausa, although Kanuri was the primary local language; later this was corrected. Planning would have benefited using exercise drills to identify weaknesses, and inventory sharing to avoid stock outs. The response by the Rural Water Supply and Sanitation Agency was perceived to be slow and an increased risk from a religious festival was not recognised. Case management was provided at treatment centres, but some partners were concerned that their work was not recognised asking, ‘Who gets the glory and the data?’ Nearly one million people received OCV and its distribution benefited from a robust infrastructure for polio vaccination. There was initial anxiety, rumour and reluctance about OCV, attributed by many to lack of formative research prior to vaccine implementation. Coordination was slow initially, but improved with activation of an emergency operations centre (EOC) that enabled implementation of incident management system to coordinate multisectoral activities and meetings held at 16:00 hours daily. The synergy between partners and government improved when each recognised the government’s leadership role.

Conclusion

Despite a timely alert of the outbreak, delayed laboratory confirmation slowed initial response. Initial responses to the outbreak were not well coordinated but improved with the EOC. Understanding behaviours and community norms through rapid formative research should improve the effectiveness of the emergency response to a cholera outbreak. OCV distribution was efficient and benefited from the polio vaccine infrastructure.

Transition to an in-facility electronic Tuberculosis register: Lessons from a South African pilot project

Background

South Africa has one of the highest incidences of Tuberculosis (TB) globally. High co-morbid HIV prevalence complicates TB management and treatment outcomes. Growing evidence suggests that integrating the TB and HIV programmes will improve the overall results.

Objectives

To describe how TB programme staff at various levels of the South African health system responded to the transition from a paper-based to an electronic register of TB data integrated with HIV programme data.

Method

Three primary health service facilities in the Cape Winelands district, Western Cape province, South Africa served as pilot sites for implementation. Semi-structured interviews were conducted with 21 TB programme staff purposively selected at facility, sub-district, district and provincial levels of the health system, based on their involvement in implementing electronic TB data. An objective-driven thematic frame was used to analyse the data.

Results

Fears about the transition included reductions in data quality, changes to the status quo and a lack of computer literacy. Participants acknowledged benefits of reduced workloads, speed of accessing patient-level data and click-of-a-button reporting. Three factors influenced the ease of adopting the new system: firstly, implementation challenged the vertical position of the TB programme, TB data and staff's conventional roles and responsibilities; secondly, perceptions of the paper-based register as functional and reliable made the transition to electronic seem unnecessary; and thirdly, lack of a process of change management challenged staff's ability to internalise the proposed change.

Conclusion

A process of change management is critical to facilitate the efficiency and effectiveness with which the electronic in-facility TB register is implemented.

Evaluating implementation of the Ebola response in Margibi County, Liberia

Implementation of the Ebola response was credited with reducing incidence of Ebola virus disease (EVD) in West Africa; however little is known about the amount and kind of Ebola response activities that were ultimately successful in addressing the 2014 outbreak. We collaboratively monitored Ebola response activities and associated effects in Margibi County, Liberia, a rural county in Liberia deeply affected by the outbreak. We used a participatory monitoring and evaluation system, including key informant interviews and document review, to systematically document activities, code them, characterize their contextual features, and discover and communicate patterns in Ebola response activities to essential stakeholders. We also measured incidence of EVD over time. Results showed a distinct pattern in Ebola response activities and key events, which corresponded with subsequent decreased EVD incidence. These data are suggestive of the role of Ebola response activities played in reducing the incidence of EVD within Margibi County, which included implementing safe burials, social mobilization and community engagement and case management. Systematic monitoring and evaluation of response activities to control disease outbreaks holds lessons for implementing and evaluating similar comprehensive and multi-sectoral community health efforts.

Nutrition Embedding Evaluation Programme: An Evaluation Technical Assistance Model for Supporting Civil Society Organizations to Conduct Quality Nutrition Impact Evaluations

The Nutrition Embedding Evaluation Programme (NEEP) was a global 4-year program (2013-2017) funded by the United Kingdom Department for International Development created to respond to gaps in the nutrition evidence base. The NEEP implementing agency-PATH-provided grants and evaluation technical assistance (ETA) to civil society organizations (CSOs) from 12 countries to conduct robust nutrition-related impact evaluations. The programmatic approach of having an intermediary agent to manage the funding and ETA mechanisms for nutrition impact evaluations is rare and therefore provides a unique opportunity to understand its effectiveness. Over the program duration, NEEP collected lessons learned that were analyzed and disaggregated into key themes considered critical for the completion of high-quality impact evaluations. From these lessons learned, NEEP provides an ETA program model that can be replicated or adapted to other international development sectors. This model highlights the key role of the three tiers (donor, ETA manager, and CSOs) in ensuring the best value for money and effective technical support for conducting impact evaluations and fostering the importance of knowledge uptake and evaluative culture for maximum knowledge diffusion. In this way, global research can be targeted to approaches that provide options to collaborate with the program implementers and contribute to a holistic evidence base to inform policy and programmatic decisions.

An illustration of how responsive feedback in a social marketing tobacco control intervention in Ghana enabled managers to make decisions that increased intervention effectiveness

This report illustrates how a feedback loop, set up to provide data and insights to a donor and designers/implementers of a social marketing tobacco prevention intervention in Ghana, helped adapt the original design of the intervention to one that was more suited to the social and media contexts of Ghana. The designers/implementers had previously, successfully implemented a tobacco control intervention with adolescents in Botswana. This experience had informed the initial intervention design in Ghana. As the feedback generated by evaluators started demonstrating just how different the Ghanaian social and media contexts were from the Botswanan one, implementers started making changes to their selection of channels, resulting in a design which was quite different from the original one. The close involvement of the donor in this process enabled implementers to make rapid changes to the design of the intervention. This illustration adds to a small but growing literature establishing the importance of feedback loops to improve the design and implementation of development interventions.

Developing a framework for the monitoring and evaluation of the Health Transformation Plan in the Islamic Republic of Iran: lessons learned

Background

Monitoring and evaluation of health system reforms are essential to ensure the achievement of their objectives. The latest heath sector reform in the Islamic Republic of Iran, namely, the Health Transformation Plan (HTP), was launched in 2014 and the country is embarking on the HTP to achieve universal health coverage (UHC).

Aims

The study aimed to develop the most appropriate monitoring and evaluation framework for the HTP in accordance with national and global goals and priorities, and to identify data gaps in its monitoring and evaluation.

Method

A case study and evidence-based approach was applied to develop the monitoring and evaluation framework. The model that was proposed jointly by the World Bank and the World Health Organization for monitoring and evaluation of UHC was used as the basis for the potential list of the indicators and key policy documents were reviewed, accordingly. The framework formulation process was carried out through a series of meetings with experts and senior managers working at the Ministry of Health and Medical Education, whose perspectives on the frameworks functionality and usage were regarded as valuable. The final draft was presented to policy-makers for input and approval.

Results

A data mapping revealed that at least nine national surveys were required to obtain the indicators for effective monitoring. At the time of framework designing, many indicators were not available or had not been updated for several years due to lack of available and appropriate data sources.

Conclusions

Results indicated that the country's health information system had many information gaps that should be filled to enable the tracking of UHC goals and measuring the success of the plan. Applying the proposed framework would increase the comparability of the country's health indicators at the global level and specify a path to successfully achieve the objectives of the reform.

Achieving the targets for universal health coverage: how is Thailand monitoring progress?

Universal health coverage (UHC) is one of the targets within the Sustainable Development Goals that the Member States of the United Nations have pledged to achieve by 2030. Target 3.8 has two monitoring indicators: 3.8.1 for coverage of essential health services, for which a compound index from 16 tracer indicators has recently been developed; and 3.8.2 for catastrophic expenditure on health. The global baseline monitoring of these two indicators in 2017 shows that the progress in many low- and middle-income countries is unlikely to be on track and achieved by 2030. The monitoring and evaluation mechanism for UHC progress is a crucial function to hold governments accountable and guide countries along their paths towards UHC. This paper outlines key monitoring and evaluation tools that Thailand uses to track UHC progress; compares the strengths and limitations of each tool; and discusses monitoring gaps and enabling factors related to development of the tools. Thailand uses several data sources to monitor three UHC dimensions: population coverage; service coverage; and financial risk protection. The four key sources are: (i) national surveys; (ii) health facility and administrative data; (iii) specific disease registries; and (iv) research. Each source provides different advantages and is used concurrently to complement the others. Despite initially being developed to track progress for national health priorities, these tools are able to monitor most of the global UHC indicators. Key enabling factors of Thai monitoring systems are a supportive infrastructure and information system; a policy requirement for routine patient data records; ownership and commitment of the key responsible organizations; multisectoral collaboration; and sustainable in-country capacities. The areas for improvement are monitoring in the non-Thai population; tracking access to essential medicines; and maximizing the use of collected data. Lessons learnt from the Thai experience could be useful for other low- and middle-income countries in developing their UHC monitoring platforms.

Responsive feedback: Towards a new paradigm to enhance intervention effectiveness

The current dominant models of intervention design in the development sector do not account for the complexity and unpredictability of implementation challenges. Decision makers and implementers need timely feedback during implementation to respond to field realities and to course-correct. This letter calls for a new approach of "responsive feedback" or "feedback loops" that promotes interactions between project designers, implementers, researchers and decision-makers to enable course corrections needed to achieve intended outcomes. A responsive feedback approach, in theory, should be agile, flexible, adaptive, iterative, and actionable. There can be multiple challenges associated with incorporating this approach into practice including donor requirements, organizational structure and culture, concerns about the additional time required to adopt such an approach, resource and operational constraints, the absence of skill sets needed for such an approach within smaller organizations and inadequate inter-departmental communication. However, these barriers to adaptation can be overcome. For responsive feedback to become a part of the culture of development organizations, commitment is needed from donors, decision-makers, project designers and implementers. We believe that, to generate opportunities for learning and adaptation, donors should provide the stimulus to break down silos between implementers and researchers.

Responsive feedback: Towards a new paradigm to enhance intervention effectiveness

The current dominant models of intervention design in the development sector do not account for the complexity and unpredictability of implementation challenges. Decision makers and implementers need timely feedback during implementation to respond to field realities and to course-correct. This letter calls for a new approach of “responsive feedback” or “feedback loops” that promotes interactions between project designers, implementers, researchers and decision-makers to enable course corrections needed to achieve intended outcomes. A responsive feedback approach, in theory, should be agile, flexible, adaptive, iterative, and actionable. There can be multiple challenges associated with incorporating this approach into practice including donor requirements, organizational structure and culture, concerns about the additional time required to adopt such an approach, resource and operational constraints, the absence of skill sets needed for such an approach within smaller organizations and inadequate inter-departmental communication. However, these barriers to adaptation can be overcome. For responsive feedback to become a part of the culture of development organizations, commitment is needed from donors, decision-makers, project designers and implementers. We believe that, to generate opportunities for learning and adaptation, donors should provide the stimulus to break down silos between implementers and researchers.

Lessons for health program monitoring and evaluation in a low resource setting

Numerous guidelines outline best practices for health program monitoring and evaluation (M&E). However, health programs are often implemented in less than ideal circumstances where these best practices may not be resourced or feasible. This article describes how M&E has been conducted for a health service delivery improvement program in remote Papua New Guinea and outlines lessons learned. The lessons learned were to integrate M&E into every aspect of the program, strengthen existing health information data, link primary data collection with existing program activities, conduct regular monitoring and feedback for early identification of implementation issues, involve the program team in evaluation, and communicate M&E data through multiple mediums to stakeholders. These lessons could be applied to other health programs implemented in low resource settings.

The Healthy Primary School of the Future: A Contextual Action-Oriented Research Approach

Background: Schools can play an important role in promoting children’s health behaviours. A Dutch initiative, ‘The Healthy Primary School of the Future’, aims to integrate health and well-being into the school system. We use a contextual action-oriented research approach (CARA) to study the implementation process. Properties of CARA are its focus on contextual differences and the use of monitoring and feedback to support and evaluate the process of change. The aim of this article is to describe the use of the approach. Methods: Four schools (each with 200–300 children, aged 4–12 years) were included; all located in low socio-economic status areas in the south of the Netherlands. Data collection methods include interviews, observations, questionnaires, and health and behavioural measurements. Research contributions include giving feedback and providing schools with a range of possibilities for additional changes. The contextual data we examine include schools’ health promoting elements, practices of teachers and parents, dominating organisational issues, and characteristics of the student population; process data include the presence of potential barriers to changes. Discussion: CARA is an adaptive research approach that generates knowledge and experiences on how to deal with health promotion in complex systems. We think this approach can set an example for research efforts in comparable initiatives.

Routinely collected infant feeding data: Time for global action

The importance of breastfeeding is clear. However, global action to support breastfeeding is hindered by the lack of reliable standard data, which continues to impede progress. Routinely collected data can monitor the effectiveness of health policy, evaluate interventions, and enhance international research collaboration and comparisons. Use of routine data to support effective public health initiatives such as smoking cessation has been demonstrated. However, the data collected about infant feeding practices worldwide is inconsistent in timing, methods, definitions, detail, storage, and consistency. Improvements to the reach and quality of routinely collected data about infant feeding are needed to strengthen the global evidence and policy base. An international collaborative effort is called for to progress this.

Understanding health worker data use in a South African antiretroviral therapy register

OBJECTIVE

To evaluate how electronic data management systems affect data use practices in antiretroviral therapy (ART) programs within local health districts, and individual health facilities.

METHODS

We used a data quality audit to establish a baseline of the quality of data in the electronic register alongside in-depth interviews with health workers and managers, to understand perceptions of data quality, data use by facility staff and challenges affecting data use.

RESULTS

The findings provide a four-level continuum of data use that can be applied to other settings and recommendations for optimising facility-level data use.

CONCLUSION

By defining four levels of data use our findings suggest the potential to encourage a structured process of moving from passive data use, to more active and engaged data use, where data could be used to anticipate patient behaviour and link that behaviour to differentiated care plans.

In the eye of the beholder: to make global health estimates useful, make them more socially robust

A plethora of new development goals and funding institutions have greatly increased the demand for internationally comparable health estimates in recent years, and have brought important new players into the field of health estimate production. These changes have rekindled debates about the validity and legitimacy of global health estimates. This paper draws on country case studies and personal experience to support our opinion that the production and use of estimates are deeply embedded in specific social, economic, political and ideational contexts, which differ at different levels of the global health architecture.

Broadly, most global health estimates tend to be made far from the local contexts in which the data upon which they are based are collected, and where the results of estimation processes must ultimately be used if they are to make a difference to the health of individuals. Internationally standardised indicators are necessary, but they are no substitute for data that meet local needs, and that fit with local ideas of what is credible and useful. In other words, data that are both technically and socially robust for those who make key decisions about health.

We suggest that greater engagement of local actors (and local data) in the formulation, communication and interpretation of health estimates would increase the likelihood that these data will be used by those most able to translate them into health gains for the longer term. Besides strengthening national information systems, this requires ongoing interaction, building trust and establishing a communicative infrastructure. Local capacities to use knowledge to improve health must be supported.

Developing a real-time PCR assay based on multiplex high-resolution melt-curve analysis: a pilot study in detection and discrimination of soil-transmitted helminth and schistosome species

With the push towards control and elimination of soil-transmitted helminthiasis and schistosomiasis in low- and middle-income countries, there is a need to develop alternative diagnostic assays that complement the current in-country resources, preferably at a lower cost. Here, we describe a novel high-resolution melt (HRM) curve assay with six PCR primer pairs, designed to sub-regions of the nuclear ribosomal locus. Used within a single reaction and dye detection channel, they are able to discriminate Ancylostoma duodenale, Necator americanus, Strongyloides stercoralis, Ascaris lumbricoides, Trichuris trichiuria and Schistosoma spp. by HRM curve analysis. Here we describe the primers and the results of a pilot assessment whereby the HRM assay was tested against a selection of archived fecal samples from Ghanaian children as characterized by Kato-Katz and real-time PCR analysis with species-specific TaqMan hydrolysis probes. The resulting sensitivity and specificity of the HRM was 80 and 98.6% respectively. We judge the assay to be appropriate in modestly equipped and resourced laboratories. This method provides a potentially cheaper alternative to the TaqMan method for laboratories in lower resource settings. However, the assay requires a more extensive assessment as the samples used were not representative of all target organisms.

Structural barriers to South African volunteer home-based caregivers providing quality care: the need for a policy for caregivers not affiliated to primary healthcare clinics

Community home-based care (CHBC) is a critical component of non-formal care in communities in Africa that have a high prevalence of HIV and tuberculosis (TB). Community carers consisting primarily of volunteers are critical role players in African healthcare systems and particularly in South Africa's strategy to fight HIV and AIDS. This paper explores the structural barriers volunteer caregivers need to overcome to provide quality CHBC. The researchers used two focus group discussions with key informants (each with four participants), and semi-structured interviews with six key informants to collect data relating to the meaning of quality CHBC. The data were coded using Tesch's data analysis technique. A major theme that emerged from the results was "Addressing structural challenges to improve the quality of CHBC". Subthemes underpinning this theme were: 1) lack of standardised training of volunteer caregivers; 2) the need for a scope of practice, parameters and legal boundaries; 3) lack of monitoring and evaluation (M&E) of CHBC; and 4) the importance of mentoring and supervision in CHBC. CHBC policy should address the need for standardised training programmes for caregivers, so that they are equipped with multiple skills. Furthermore CHBC policy must emphasise mentoring as well as M&E to encourage quality care. Finally, the policy should provide a clear scope of practice for caregivers to regulate their competencies and boundaries.