Practical lessons for phone-based assessments of learning

Live tutoring calls did not improve learning during the COVID-19 pandemic in Sierra Leone

Education systems regularly face unexpected school closures, whether due to disease outbreaks, natural disasters, or other adverse shocks. In low-income countries where internet access is scarce, distance learning - the most common educational solution - is often passive, via TV or radio, with little opportunity for teacher-student interaction. In this paper we evaluate the effectiveness of live tutoring calls from teachers, designed to supplement radio instruction during the 2020 school closures prompted by the COVID-19 pandemic. We do this with a randomised controlled trial with 4,399 primary school students in Sierra Leone. Tutoring calls led to some limited increase in educational activity, but had no effect on mathematics or language test scores, whether for girls or boys, and whether provided by public or private school teachers. Even having received tutoring calls, one in three children reported not listening to educational radio at all, so limited take-up may partly explain our results.

Teaching and Incentives: Substitutes or Complements?

Interventions to promote learning are often categorized into supply- and demand-side approaches. In a randomized experiment to promote learning about COVID-19 among Mozambican adults, we study the interaction between a supply and a demand intervention, respectively: teaching via targeted feedback, and providing financial incentives to learners. In theory, teaching and learner-incentives may be substitutes (crowding out one another) or complements (enhancing one another). Experts surveyed in advance predicted a high degree of substitutability between the two treatments. In contrast, we find substantially more complementarity than experts predicted. Combining teaching and incentive treatments raises COVID-19 knowledge test scores by 0.5 standard deviations, though the standalone teaching treatment is the most cost-effective. The complementarity between teaching and incentives persists in the longer run, over nine months post-treatment.

Respondent Fatigue Reduces Dietary Diversity Scores Reported from Mobile Phone Surveys in Ethiopia during the COVID-19 Pandemic

BACKGROUND

The computer-assisted telephone interview (CATI) has been used extensively during the COVID-19 pandemic, but the effects of respondent fatigue during these interviews on responses to questions about diet are unknown.

OBJECTIVES

We designed an experiment that randomized the placement of a survey module on the dietary diversity of rural Ethiopian women and assessed whether responses were altered by placing this module earlier or later in a phone survey.

METHODS

Two CATIs were implemented; in the second, women were randomly assigned to answer questions on diet diversity either earlier or later in the interview. Women's Dietary Diversity Scores were the primary outcome. Secondary outcomes were dichotomous measures of consumption from four or more and five or more food groups and consumption of food groups consumed frequently, often, and rarely. Impacts were assessed using a respondent fixed effects model.

RESULTS

Delaying the food consumption module by 15 min in the interview led to an 8%-17% (P < 0.01) decrease in reported Dietary Diversity Scores, a 28% (P < 0.01) decrease in the number of women who consumed a minimum of four dietary groups, and a 40% (P < 0.01) and 11% (P < 0.01) decrease in the reporting of consumption of animal source foods and fruits and vegetables, respectively. Moving the food consumption module closer to the beginning of the interview increased the number of reported food groups consumed by older women, women with a below-median education level, and women in larger households.

CONCLUSIONS

Our findings suggest that comparisons of descriptive statistics across studies and countries on metrics such as food security and dietary quality may be confounded by where these modules are placed in the interview, thus highlighting trade-offs between volume of information collected and data quality when designing CATI surveys.

Medium-term protective effects of quality early childhood education during the COVID-19 pandemic in Ghana

The COVID‐19 pandemic led to extended school closures globally. Access to remote learning opportunities during this time was vastly unequal within and across countries. Higher‐quality early childhood education (ECE) can improve later academic outcomes, but longer‐term effects during crises are unknown. This study provides the first experimental evidence of how previously attending a higher‐quality ECE program affected child engagement in remote learning and academic scores during pandemic‐related school closures in Ghana. Children (N = 1668; 50.1% male; M_age = 10.1 years; all Ghanaian nationals) who attended higher‐quality ECE at age 4 or 5 years had greater engagement in remote learning (d = .14) in October 2020, but not better language and literacy and math scores. Previous exposure to higher‐quality ECE may support educational engagement during crises.

Experimental evidence on learning using low-tech when school is out

School closures occurred extensively during the COVID-19 pandemic, and occur in other settings, such as teacher strikes and natural disasters. The cost of school closures has proven to be substantial, particularly for households of lower socioeconomic status, but little evidence exists on how to mitigate these learning losses. This paper provides experimental evidence on strategies to support learning when schools close. We conduct a large-scale randomized trial testing two low-technology interventions-SMS messages and phone calls-with parents to support their child in Botswana. The combined treatment improves learning by 0.12 standard deviations, which translates to 0.89 standard deviations of learning per US$100, ranking among the most cost-effective interventions to improve learning. We develop remote assessment innovations, which show robust learning outcomes. Our findings have immediate policy relevance and long-run implications for the role of technology and parents to support education provision during school disruptions.

Sauti ya Vijana (SYV; The Voice of Youth): Longitudinal Outcomes of an Individually Randomized Group Treatment Pilot Trial for Young People Living with HIV in Tanzania

Sauti ya Vijana is a mental health and life skills intervention delivered by young adult group leaders for the improvement of HIV outcomes in young people living with HIV in Tanzania. This pilot randomized controlled trial estimated exploratory intervention effectiveness compared to standard of care. YPLWH (N = 105) were randomized to receive intervention or SOC. The mean age of participants was 18.1 years and 53% were female. Mean scores on mental health measures (Patient Health Questionnaire [PHQ-9], Strengths and Difficulties Questionnaire [SDQ], UCLA Trauma) were asymptomatic to mild in both study arms through 30-month follow-up with a non-significant fluctuation of 1-2 points. The mean self-reported adherence was higher in the intervention arm across all time points (but the confidence interval contained the null at all time points except 6 months). Risk ratio of virologic suppression (HIV RNA < 400 copies/mL) in the intervention arm compared to SOC was 1.15 [95% CI = 0.95, 1.39]) at 6-months, 1.17 [95% CI: 0.92, 1.48] at 12-months, and 0.99 [95% CI 0.76, 1.31] at 18-months. Though these findings were not powered for statistical significance, the trends in HIV outcomes suggest that SYV holds promise for improving antiretroviral therapy (ART) adherence and virologic suppression in YPLWH.

Potential effects of COVID-19 school closures on foundational skills and Country responses for mitigating learning loss

This article investigates to what extent disrupted schooling and dropout affects children's acquisition of foundational skills prior to and during the COVID-19 pandemic. Using household survey data from thirteen low- and lower-middle-income countries, we find that missing or dropping out of school is associated with lower reading and numeracy outcomes. Drawing on global surveys conducted during the pandemic, we find that countries' remote learning responses are often inadequate to keep all children learning, avoid dropout, and mitigate the learning losses our findings predict, particularly for marginalized children and those at the pre-primary level.

Who do phone surveys miss, and how to reduce exclusion: recommendations from phone surveys in nine Indian states

Computer-assisted telephone interviews (CATI) through mobile phones are a low-cost, rapid and safe way to collect data. However, decisions for how such mobile phone surveys are designed and implemented, and their data analysed, can have implications for the sample reached, and in turn affect the generalisability of sample estimates. In this practice paper, we propose a framework for extending the use of CATI–mobile phone surveys in India, which can be applied broadly to future surveys conducted using this method. Across the stages of design, implementation and analysis, we outline challenges in ensuring that the data collected through such surveys are representative and provide recommendations for reducing non-coverage and non-response errors, thereby enabling practitioners in India to use CATI–mobile phone surveys to estimate population statistics with lower bias. We support our analysis by drawing on primary data that we collected in five mobile phone surveys across nine Indian states in 2020. Our recommendations can help practitioners in India improve the representativeness of data collected through mobile phone surveys and generate more accurate estimates.