A multilevel analysis of the impacts of services provided by the U.K. Employment Retention and Advancement Demonstration

How Three-Arm Random Assignment Within Sites can Improve Non-Experimental Cross-Site Estimates of the Relationship Between Program Characteristics and Impact

This article considers a new method, called Cross-Site Attributional Model Improved by Calibration to Within-Site Individual Randomization Findings (abbreviated as "CAMIC"), which seeks to reduce bias in analyses of the contributions of a program's various design, implementation, and contextual characteristics to its overall impacts. It requires a multi-site experiment where some (or all) sites randomize individuals to one of three arms: a standard treatment group, an enhanced treatment group (that receives the standard treatment plus a program "enhancement"), or a control group (that has no access to the program). A recent evaluation-that of the Health Profession Opportunity Grants (HPOG) program-provides a motivating example of this design and its potential for both methodological and substantive lessons; and the article considers other evaluations that would be fitting for deploying and learning from CAMIC. We conclude that the promise for CAMIC lies in situations where the correlations between the selected program enhancement and alternative program characteristics of interest are relatively high, implying that producing an experimental estimate of the enhancement can reduce bias in the estimation of other non-randomized program characteristics.

Welfare-to-work interventions and their effects on the mental and physical health of lone parents and their children

BACKGROUND

Lone parents in high-income countries have high rates of poverty (including in-work poverty) and poor health. Employment requirements for these parents are increasingly common. 'Welfare-to-work' (WtW) interventions involving financial sanctions and incentives, training, childcare subsidies and lifetime limits on benefit receipt have been used to support or mandate employment among lone parents. These and other interventions that affect employment and income may also affect people's health, and it is important to understand the available evidence on these effects in lone parents.

OBJECTIVES

To assess the effects of WtW interventions on mental and physical health in lone parents and their children living in high-income countries. The secondary objective is to assess the effects of welfare-to-work interventions on employment and income.

SEARCH METHODS

We searched the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE Ovid, Embase Ovid, PsycINFO EBSCO, ERIC EBSCO, SocINDEX EBSCO, CINAHL EBSCO, Econlit EBSCO, Web of Science ISI, Applied Social Sciences Index and Abstracts (ASSIA) via Proquest, International Bibliography of the Social Sciences (IBSS) via ProQuest, Social Services Abstracts via Proquest, Sociological Abstracts via Proquest, Campbell Library, NHS Economic Evaluation Database (NHS EED) (CRD York), Turning Research into Practice (TRIP), OpenGrey and Planex. We also searched bibliographies of included publications and relevant reviews, in addition to many relevant websites. We identified many included publications by handsearching. We performed the searches in 2011, 2013 and April 2016.

SELECTION CRITERIA

Randomised controlled trials (RCTs) of mandatory or voluntary WtW interventions for lone parents in high-income countries, reporting impacts on parental mental health, parental physical health, child mental health or child physical health.

DATA COLLECTION AND ANALYSIS

One review author extracted data using a standardised extraction form, and another checked them. Two authors independently assessed risk of bias and the quality of the evidence. We contacted study authors to obtain measures of variance and conducted meta-analyses where possible. We synthesised data at three time points: 18 to 24 months (T1), 25 to 48 months (T2) and 49 to 72 months (T3).

MAIN RESULTS

Twelve studies involving 27,482 participants met the inclusion criteria. Interventions were either mandatory or voluntary and included up to 10 discrete components in varying combinations. All but one study took place in North America. Although we searched for parental health outcomes, the vast majority of the sample in all included studies were female. Therefore, we describe adult health outcomes as 'maternal' throughout the results section. We downgraded the quality of all evidence at least one level because outcome assessors were not blinded. Follow-up ranged from 18 months to six years. The effects of welfare-to-work interventions on health were generally positive but of a magnitude unlikely to have any tangible effects.At T1 there was moderate-quality evidence of a very small negative impact on maternal mental health (standardised mean difference (SMD) 0.07, 95% Confidence Interval (CI) 0.00 to 0.14; N = 3352; studies = 2)); at T2, moderate-quality evidence of no effect (SMD 0.00, 95% CI 0.05 to 0.05; N = 7091; studies = 3); and at T3, low-quality evidence of a very small positive effect (SMD -0.07, 95% CI -0.15 to 0.00; N = 8873; studies = 4). There was evidence of very small positive effects on maternal physical health at T1 (risk ratio (RR) 0.85, 95% CI 0.54 to 1.36; N = 311; 1 study, low quality) and T2 (RR 1.06, 95% CI 0.95 to 1.18; N = 2551; 2 studies, moderate quality), and of a very small negative effect at T3 (RR 0.97, 95% CI 0.91 to 1.04; N = 1854; 1 study, low quality).At T1, there was moderate-quality evidence of a very small negative impact on child mental health (SMD 0.01, 95% CI -0.06 to 0.09; N = 2762; studies = 1); at T2, of a very small positive effect (SMD -0.04, 95% CI -0.08 to 0.01; N = 7560; studies = 5), and at T3, there was low-quality evidence of a very small positive effect (SMD -0.05, 95% CI -0.16 to 0.05; N = 3643; studies = 3). Moderate-quality evidence for effects on child physical health showed a very small negative effect at T1 (SMD -0.05, 95% CI -0.12 to 0.03; N = 2762; studies = 1), a very small positive effect at T2 (SMD 0.07, 95% CI 0.01 to 0.12; N = 7195; studies = 3), and a very small positive effect at T3 (SMD 0.01, 95% CI -0.04 to 0.06; N = 8083; studies = 5). There was some evidence of larger negative effects on health, but this was of low or very low quality.There were small positive effects on employment and income at 18 to 48 months (moderate-quality evidence), but these were largely absent at 49 to 72 months (very low to moderate-quality evidence), often due to control group members moving into work independently. Since the majority of the studies were conducted in North America before the year 2000, generalisabilty may be limited. However, all study sites were similar in that they were high-income countries with developed social welfare systems.

AUTHORS' CONCLUSIONS

The effects of WtW on health are largely of a magnitude that is unlikely to have tangible impacts. Since income and employment are hypothesised to mediate effects on health, it is possible that these negligible health impacts result from the small effects on economic outcomes. Even where employment and income were higher for the lone parents in WtW, poverty was still high for the majority of the lone parents in many of the studies. Perhaps because of this, depression also remained very high for lone parents whether they were in WtW or not. There is a lack of robust evidence on the health effects of WtW for lone parents outside North America.

Welfare-to-work interventions and their effects on the mental and physical health of lone parents and their children

BACKGROUND

Lone parents in high-income countries have high rates of poverty (including in-work poverty) and poor health. Employment requirements for these parents are increasingly common. 'Welfare-to-work' (WtW) interventions involving financial sanctions and incentives, training, childcare subsidies and lifetime limits on benefit receipt have been used to support or mandate employment among lone parents. These and other interventions that affect employment and income may also affect people's health, and it is important to understand the available evidence on these effects in lone parents.

OBJECTIVES

To assess the effects of WtW interventions on mental and physical health in lone parents and their children living in high-income countries. The secondary objective is to assess the effects of welfare-to-work interventions on employment and income.

SEARCH METHODS

We searched the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE Ovid, Embase Ovid, PsycINFO EBSCO, ERIC EBSCO, SocINDEX EBSCO, CINAHL EBSCO, Econlit EBSCO, Web of Science ISI, Applied Social Sciences Index and Abstracts (ASSIA) via Proquest, International Bibliography of the Social Sciences (IBSS) via ProQuest, Social Services Abstracts via Proquest, Sociological Abstracts via Proquest, Campbell Library, NHS Economic Evaluation Database (NHS EED) (CRD York), Turning Research into Practice (TRIP), OpenGrey and Planex. We also searched bibliographies of included publications and relevant reviews, in addition to many relevant websites. We identified many included publications by handsearching. We performed the searches in 2011, 2013 and April 2016.

SELECTION CRITERIA

Randomised controlled trials (RCTs) of mandatory or voluntary WtW interventions for lone parents in high-income countries, reporting impacts on parental mental health, parental physical health, child mental health or child physical health.

DATA COLLECTION AND ANALYSIS

One review author extracted data using a standardised extraction form, and another checked them. Two authors independently assessed risk of bias and the quality of the evidence. We contacted study authors to obtain measures of variance and conducted meta-analyses where possible. We synthesised data at three time points: 18 to 24 months (T1), 25 to 48 months (T2) and 49 to 72 months (T3).

MAIN RESULTS

Twelve studies involving 27,482 participants met the inclusion criteria. Interventions were either mandatory or voluntary and included up to 10 discrete components in varying combinations. All but one study took place in North America. Although we searched for parental health outcomes, the vast majority of the sample in all included studies were female. Therefore, we describe adult health outcomes as 'maternal' throughout the results section. We downgraded the quality of all evidence at least one level because outcome assessors were not blinded. Follow-up ranged from 18 months to six years. The effects of welfare-to-work interventions on health were generally positive but of a magnitude unlikely to have any tangible effects.At T1 there was moderate-quality evidence of a very small negative impact on maternal mental health (standardised mean difference (SMD) 0.07, 95% Confidence Interval (CI) 0.00 to 0.14; N = 3352; studies = 2)); at T2, moderate-quality evidence of no effect (SMD 0.00, 95% CI 0.05 to 0.05; N = 7091; studies = 3); and at T3, low-quality evidence of a very small positive effect (SMD -0.07, 95% CI -0.15 to 0.00; N = 8873; studies = 4). There was evidence of very small positive effects on maternal physical health at T1 (risk ratio (RR) 0.85, 95% CI 0.54 to 1.36; N = 311; 1 study, low quality) and T2 (RR 1.06, 95% CI 0.95 to 1.18; N = 2551; 2 studies, moderate quality), and of a very small negative effect at T3 (RR 0.97, 95% CI 0.91 to 1.04; N = 1854; 1 study, low quality).At T1, there was moderate-quality evidence of a very small negative impact on child mental health (SMD 0.01, 95% CI -0.06 to 0.09; N = 2762; studies = 1); at T2, of a very small positive effect (SMD -0.04, 95% CI -0.08 to 0.01; N = 7560; studies = 5), and at T3, there was low-quality evidence of a very small positive effect (SMD -0.05, 95% CI -0.16 to 0.05; N = 3643; studies = 3). Moderate-quality evidence for effects on child physical health showed a very small negative effect at T1 (SMD -0.05, 95% CI -0.12 to 0.03; N = 2762; studies = 1), a very small positive effect at T2 (SMD 0.07, 95% CI 0.01 to 0.12; N = 7195; studies = 3), and a very small positive effect at T3 (SMD 0.01, 95% CI -0.04 to 0.06; N = 8083; studies = 5). There was some evidence of larger negative effects on health, but this was of low or very low quality.There were small positive effects on employment and income at 18 to 48 months (moderate-quality evidence), but these were largely absent at 49 to 72 months (very low to moderate-quality evidence), often due to control group members moving into work independently. Since the majority of the studies were conducted in North America before the year 2000, generalisabilty may be limited. However, all study sites were similar in that they were high-income countries with developed social welfare systems.

AUTHORS' CONCLUSIONS

The effects of WtW on health are largely of a magnitude that is unlikely to have tangible impacts. Since income and employment are hypothesised to mediate effects on health, it is possible that these negligible health impacts result from the small effects on economic outcomes. Even where employment and income were higher for the lone parents in WtW, poverty was still high for the majority of the lone parents in many of the studies. Perhaps because of this, depression also remained very high for lone parents whether they were in WtW or not. There is a lack of robust evidence on the health effects of WtW for lone parents outside North America.