Heterogeneous treatment effects in social policy studies: An assessment of contemporary articles in the health and social sciences

Port Cleaner Trucks and Environmental Justice in the Greater New York Area

This paper examines local air pollution levels in disadvantaged neighborhoods near ports after the adoption of cleaner diesel trucks under the Port Drayage Truck Replacement Programs (TRPs). Using annual air pollutant data and census demographics at the tract level, we estimate changes in air quality through a Difference-in-Differences-in-Differences (DDD) model. Our findings indicate that the impact of TRPs was mixed: near-port Black/African American communities were likely worse off, while near-port Hispanic/Latinos saw improvements. Since 2010, air quality improvements in Greater New York (GNY) were 4.5% greater than in near-port communities. Black-majority tracts near ports experienced 6.4% higher NO2 than Black-majority tracts in GNY. Within near-port areas, Black-majority tracts had 2.7% higher NO2 than non-Black-majority tracts. The relative gap between Black and non-Black populations in near-port areas widened by about 2% compared to GNY. Conversely, near-port Hispanic-majority tracts had NO2 levels similar to GNY Black-majority and near-port non-Hispanic-majority areas, with the relative gap between Hispanic and non-Hispanic populations in near-port areas narrowing by 4% compared to GNY. These disparities highlight the disproportionate benefits and drawbacks of TRPs, underscoring the need for more targeted interventions. To address these inequities, we recommend an integrated policy approach, including interstate collaboration to standardize truck regulations, stricter on-road emission controls in near-port areas, and accelerated adoption of clean technologies like electric and hydrogen fuel cell vehicles. These actions could reduce environmental health disparities and promote environmental justice by addressing the systemic vulnerabilities of near-port communities.

Poverty and Childhood Obesity: Current Evidence and Methodologies for Future Research

PURPOSE OF REVIEW

This narrative review summarizes current knowledge on the link between poverty and childhood obesity, and then describes conventional and modern epidemiologic methods for causal inference that may help provide more robust evidence on how poverty reduction can prevent childhood obesity.

RECENT FINDINGS

Household poverty has been consistently associated with increased risk of childhood obesity across observational studies in industrialized countries. Due to ethical and feasibility limitations, few randomized controlled trials directly test the effect of poverty reduction. A growing number of studies use quasi-experimental methods to study the effects of poverty reduction policies on childhood obesity. These methods include instrumental variables, difference-in-differences, interrupted time series analysis, and regression discontinuity. Other complementary methods such as causal mediation analysis allow us to elucidate the mechanisms of how poverty reduction affects childhood obesity outcomes, while examining heterogeneous treatment effects using cutting-edge machine learning algorithms may further identify subpopulations that benefit the most from poverty interventions. Despite the strong associations between poverty and childhood obesity observed in industrialized countries, current evidence about the causal effect of poverty reduction on childhood obesity is mixed. This is likely due to the complex etiology of childhood obesity and potentially unintended effects of policies. Future studies that leverage advances in causal inference with quasi-experimental approaches will help provide more robust evidence to help guide practitioners and policymakers in ongoing childhood obesity prevention efforts.

Evidence for Public Policies to Prevent Suicide Death in the United States

Suicide rates have increased in the United States in recent years. Public policies have great potential to prevent suicide death, and well-designed quasi-experimental studies have identified policies that are effective at reducing suicide rates; however, evidence about these policies has not been synthesized. This review summarizes evidence across three domains of public policies: (a) policies that affect structural determinants of suicide risk (e.g., policies that improve economic security), (b) policies that promote access to clinical services (e.g., Medicaid expansion), and (c) policies that limit access to lethal means for completing suicide (e.g., policies that restrict access to firearms). The historical context of suicide prevention in US public policy is provided, considerations for successful suicide prevention policy implementation are discussed-such as policy awareness among key groups, enforcement, and sufficient funding-and priority areas for future research are enumerated.

Early childhood education and handgun carrying, serious fighting, and assault charges: a retrospective cohort study

High-quality early childhood education may buffer against social and structural drivers of interpersonal violence. We examined the association of Head Start-a large-scale early childhood education program for low-income children, launched in 1965 as part of the War on Poverty-with handgun carrying, serious fighting, and assault charges among 4281 individuals born between 1980 and 1984 in the National Longitudinal Survey of Youth 1997. We found that attending Head Start vs other childcare was associated with 0.77 times the risk of handgun carrying by age 31 (95% CI: 0.60, 0.99) and 0.79 times the risk of serious fighting by age 24 (95% CI: 0.64, 0.98) among Black males. No reduction in the risk of outcomes was observed among other subpopulations or when comparing Head Start with solely parental childcare. Expanding access to high-quality early childhood education programs may reduce later-life handgun carrying and serious fighting among those at highest risk, thus reducing racialized disparities. Results suggest that early-life investments in the social, economic, and human capital of structurally disadvantaged children and families may be effective and equitable tools to prevent violence and firearm-related harms.

Two-step pragmatic subgroup discovery for heterogeneous treatment effects analyses: perspectives toward enhanced interpretability

Effect heterogeneity analyses using causal machine learning algorithms have gained popularity in recent years. However, the interpretation of estimated individualized effects requires caution because insights from these data-driven approaches might be misaligned with the contextual needs of a human audience. Thus, a practical framework that integrates advanced machine learning methods and decision-making remains critically needed to achieve effective implementation and scientific communication. We introduce a 2-step framework to identify characteristics associated with substantial effect heterogeneity in a practically relevant format. The proposed framework applies distinct sets of covariates for (i) estimation of individualized effects and (ii) subgroup discovery and shows the subgroups with heterogeneity based on highly interpretable if-then rules. By referring to existing metrics of interpretability, we describe how each step contributes to leveraging a theoretical advantage of machine learning models while creating an interpretable and practically relevant framework. We applied the pragmatic subgroup discovery framework for the Look AHEAD (Action for Health in Diabetes) trial to assess practically relevant and comprehensive insights into the effect heterogeneities of intense lifestyle intervention for individuals with diabetes on cardiovascular mortality. Our analysis identified (i) individuals with history of cardiovascular disease and myocardial infarction had the least benefit from the intervention, while (ii) individuals with no history of cardiovascular diseases and HbA1c < 7% received the highest benefit. In summary, our practical framework for heterogeneous effects discovery could be a generic strategy to ensure both effective implementation and scientific communication when applying machine learning algorithms in epidemiological research.

Assessing Links Between Alcohol Exposure and Firearm Violence: A Scoping Review Update

BACKGROUND

Firearm violence remains a leading cause of death and injury in the United States. Prior research supports that alcohol exposures, including individual-level alcohol use and alcohol control policies, are modifiable risk factors for firearm violence, yet additional research is needed to support prevention efforts.

OBJECTIVES

This scoping review aims to update a prior 2016 systematic review on the links between alcohol exposure and firearm violence to examine whether current studies indicate causal links between alcohol use, alcohol interventions, and firearm violence-related outcomes.

ELIGIBILITY CRITERIA

Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines, a comprehensive search of published studies was conducted, replicating the search strategy of the prior review but focusing on studies published since 2015. The review included published studies of humans, conducted in general populations of any age, gender, or racial/ethnic group, that examined the relationship between an alcohol-related exposure and an outcome involving firearm violence or risks for firearm violence. Excluded were small studies restricted to special populations, forensic or other technical studies, non-original research articles such as reviews, and studies that relied solely on descriptive statistics or did not adjust for confounders.

SOURCES OF EVIDENCE

The review included published studies indexed in PubMed, Web of Science, and Scopus. Eligible articles were published on or after January 1, 2015. The latest search was conducted on December 15, 2023.

CHARTING METHODS

Using a structured data collection instrument, data were extracted on the characteristics of each study, including the dimension of alcohol exposure, the dimension of firearm violence, study population, study design, statistical analysis, source of funding, main findings, and whether effect measure modification was assessed and, if so, along what dimensions. Two authors independently conducted title/abstract screening, full-text screening, and data extraction until achieving 95% agreement, with discrepancies resolved through discussion.

RESULTS

The search yielded 797 studies. Of these, 754 were excluded and 43 met the final inclusion criteria. Studies addressed a range of alcohol exposures and firearm violence-related outcomes, primarily with cross-sectional study designs; 40% considered effect measure modification by any population characteristic. Findings from the 21 studies examining the relationship of individual-level alcohol use or alcohol use disorder (AUD) with firearm ownership, access, unsafe storage, or carrying indicated a strong and consistent positive association. Seven studies examined associations of individual-level alcohol use or AUD with firearm injury or death; these also indicated a pattern of positive associations, but the magnitude and precision of the estimates varied. Eight studies examined the impact of neighborhood proximity or density of alcohol outlets and found mixed results that were context- and study design-dependent. Two studies linked prior alcohol-related offenses to increased risk of firearm suicide and perpetration of violent firearm crimes among a large cohort of people who purchased handguns, and two studies linked policies prohibiting firearm access among individuals with a history of alcohol-related offenses to reductions in firearm homicide and suicide. Finally, four studies examined alcohol control policies and found that greater restrictiveness was generally associated with reductions in firearm homicide or firearm suicide.

CONCLUSIONS

Findings from this scoping review continue to support a causal relationship between alcohol exposures and firearm violence that extends beyond acute alcohol use to include AUD and alcohol-related policies. Policies controlling the availability of alcohol and prohibiting firearm access among individuals with alcohol-related offense histories show promise for the prevention of firearm violence. Additional research examining differential impacts by population subgroup, alcohol use among perpetrators of firearm violence, policies restricting alcohol outlet density, and randomized or quasi-experimental study designs with longitudinal follow-up would further support inferences to inform prevention efforts.

Primer on Health Equity Research in Health Economics and Outcomes Research: An ISPOR Special Interest Group Report

OBJECTIVES

Disparities in health and healthcare between more and less socially advantaged groups are pervasive, multidimensional, and far-reaching. The material and social conditions in which people are born, grow, work, live, and age are systematically associated with their health and with the volume, quality, and outcomes of care received by the vast majority of the general population, as well as by specific marginalized populations. The field of health economics and outcomes research (HEOR) has an important role in supporting health equity goals. This publication aimed to act as a "primer" for conducting health equity research within the field of HEOR, establishing foundational understanding of key concepts.

METHODS

The ISPOR Special Interest Group on Health Equity Research was established in 2021 to advance equity-informative methods and data to better enable researchers to empirically investigate-and ultimately reduce-unfair social differences in health. This publication was developed by the ISPOR Special Interest Group leadership team with input from the group membership.

RESULTS

The resultant publication provides an overview of health equity research methods and data considerations as they relate to HEOR-relevant topics including clinical trials, real-world evidence and economic evaluation. Reflecting the current body of research on health equity in HEOR, particular focus is given to the latter. It also brings together a list of core reference material to support future learning.

CONCLUSIONS

This report provides the HEOR community with a tailored "state of play" overview of health equity, to support development of foundational understanding and inspire increased engagement.

Machine learning for detection of heterogeneous effects of Medicaid coverage on depression

In 2008, Oregon expanded its Medicaid program using a lottery, creating a rare opportunity to study the effects of Medicaid coverage using a randomized controlled design (Oregon Health Insurance Experiment). Analysis showed that Medicaid coverage lowered the risk of depression. However, this effect may vary between individuals, and the identification of individuals likely to benefit the most has the potential to improve the effectiveness and efficiency of the Medicaid program. By applying the machine learning causal forest to data from this experiment, we found substantial heterogeneity in the effect of Medicaid coverage on depression; individuals with high predicted benefit were older and had more physical or mental health conditions at baseline. Expanding coverage to individuals with high predicted benefit generated greater reduction in depression prevalence than expanding to all eligible individuals (21.5 vs 8.8 percentage-point reduction; adjusted difference = +12.7 [95% CI, +4.6 to +20.8]; P = 0.003), at substantially lower cost per case prevented ($16 627 vs $36 048; adjusted difference = -$18 598 [95% CI, -156 953 to -3120]; P = 0.04). Medicaid coverage reduces depression substantially more in a subset of the population than others, in ways that are predictable in advance. Targeting coverage on those most likely to benefit could improve the effectiveness and efficiency of insurance expansion. This article is part of a Special Collection on Mental Health.

Considerations for Subgroup Analyses in Cluster-Randomized Trials Based on Aggregated Individual-Level Predictors

In research assessing the effect of an intervention or exposure, a key secondary objective often involves assessing differential effects of this intervention or exposure in subgroups of interest; this is often referred to as assessing effect modification or heterogeneity of treatment effects (HTE). Observed HTE can have important implications for policy, including intervention strategies (e.g., will some patients benefit more from intervention than others?) and prioritizing resources (e.g., to reduce observed health disparities). Analysis of HTE is well understood in studies where the independent unit is an individual. In contrast, in studies where the independent unit is a cluster (e.g., a hospital or school) and a cluster-level outcome is used in the analysis, it is less well understood how to proceed if the HTE analysis of interest involves an individual-level characteristic (e.g., self-reported race) that must be aggregated at the cluster level. Through simulations, we show that only individual-level models have power to detect HTE by individual-level variables; if outcomes must be defined at the cluster level, then there is often low power to detect HTE by the corresponding aggregated variables. We illustrate the challenges inherent to this type of analysis in a study assessing the effect of an intervention on increasing COVID-19 booster vaccination rates at long-term care centers.

Verbal Learning and Memory Deficits across Neurological and Neuropsychiatric Disorders: Insights from an ENIGMA Mega Analysis

Deficits in memory performance have been linked to a wide range of neurological and neuropsychiatric conditions. While many studies have assessed the memory impacts of individual conditions, this study considers a broader perspective by evaluating how memory recall is differentially associated with nine common neuropsychiatric conditions using data drawn from 55 international studies, aggregating 15,883 unique participants aged 15-90. The effects of dementia, mild cognitive impairment, Parkinson's disease, traumatic brain injury, stroke, depression, attention-deficit/hyperactivity disorder (ADHD), schizophrenia, and bipolar disorder on immediate, short-, and long-delay verbal learning and memory (VLM) scores were estimated relative to matched healthy individuals. Random forest models identified age, years of education, and site as important VLM covariates. A Bayesian harmonization approach was used to isolate and remove site effects. Regression estimated the adjusted association of each clinical group with VLM scores. Memory deficits were strongly associated with dementia and schizophrenia (p < 0.001), while neither depression nor ADHD showed consistent associations with VLM scores (p > 0.05). Differences associated with clinical conditions were larger for longer delayed recall duration items. By comparing VLM across clinical conditions, this study provides a foundation for enhanced diagnostic precision and offers new insights into disease management of comorbid disorders.

Understanding the roles of state demographics and state policies in epidemiologic studies of maternal-child health disparities

Disparities in maternal-child health outcomes by race and ethnicity highlight structural differences in the opportunity for optimal health in the United States. Examples of these differences include access to state-level social policies that promote maternal-child health. States vary in their racial and ethnic composition as a result of the complex history of policies and laws related to slavery, Indigenous genocide and relocation, segregation, immigration, and settlement in the United States. States also vary in the social policies they enact. As a result, correlations exist between the demographic makeup of a state's population and the presence or absence of social policies in that state. These correlations become a mechanism by which racial and ethnic disparities in maternal-child health outcomes can operate. In this commentary, we use the example of 3 labor-related policies actively under consideration at state and federal levels (paid parental leave, paid sick leave, and reasonable accommodations during pregnancy) to demonstrate how correlations between state demographics and presence of these state policies could cause or exacerbate racial and ethnic disparities in maternal-child health outcomes. We conclude with a call for researchers to consider how the geographic distribution of racialized populations and state policies could contribute to maternal-child health disparities.

Harnessing causal forests for epidemiologic research: key considerations

Assessing heterogeneous treatment effects (HTEs) is an essential task in epidemiology. The recent integration of machine learning into causal inference has provided a new, flexible tool for evaluating complex HTEs: causal forest. In a recent paper, Jawadekar et al (Am J Epidemiol. 2023;192(7):1155-1165) introduced this innovative approach and offered practical guidelines for applied users. Building on their work, this commentary provides additional insights and guidance to promote the understanding and application of causal forest in epidemiologic research. We start with conceptual clarifications, differentiating between honesty and cross-fitting, and exploring the interpretation of estimated conditional average treatment effects. We then delve into practical considerations not addressed by Jawadekar et al, including motivations for estimating HTEs, calibration approaches, and ways to leverage causal forest output with examples from simulated data. We conclude by outlining challenges to consider for future advancements and applications of causal forest in epidemiologic research.

Sample size considerations for assessing treatment effect heterogeneity in randomized trials with heterogeneous intracluster correlations and variances

An important consideration in the design and analysis of randomized trials is the need to account for outcome observations being positively correlated within groups or clusters. Two notable types of designs with this consideration are individually randomized group treatment trials and cluster randomized trials. While sample size methods for testing the average treatment effect are available for both types of designs, methods for detecting treatment effect modification are relatively limited. In this article, we present new sample size formulas for testing treatment effect modification based on either a univariate or multivariate effect modifier in both individually randomized group treatment and cluster randomized trials with a continuous outcome but any types of effect modifier, while accounting for differences across study arms in the outcome variance, outcome intracluster correlation coefficient (ICC) and the cluster size. We consider cases where the effect modifier can be measured at either the individual level or cluster level, and with a univariate effect modifier, our closed-form sample size expressions provide insights into the optimal allocation of groups or clusters to maximize design efficiency. Overall, our results show that the required sample size for testing treatment effect heterogeneity with an individual-level effect modifier can be affected by unequal ICCs and variances between arms, and accounting for such between-arm heterogeneity can lead to more accurate sample size determination. We use simulations to validate our sample size formulas and illustrate their application in the context of two real trials: an individually randomized group treatment trial (the AWARE study) and a cluster randomized trial (the K-DPP study).

Testing for intersectional measurement invariance with the alignment method: Evaluation of the 8-item patient health questionnaire

OBJECTIVE

To demonstrate the use of the alignment method to evaluate whether surveys function similarly (i.e., have evidence of measurement invariance) across culturally diverse intersectional groups. Intersectionality theory recognizes the interconnected nature of social categories such as race, gender, ethnicity, and socioeconomic status.

DATA SOURCES

A total of 30,215 American adult's responses to the eight-item Patient Health Questionnaire depression assessment scale (PHQ-8) from the 2019 National Health Interview Survey (NHIS).

STUDY DESIGN

Using the alignment method, we examined the measurement invariance (equivalence) of the PHQ-8 depression assessment scale across 16 intersectional subgroups defined at the intersection of age (under 52, 52 and older), gender (male, female), race (Black, non-Black), and education (no bachelor's degree, bachelor's degree).

PRINCIPAL FINDINGS

Overall, 24% of the factor loadings and 5% of the item intercepts showed evidence of differential functioning across one or more of the intersectional groups. These levels fall beneath the benchmark of 25% suggested for determining measurement invariance with the alignment method.

CONCLUSIONS

The results of the alignment study suggest that the PHQ-8 functions similarly across the intersectional groups examined, despite some evidence of different factor loadings and item intercepts in some groups (i.e., noninvariance). By examining measurement invariance through an intersectional lens, researchers can investigate how a person's multiple identities and social positions possibly contribute to their response behavior on an assessment scale.

A quantitative assessment of the frequency and magnitude of heterogeneous treatment effects in studies of the health effects of social policies

Substantial heterogeneity in effects of social policies on health across subgroups may be common, but has not been systematically characterized. Using a sample of 55 contemporary studies on health effects of social policies, we recorded how often heterogeneous treatment effects (HTEs) were assessed, for what subgroups (e.g., male, female), and the subgroup-specific effect estimates expressed as Standardized Mean Differences (SMDs). For each study, outcome, and dimension (e.g., gender), we fit a random-effects meta-analysis. We characterized the magnitude of heterogeneity in policy effects using the standard deviation of the subgroup-specific effect estimates (τ). Among the 44% of studies reporting subgroup-specific estimates, policy effects were generally small (<0.1 SMDs) with mixed impacts on health (67% beneficial) and disparities (50% implied narrowing of disparities). Across study-outcome-dimensions, 54% indicated any heterogeneity in effects, and 20% had τ > 0.1 SMDs. For 26% of study-outcome-dimensions, the magnitude of τ indicated that effects of opposite signs were plausible across subgroups. Heterogeneity was more common in policy effects not specified a priori. Our findings suggest social policies commonly have heterogeneous effects on health of different populations; these HTEs may substantially impact disparities. Studies of social policies and health should routinely evaluate HTEs.

Accounting for expected attrition in the planning of cluster randomized trials for assessing treatment effect heterogeneity

BACKGROUND

Detecting treatment effect heterogeneity is an important objective in cluster randomized trials and implementation research. While sample size procedures for testing the average treatment effect accounting for participant attrition assuming missing completely at random or missing at random have been previously developed, the impact of attrition on the power for detecting heterogeneous treatment effects in cluster randomized trials remains unknown.

METHODS

We provide a sample size formula for testing for a heterogeneous treatment effect assuming the outcome is missing completely at random. We also propose an efficient Monte Carlo sample size procedure for assessing heterogeneous treatment effect assuming covariate-dependent outcome missingness (missing at random). We compare our sample size methods with the direct inflation method that divides the estimated sample size by the mean follow-up rate. We also evaluate our methods through simulation studies and illustrate them with a real-world example.

RESULTS

Simulation results show that our proposed sample size methods under both missing completely at random and missing at random provide sufficient power for assessing heterogeneous treatment effect. The proposed sample size methods lead to more accurate sample size estimates than the direct inflation method when the missingness rate is high (e.g., ≥ 30%). Moreover, sample size estimation under both missing completely at random and missing at random is sensitive to the missingness rate, but not sensitive to the intracluster correlation coefficient among the missingness indicators.

CONCLUSION

Our new sample size methods can assist in planning cluster randomized trials that plan to assess a heterogeneous treatment effect and participant attrition is expected to occur.

Assessment of the predictive power of a causal variable: An application to the Head Start impact study

In a study attempting to estimate a causal effect of a causal variable, an assessment of the predictive power of the causal variable can shed light on the heterogeneity around its average effect. Using data from the Head Start Impact Study, a randomized controlled trial of the Head Start, a nation-wide early childhood education program in the United States, we provide a parallel comparison between measures of average effect and predictive power of the Head Start on five cognitive outcomes. We observed that one year of the Head Start increased scores for all five outcomes, with effect sizes ranging from 0.12 to 0.19 standard deviations. Percent variation explained by the Head Start ranged from 0.56 to 1.62%. For binary versions of the outcomes, the overall pattern remained; the Head Start on average improved the outcomes by meaningful magnitudes. In contrast, in a fully adjusted model, the Head Start only improved area under the curve (AUC) by less than 1% and its influence on the variance of predicted probabilities was negligible. The Head-Start-only model only achieved AUC ranging from 50.22 to 55.24%. Negligible predictive power despite the significant average effect suggests that the heterogeneity in effects may be large. The average effect estimates may not generalize well to different populations or different Head Start program settings. Assessment of the predictive power of a causal variable in randomized data should be a routine practice as it can provide helpful information on the causal effect and especially its heterogeneity.