Quasi-experimental methods for pharmacoepidemiology: difference-in-differences and synthetic control methods with case studies for vaccine evaluation

Difference-in-differences and synthetic control methods have become common study designs for evaluating the effects of policy changes, including health policies. They also have potential for providing real-world effectiveness and safety evidence in pharmacoepidemiology. To effectively add to the toolkit of the field, however, designs-including both their benefits and drawbacks-must be well understood. Quasi-experimental designs provide an opportunity to estimate the average treatment effect on the treated without requiring the measurement of all possible confounding factors, and to assess population-level effects. This requires, however, other key assumptions, including the parallel trends or stable weighting assumptions, a lack of other concurrent events that could alter time trends, and an absence of contamination between exposed and unexposed units. The targeted estimands are also highly specific to the settings of the study, and combining across units or time periods can be challenging. Case studies are presented for three vaccine evaluation studies, showcasing some of these challenges and opportunities in a specific field of pharmacoepidemiology. These methods provide feasible and valuable sources of evidence in various pharmacoepidemiologic settings and can be improved through research to identify and weigh the advantages and disadvantages in those settings.

Adult Child Schooling and Older Parents' Cognitive Outcomes in the Survey of Health, Aging and Retirement in Europe (SHARE): A Quasi-Experimental Study

A growing body of research suggests that adult child educational attainment benefits older parents' cognitive outcomes via financial (e.g., direct monetary transfers) and nonfinancial (e.g., psychosocial) mechanisms. Quasi-experimental studies are needed to circumvent confounding bias. No such quasi-experimental studies have been completed in higher-income countries, where financial transfers from adult children to aging parents are rare. Using data on 8,159 adults aged ≥50 years in the Survey for Health, Aging and Retirement in Europe (2004/2005), we leveraged changes in compulsory schooling laws as quasi-experiments. Each year of increased schooling among respondents' oldest children was associated with better verbal fluency (β = 0.07, 95% CI: 0.02, 0.12) scores; overall associations with verbal memory scores were null, with mixed and imprecise evidence of association in models stratified by parent gender. We also evaluated associations with psychosocial outcomes as potentially important mechanisms. Increased schooling among respondents' oldest children was associated with higher quality-of-life scores and fewer depressive symptoms. Our findings present modest albeit inconsistent evidence that increases in schooling may have an "upward" influence on older parents' cognitive performance even in settings where financial transfers from adult children to their parents are uncommon. Associations with parents' psychosocial outcomes were more robust.

Evaluating effectiveness of public health intervention strategies for mitigating COVID-19 pandemic

Coronavirus disease 2019 (COVID-19) pandemic is an unprecedented global public health challenge. In the United States (US), state governments have implemented various non-pharmaceutical interventions (NPIs), such as physical distance closure (lockdown), stay-at-home order, mandatory facial mask in public in response to the rapid spread of COVID-19. To evaluate the effectiveness of these NPIs, we propose a nested case-control design with propensity score weighting under the quasi-experiment framework to estimate the average intervention effect on disease transmission across states. We further develop a method to test for factors that moderate intervention effect to assist precision public health intervention. Our method takes account of the underlying dynamics of disease transmission and balance state-level pre-intervention characteristics. We prove that our estimator provides causal intervention effect under assumptions. We apply this method to analyze US COVID-19 incidence cases to estimate the effects of six interventions. We show that lockdown has the largest effect on reducing transmission and reopening bars significantly increase transmission. States with a higher percentage of non-White population are at greater risk of increased R t $$ {R}_t $$ associated with reopening bars.

Synthetic Control Methods for the Evaluation of Single-Unit Interventions in Epidemiology: A Tutorial

Evaluating the impacts of population-level interventions (e.g., changes to state legislation) can be challenging as conducting randomized experiments is often impractical and inappropriate, especially in settings where the intervention is implemented in a single, aggregate unit (e.g., a country or state). A common nonrandomized alternative is to compare outcomes in the treated unit(s) with unexposed controls both before and after the intervention. However, the validity of these designs depends on the use of controls that closely resemble the treated unit on before-intervention characteristics and trends on the outcome, and suitable controls may be difficult to find because the number of potential control regions is typically limited. The synthetic control method provides a potential solution to these problems by using a data-driven algorithm to identify an optimal weighted control unit—a “synthetic control”—based on data from before the intervention from available control units. While popular in the social sciences, the method has not garnered as much attention in health research, perhaps due to a lack of accessible texts aimed at health researchers. We address this gap by providing a comprehensive, nontechnical tutorial on the synthetic control method, using a worked example evaluating Florida’s “stand your ground” law to illustrate methodological and practical considerations.

Accounting for Confounding in Observational Studies

The goal of this review is to enable clinical psychology researchers to more rigorously test competing hypotheses when studying risk factors in observational studies. We argue that there is a critical need for researchers to leverage recent advances in epidemiology/biostatistics related to causal inference and to use innovative approaches to address a key limitation of observational research: the need to account for confounding. We first review theoretical issues related to the study of causation, how causal diagrams can facilitate the identification and testing of competing hypotheses, and the current limitations of observational research in the field. We then describe two broad approaches that help account for confounding: analytic approaches that account for measured traits and designs that account for unmeasured factors. We provide descriptions of several such approaches and highlight their strengths and limitations, particularly as they relate to the etiology and treatment of behavioral health problems.

Evaluating Effectiveness of Public Health Intervention Strategies for Mitigating COVID-19 Pandemic

Coronavirus disease 2019 (COVID-19) pandemic is an unprecedented global public health challenge. In the United States (US), state governments have implemented various non-pharmaceutical interventions (NPIs), such as physical distance closure (lockdown), stay-at-home order, mandatory facial mask in public in response to the rapid spread of COVID-19. To evaluate the effectiveness of these NPIs, we propose a nested case-control design with propensity score weighting under the quasi-experiment framework to estimate the average intervention effect on disease transmission across states. We further develop a method to test for factors that moderate intervention effect to assist precision public health intervention. Our method takes account of the underlying dynamics of disease transmission and balance state-level pre-intervention characteristics. We prove that our estimator provides causal intervention effect under assumptions. We apply this method to analyze US COVID-19 incidence cases to estimate the effects of six interventions. We show that lockdown has the largest effect on reducing transmission and reopening bars significantly increase transmission. States with a higher percentage of non-white population are at greater risk of increased $R_t$ associated with reopening bars.

Opioid use in older adults and Medicare Part D

OBJECTIVE

To determine whether the introduction of prescription drug coverage under Medicare Part D increased opioid prescriptions, patient care-seeking for pain, and pain diagnoses among elderly Medicare-eligible adults.

STUDY SETTING

Office visits by adults aged 18 years or older from the 2000-2016 National Ambulatory Medical Care Survey (12 375 207 253 office visits), and respondents from the 2000-2017 Medical Expenditure Panel Survey (4 023 418 681 individuals).

STUDY DESIGN

We compared care-seeking for pain, provider-assigned pain diagnoses, and opioid prescriptions before and after the Medicare eligibility age of 65, and before and after Part D's implementation using a regression discontinuity, difference-in-differences design. Analyses were adjusted for age, sex, race, and year.

PRINCIPAL FINDINGS

Patient care-seeking for pain increased by 11.4 office visits per 100 people (95% confidence interval 2.0-20.8), or 29%, in response to the implementation of Part D. Opioid prescriptions and diagnoses of pain-related conditions did not change significantly, but the financing of opioid prescriptions shifted from private to public payers at age 65.

CONCLUSIONS

The introduction of Medicare Part D was not associated with increased opioid use among older adults. Rather, opioid use among the elderly has been driven by high levels of opioid use among commercially insured adults who subsequently age into Medicare. Our findings raise the question of whether more judicious prescribing to younger adults coupled with concerted efforts to deprescribe opioids when appropriate may prevent problematic opioid use among the elderly.

Probing the Effective Treatment Thresholds for Alteplase in Acute Ischemic Stroke With Regression Discontinuity Designs

Randomized Controlled Trials (RCTs) are considered the gold standard for measuring the efficacy of medical interventions. However, RCTs are expensive, and use a limited population. Techniques to estimate the effects of stroke interventions from observational data that minimize confounding would be useful. We used regression discontinuity design (RDD), a technique well-established in economics, on the Get With The Guidelines-Stroke (GWTG-Stroke) data set. RDD, based on regression, measures the occurrence of a discontinuity in an outcome (e.g., odds of home discharge) as a function of an intervention (e.g., alteplase) that becomes significantly more likely when crossing the threshold of a continuous variable that determines that intervention (e.g., time from symptom onset, since alteplase is only given if symptom onset is less than e.g., 3 h). The technique assumes that patients near either side of a threshold (e.g., 2.99 and 3.01 h from symptom onset) are indistinguishable other than the use of the treatment. We compared outcomes of patients whose estimated onset to treatment time fell on either side of the treatment threshold for three cohorts of patients in the GWTG-Stroke data set. This data set spanned three different treatment thresholds for alteplase (3 h, 2003-2007, N = 1,869; 3 h, 2009-2016, N = 13,086, and 4.5 h, 2009-2016, N = 6,550). Patient demographic characteristics were overall similar across the treatment thresholds. We did not find evidence of a discontinuity in clinical outcome at any treatment threshold attributable to alteplase. Potential reasons for failing to find an effect include violation of some RDD assumptions in clinical care, large sample sizes required, or already-well-chosen treatment threshold.

Caesarean section and children's health: A quasi-experimental design

The prevalence of inflammatory child health conditions-such as asthma, eczema, and food allergy-and their associated costs have increased rapidly over the last 30 years. While environmental factors likely underpin these increases, recent studies explain only a fraction of the trend and rely on associational methods. Caesarean (or C-) section rates increased dramatically in the period of interest, and this method of delivery is an understudied environmental factor linked to child health outcomes via the gut microbiome. We fuse 22 years of birth cohort data from the United States National Surveys of Children's Health with C-section rates from the National Vital Statistics System generated for subgroups based on state, sex, race, Hispanic origin, and birth year. Then, we model the effects of C-section on rates of asthma, eczema, and food allergy using a quasi-experimental fixed effects design. We find that C-section significantly predicts food allergy, with qualitatively significant implications.

The short-term effects of the earned income tax credit on health care expenditures among US adults

OBJECTIVE

To test the hypothesis that the earned income tax credit (EITC)-the largest US poverty alleviation program-affects short-term health care expenditures among US adults.

DATA SOURCES

Adult participants in the 1997-2012 waves of the US Medical Expenditure Panel Survey (MEPS) (N = 1 282 080).

STUDY DESIGN

We conducted difference-in-differences analyses, comparing health care expenditures among EITC-eligible adults in February (immediately following EITC refund receipt) with expenditures during other months, using non-EITC-eligible individuals to difference out seasonal variation in health care expenditures. Outcomes included total out-of-pocket expenditures as well as spending on specific categories such as outpatient visits and inpatient hospitalizations. We conducted subgroup analyses to examine heterogeneity by insurance status.

PRINCIPAL FINDINGS

EITC refund receipt was not associated with short-term changes in total expenditures, nor any expenditure subcategories. Results were similar by insurance status and robust to numerous alternative specifications.

CONCLUSIONS

EITC refunds are not associated with short-term changes in health care expenditures among US adults. This may be because the refund is spent on other expenses, because of income smoothing, or because of similar refund-related variation in health care expenditures among noneligible adults. Future studies should examine whether other types of income supplementation affect health care expenditures, particularly among individuals in poverty.

A Future for Observational Epidemiology: Clarity, Credibility, Transparency

Observational studies are ambiguous, difficult, and necessary for epidemiology. Presently, there are concerns that the evidence produced by most observational studies in epidemiology is not credible and contributes to research waste. I argue that observational epidemiology could be improved by focusing greater attention on 1) defining questions that make clear whether the inferential goal is descriptive or causal; 2) greater utilization of quantitative bias analysis and alternative research designs that aim to decrease the strength of assumptions needed to estimate causal effects; and 3) promoting, experimenting with, and perhaps institutionalizing both reproducible research standards and replication studies to evaluate the fragility of study findings in epidemiology. Greater clarity, credibility, and transparency in observational epidemiology will help to provide reliable evidence that can serve as a basis for making decisions about clinical or population-health interventions.

Assessing Correspondence Between Experimental and Nonexperimental Estimates in Within-Study Comparisons

In within-study comparison (WSC) designs, treatment effects from a nonexperimental design, such as an observational study or a regression-discontinuity design, are compared to results obtained from a well-designed randomized control trial with the same target population. The goal of the WSC is to assess whether nonexperimental and experimental designs yield the same results in field settings. A common analytic challenge with WSCs, however, is the choice of appropriate criteria for determining whether nonexperimental and experimental results replicate. This article examines different distance-based correspondence measures for assessing correspondence in experimental and nonexperimental estimates. Distance-based measures investigate whether the difference in estimates is small enough to claim equivalence of methods. We use a simulation study to examine the statistical properties of common correspondence measures and recommend a new and straightforward approach that combines traditional significance testing and equivalence testing in the same framework. The article concludes with practical advice on assessing and interpreting results in WSC contexts.

Threats to the Internal Validity of Experimental and Quasi-Experimental Research in Healthcare

The article defines, describes, and discusses the seven threats to the internal validity of experiments discussed by Donald T. Campbell in his classic 1957 article: history, maturation, testing, instrument decay, statistical regression, selection, and mortality. These concepts are said to be threats to the internal validity of experiments because they pose alternate explanations for the apparent causal relationship between the independent variable and dependent variable of an experiment if they are not adequately controlled. A series of simple diagrams illustrate three pre-experimental designs and three true experimental designs discussed by Campbell in 1957 and several quasi-experimental designs described in his book written with Julian C. Stanley in 1966. The current article explains why each design controls for or fails to control for these seven threats to internal validity.

Can't Count or Won't Count? Embedding Quantitative Methods in Substantive Sociology Curricula: A Quasi-Experiment

This paper reports on a quasi-experiment in which quantitative methods (QM) are embedded within a substantive sociology module. Through measuring student attitudes before and after the intervention alongside control group comparisons, we illustrate the impact that embedding has on the student experience. Our findings are complex and even contradictory. Whilst the experimental group were less likely to be distrustful of statistics and appreciate how QM inform social research, they were also less confident about their statistical abilities, suggesting that through 'doing' quantitative sociology the experimental group are exposed to the intricacies of method and their optimism about their own abilities is challenged. We conclude that embedding QM in a single substantive module is not a 'magic bullet' and that a wider programme of content and assessment diversification across the curriculum is preferential.

Is Self-Regulation "All in the family"? Testing Environmental Effects using Within-Family Quasi-Experiments

Most of the individual difference variance in the population is found within families, yet studying the processes causing this variation is difficult due to confounds between genetic and nongenetic influences. Quasi-experiments can be used to test hypotheses regarding environment exposure (e.g., timing, duration) while controlling for genetic confounds. To illustrate, two studies of cognitive self-regulation in childhood (i.e., working memory [WM], effortful control [EC], attention span/persistence [A/P]) are presented. Study 1 utilized an identical twin differences design (N = 85 to 98 pairs) to control for genetic differences while using relative twin birth weight difference to predict relative twin difference in WM and EC. Larger relative twin difference in WM and EF was predicted by the combination of shorter gestation and larger relative birth weight difference. Study 2 utilized an adoptive sibling relative difference design (N = 123 same-sex pairs) to control for genetic similarity while using relative sibling difference in the age at time of adoption to predict relative sibling difference in A/P. Larger relative sibling difference in A/P was predicted by the combination of larger relative difference in time in the adoptive home and age at adoption. Within-family quasi-experimental designs allow stronger inferences about hypothesized environmental influences than between-family designs permit.

Aircraft noise, health, and residential sorting: evidence from two quasi-experiments

We explore two unexpected changes in flight regulations to estimate the causal effect of aircraft noise on health. Detailed measures of noise are linked with longitudinal data on individual health outcomes based on the exact address information. Controlling for individual heterogeneity and spatial sorting into different neighborhoods, we find that aircraft noise significantly increases sleeping problems and headaches. Models that do not control for such heterogeneity and sorting substantially underestimate the negative health effects, which suggests that individuals self-select into residence based on their unobserved sensitivity to noise. Our study demonstrates that the combination of quasi-experimental variation and panel data is very powerful for identifying causal effects in epidemiological field studies.

All in the Family: Comparing Siblings to Test Causal Hypotheses Regarding Environmental Influences on Behavior

Psychologists in both basic and applied fields are keenly interested in the environmental influences that shape our lives. Therefore, researchers test causal hypotheses to construct models of environmental influences that can withstand attempts at refutation. Randomized experiments provide the strongest tests of causal hypotheses, but are not always feasible and their assumptions cannot always be met. In such cases, a number of quasi-experimental research designs can be used to substantially reduce confounding in tests of causal hypotheses. Sibling-comparison designs provide robust quasi-experimental tests of causal environmental hypotheses, but they are underused in psychology in spite of their power, feasibility, and convenience.

Maternal age at childbirth and offspring disruptive behaviors: testing the causal hypothesis

BACKGROUND

Recent studies suggest that the association between maternal age at childbearing (MAC) and children's disruptive behaviors is the result of family factors that are confounded with both variables, rather than a casual effect of environmental factors specifically related to MAC. These studies, however, relied on restricted samples and did not use the strongest approach to test causal influences.

METHODS

Using data on 9,171 4-9-year-old and 6,592 10-13-year-old offspring of women from a nationally representative sample of US households, we conducted sibling-comparison analyses. The analyses ruled out all genetic factors that could confound the association, as well as all environmental confounds that differ between unrelated nuclear families, providing a strong test of the causal hypothesis that the environments of children born at different maternal ages influence mother- and self-reported disruptive behaviors.

RESULTS

When these genetic and environmental confounds were ruled out as alternative explanations, the relation between environments within nuclear families specifically associated with MAC and disruptive behaviors was robust, with the association being stronger for second- and third-born children.

CONCLUSIONS

Environmental factors specifically associated with early MAC within nuclear families account for increased risk of offspring disruptive behaviors, especially in later-born children.