Can synthetic controls improve causal inference in interrupted time series evaluations of public health interventions?

Evaluating the effects of the 2017 National Health Insurance coverage expansion on amenable mortality and its disparities between areas in South Korea using Bayesian structural time-series models

To improve the low coverage rate of the National Health Insurance (NHI), South Korea implemented the NHI coverage expansion plan in 2017 to cover medically essential non-covered services and reduce copayment rates. This study aimed to estimate the effects of the 2017 NHI coverage expansion on amenable mortality and its disparities between areas in South Korea under a controlled interrupted time-series design using Bayesian structural time-series models. Age-standardized amenable mortality rates and rate differences (RDs) and rate ratios (RRs) between areas for amenable mortality were calculated monthly between July 2012 and December 2021 and used as the response series. The non-equivalent control series were monthly non-avoidable mortality rates and their regional disparities. After the coverage expansion, amenable mortality rates decreased for both males (-8.8%, 95% credible interval [CrI] -13.4% to -3.9%) and females (-8.3%, 95% CrI -13.4% to -2.4%), with the largest decline in the non-Seoul-Capital metropolitan area (-11.6%, 95% CrI -16.5% to -6.3%) rather than the Seoul Capital Area (-7.5%, 95% CrI -11.9% to -2.5%) and a non-significant reduction in the non-Seoul-Capital non-metropolitan area in females. RDs and RRs between areas for amenable mortality decreased non-significantly (-16.2%, 95% CrI -31.3% to 2.6% for RD and -1.2%, 95% CrI -3.7% to 1.5% for RR), except for a significant decrease in RD in males (-21.8%, 95% CrI -38.0% to -1.5%), and decreased less in females than in males. The coverage expansion was generally effective in reducing amenable mortality rates by area, but had limited effects in closing amenable mortality disparities between areas, favoring males and the non-Seoul-Capital metropolitan area. These results implied that additional measures are necessary to improve access to quality health care for females and underserved areas to enhance the effectiveness of the coverage expansion.

How to Use Quasi-Experimental Methods in Cardiovascular Research: A Review of Current Practice

BACKGROUND

Quasi-experimental methods (QEMs) are a family of techniques used to estimate causal relationships when randomized controlled trials are unfeasible or unethical. They offer a powerful alternative to observational studies by introducing random assignment of individuals or groups into their design, thereby offering stronger means of establishing causation. The use of QEMs in cardiovascular research has not been systematically examined to determine steps toward improving and expanding their use.

METHODS

We identified 4 main techniques using a systematic search strategy from 2016 to 2021: instrumental variable analysis, interrupted time series analysis, difference-in-differences analysis, and regression discontinuity designs. QEMs are examined as alternatives to randomized controlled trials and traditional observational studies; as more observational data becomes available to researchers, there are more opportunities to apply these techniques. Eligible articles were selected based on publication in high-ranked journals. The quality of eligible articles was appraised using the Joanna Briggs Institute checklist for quasi-experimental studies.

RESULTS

Data from 380 studies were extracted based on our inclusion criteria. Forty-two of these studies were published in the top 10 medical or top 20 cardiovascular disease journals, and 25 studies were included after quality appraisal. The review identifies the main features and limitations associated with each technique, providing readers with practical guidance on how to apply these to their research. A graphical decision aid was developed to facilitate the routine use of QEMs.

CONCLUSIONS

The use of QEMs in cardiovascular research published in contemporary, high-impact articles was examined. Findings are biased toward this segment of literature, which represents the latest developments in this growing area of cardiovascular research. The decision aid is a novel schematic that researchers can adopt into practice.

Construction and effectiveness evaluation of a knowledge-based infectious disease monitoring and decision support system

To improve the hospital's ability to proactively detect infectious diseases, a knowledge-based infectious disease monitoring and decision support system was established based on real medical records and knowledge rules. The effectiveness of the system was evaluated using interrupted time series analysis. In the system, a monitoring and alert rule library for infectious diseases was generated by combining infectious disease diagnosis guidelines with literature and a real medical record knowledge map. The system was integrated with the electronic medical record system, and doctors were provided with various types of real-time warning prompts when writing medical records. The effectiveness of the system's alerts was analyzed from the perspectives of false positive rates, rule accuracy, alert effectiveness, and missed case rates using interrupted time series analysis. Over a period of 12 months, the system analyzed 4,497,091 medical records, triggering a total of 12,027 monitoring alerts. Of these, 98.43% were clinically effective, while 1.56% were invalid alerts, mainly owing to the relatively rough rules generated by the guidelines leading to several false alarms. In addition, the effectiveness of the system's alerts, distribution of diagnosis times, and reporting efficiency of doctors were analyzed. 89.26% of infectious disease cases could be confirmed and reported by doctors within 5 min of receiving the alert, and 77.6% of doctors could complete the filling of 33 items of information within 2 min, which is a reduction in time compared to the past. The timely reminders from the system reduced the rate of missed cases by doctors; the analysis using interrupted time series method showed an average reduction of 4.4037% in the missed-case rate. This study proposed a knowledge-based infectious disease decision support system based on real medical records and knowledge rules, and its effectiveness was verified. The system improved the management of infectious diseases, increased the reliability of decision-making, and reduced the rate of underreporting.

Evaluating the impact of prescriptive authority for psychologists on the rate of deaths attributed to mental illness

BACKGROUND

Five states have enacted policies granting prescriptive authority to psychologists in an effort to increase access to psychoactive medications; however, little is known regarding the public health impact of these policies. Policies in two of these states, New Mexico and Louisiana, have had sufficient time to license more than a handful of prescribing psychologists. This study estimates the impact of psychologist prescriptive authority policies in New Mexico and Louisiana on deaths attributable to mental illness and suicides.

METHODS

State-level annual death rates from all 50 states were obtained for deaths with an underlying cause of death attributable to mental illness and to suicide (1999-2013) from the Centers for Disease Control and Prevention's WONDER database. State characteristics were collected for the pre-policy time period (1999-2004). We estimated the impact of the policy on the rates of deaths attributable to mental illness and to suicide using a comparative interrupted time series design, and policy effect estimates were generated for New Mexico and Louisiana separately. We used the synthetic control method to create synthetic New Mexico and synthetic Louisiana for use as the comparators.

RESULTS

Immediately following the start of psychologist prescribing, the rate of deaths attributable to mental illness declined by 4.55 deaths per 100,000 (95% CI: [-8.30, -0.79]) in New Mexico relative to the control, but there was no change in Louisiana. There was no immediate change in the suicide rate in either state; however, the annual change in the overall suicide rate was 0.12 suicides per 100,000 (95% CI: [-0.18, -0.06]) per year lower than expected in Louisiana following implementation.

CONCLUSIONS

These findings suggest that policies granting prescriptive authority to psychologists have the potential to reduce the mental health mortality gap, though considerable questions remain.

Effects of the Pelotas (Brazil) Peace Pact on violence and crime: a synthetic control analysis

Background

City-led interventions are increasingly advocated to achieve the UN's Sustainable Development Goal to reduce violence for all. We used a new quantitative evaluation method to examine whether a flagship programme, called the "Pelotas Pact for Peace" (the Pacto), has been effective in reducing violence and crime in the city of Pelotas, Brazil.

Methods

We used synthetic control methodology to assess the effects of the Pacto from August 2017 to December 2021, and separately before and during the COVID-19 pandemic. Outcomes included monthly rates of homicide and property crime, and yearly rates of assault against women and school drop-out. We constructed synthetic controls (counterfactuals) based on weighted averages from a donor pool of municipalities in Rio Grande do Sul. Weights were identified using pre-intervention outcome trends and confounders (sociodemographics, economics, education, health and development, and drug trafficking).

Findings

The Pacto led to an overall 9% reduction in homicide and 7% reduction in robbery in Pelotas. These effects were not uniform across the full post-intervention period as clear effects were only seen during the pandemic period. A 38% reduction in homicide was also specifically associated with the criminal justice strategy of Focussed Deterrence. No significant effects were found for non-violent property crimes, violence against women, and school dropout, irrespective of the post-intervention period.

Interpretation

City-level interventions that combine public health and criminal justice approaches could be effective in tackling violence in Brazil. Continued monitoring and evaluation efforts are increasingly needed as cities are proposed as key opportunities for reducing violence for all.

Funding

This research was funded by the Wellcome Trust [grant number: 210735_Z_18_Z].

Using synthetic controls to estimate the population-level effects of Ontario's recently implemented overdose prevention sites and consumption and treatment services

BACKGROUND

Between 2017 and 2020, Ontario implemented overdose prevention sites (OPS) and consumption and treatment services (CTS) in nine of its 34 public health units (PHU). We tested for the effect of booth-hours (spaces within OPS/CTSs for supervised consumption) on opioid-related health service use and mortality rates at the provincial- (aggregate) and PHU-level.

METHODS

We used monthly rates of all opioid-related emergency department (ED) visits, hospitalizations, and deaths between January 2015 and March 2021 as our three outcomes. For each PHU that implemented OPS/CTSs, we created a synthetic control as a weighted combination of unexposed PHUs. Our exposure was the time-varying rate of booth-hours provided. We estimated the population-level effects of the intervention on each outcome per treated/synthetic-control pair using controlled interrupted time series with segmented regression; and tested for the aggregate effect using a multiple baseline approach. We adjusted for time-varying provision of prescription opioids for pain management, opioid agonist treatment (OAT), and naloxone kits; and corrected for seasonality and autocorrelation. All rates were per 100,000 population. For sensitivity analysis, we restricted the post-implementation period to before COVID-19 public health measures were implemented (March 2020).

RESULTS

Our aggregate analyses found no effect per booth-hour on ED visits (0.00, 95% CI: -0.01, 0.01; p-value=0.6684), hospitalizations (0.00, 95% CI: 0.00, 0.00; p-value=0.9710) or deaths (0.00, 95% CI: 0.00, 0.00; p-value=0.2466). However, OAT reduced ED visits (-0.20, 95% CI: -0.35, -0.05; p-value=0.0103) and deaths (-0.04, 95% CI: -0.05, -0.03; p-value=<0.0001). Conversely, prescription opioids for pain management modestly increased deaths (0.0008, 95% CI: 0.0002, 0.0015; p-value=0.0157) per 100,000 population, respectively. Except for a few treated PHU/synthetic control pairs, disaggregate results were congruent with overall findings.

CONCLUSION

Booth-hours had no population-level effect on opioid-related overdose ED visit, hospitalization, or death rates.

Vaccination and Government Stringent Control as Effective Strategies in Preventing SARS-CoV-2 Infections: A Global Perspective

With the rapid implementation of global vaccination against the coronavirus disease 2019 (COVID-19), the threat posed by the disease has been mitigated, yet it remains a major global public health concern. Few studies have estimated the effects of vaccination and government stringent control measures on the disease transmission from a global perspective. To address this, we collected 216 countries' data on COVID-19 daily reported cases, daily vaccinations, daily government stringency indexes (GSIs), and the human development index (HDI) from the dataset of the World Health Organization (WHO) and the Our World in Data COVID-19 (OWID). We utilized the interrupted time series (ITS) model to examine how the incidence was affected by the vaccination and GSI at continental and country levels from 22 January 2020 to 13 February 2022. We found that the effectiveness of vaccination was better in Europe, North America, and Africa than in Asia, South America, and Oceania. The long-term effects outperformed the short-term effects in most cases. Countries with a high HDI usually had a high vaccination coverage, resulting in better vaccination effects. Nonetheless, some countries with high vaccination coverage did not receive a relatively low incidence due to the weaker GSI. The results suggest that in addition to increasing population vaccination coverage, it is crucial to maintain a certain level of government stringent measures to prevent and control the disease. The strategy is particularly appropriate for countries with low vaccination coverage at present.

Evaluating the Impact of Meningococcal Vaccines With Synthetic Controls

Invasive meningococcal disease (IMD) has a low and unpredictable incidence, presenting challenges for real-world evaluations of meningococcal vaccines. Traditionally, meningococcal vaccine impact is evaluated by predicting counterfactuals from pre-immunization IMD incidences, possibly controlling for IMD in unvaccinated age groups, but the selection of controls can influence results. We retrospectively applied a synthetic control (SC) method, previously used for pneumococcal disease, to data from 2 programs for immunization of infants against serogroups B and C IMD in England and Brazil. Time series of infectious/noninfectious diseases in infants and IMD cases in older unvaccinated age groups were used as candidate controls, automatically combined in a SC through Bayesian variable selection. SC closely predicted IMD in absence of vaccination, adjusting for nontrivial changes in IMD incidence. Vaccine impact estimates were in line with previous assessments. IMD cases in unvaccinated age groups were the most frequent SC-selected controls. Similar results were obtained when excluding IMD from control sets and using other diseases only, particularly respiratory diseases and measles. Using non-IMD controls may be important where there are herd immunity effects. SC is a robust and flexible method that addresses uncertainty introduced when equally plausible controls exhibit different post-immunization behaviors, allowing objective comparisons of IMD programs between countries.

Effect of excise tax on sugar-sweetened beverages in Catalonia, Spain, three and a half years after its introduction

Background

The World Health Organisation urges countries to levy specific excise taxes on SSBs. Currently, more than 50 countries have introduced some type of tax on SSBs. In March 2017, the Autonomous Region of Catalonia approved the introduction of a tiered excise tax on SSBs for public health reasons. To evaluate the effect of the Catalonian excise tax on the price and purchase of sugar-sweetened beverages (SSBs) and their possible substitutes, i.e., non-sugar-sweetened beverages (NSSBs) and bottled water, three and half years after its introduction, and 1 year after the outbreak of the COVID-19 pandemic.

Methods

We analysed purchase data on soft drinks, fruit drinks and water, sourced from the Ministry of Agriculture food-consumption panel, in a random sample of 12,500 households across Spain. We applied the synthetic control method to infer the causal impact of the intervention, based on a Bayesian structural time-series model which predicts the counterfactual response that would have occurred in Catalonia, had no intervention taken place.

Results

As compared to the predicted (counterfactual) response, per capita purchases of SSBs fell by 0.17 l three and a half years after implementing the SSB tax in Catalonia, a 16.7% decline (95% CI: − 23.18, − 8.74). The mean SSB price rose by 0.11 €/L, an 11% increase (95% CI: 9.0, 14.1). Although there were no changes in mean NSSB prices, NSSB consumption rose by 0.19 l per capita, a 21.7% increase (95% CI: 18.25, 25.54). There were no variations in the price or consumption of bottled water. The effects were progressively greater over time, with SSB purchases decreasing by 10.4% at 1 year, 12.3% at 2 years, 15.3% at 3 years, and 16.7% at three and a half years of the tax’s introduction.

Conclusions

The Catalonian SSB excise tax had a sustained and progressive impact over time, with a fall in consumption of as much as 16.7% three and half years after its introduction. The observed NSSB substitution effect should be borne in mind when considering the application of this type of tax to the rest of Spain.

Analysis of "Stand Your Ground" Self-defense Laws and Statewide Rates of Homicides and Firearm Homicides

IMPORTANCE

Most US states have amended self-defense laws to enhance legal immunities for individuals using deadly force in public. Despite concerns that "stand your ground" (SYG) laws unnecessarily encourage the use of deadly violence, their impact on violent deaths and how this varies across states and demographic groups remains unclear.

OBJECTIVE

To evaluate the association of SYG laws with homicide and firearm homicide, nationally and by state, while considering variation by the race, age, and sex of individuals who died by homicide.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study used a controlled, multiple-baseline and -location interrupted time series design, using natural variation in the timings and locations of SYG laws to assess associations. Changes in homicide and firearm homicide were modeled using Poisson regression analyses within a generalized additive model framework. Analyses included all US states that enacted SYG laws between 2000 and 2016 and states that did not have SYG laws enacted during the full study period, 1999 to 2017. Data were analyzed from November 2019 to December 2020.

EXPOSURES

SYG self-defense laws enacted by statute between January 1, 2000, to December 31, 2016.

MAIN OUTCOMES AND MEASURES

The main outcomes were statewide monthly rates of homicide and firearm-related homicide (per 100 000 persons) from January 1, 1999, to December 31, 2017, grouped by characteristics (ie, race, age, sex) of individuals who died by homicide.

RESULTS

Forty-one states were analyzed, including 23 states that enacted SYG laws during the study period and 18 states that did not have SYG laws, with 248 358 homicides (43.7% individuals aged 20-34 years; 77.9% men and 22.1% women), including 170 659 firearm homicides. SYG laws were associated with a mean national increase of 7.8% in monthly homicide rates (incidence rate ratio [IRR],1.08; 95% CI, 1.04-1.12; P < .001) and 8.0% in monthly firearm homicide rates (IRR, 1.08; 95% CI, 1.03-1.13; P = .002). SYG laws were not associated with changes in the negative controls of suicide (IRR, 0.99; 95% CI, 0.98-1.01) or firearm suicide (IRR, 1.00; 95% CI, 0.98-1.02). Increases in violent deaths varied across states, with the largest increases (16.2% to 33.5%) clustering in the South (eg, Alabama, Florida, Georgia, Louisiana). There were no differential associations of SYG laws by demographic group.

CONCLUSIONS AND RELEVANCE

These findings suggest that adoption of SYG laws across the US was associated with increases in violent deaths, deaths that could potentially have been avoided.

Intervention targeted at physicians' treatment of musculoskeletal disorders and sickness certification: an interrupted time series analysis

OBJECTIVE

An intervention was carried out at the occupational healthcare services (OHS) of the City of Helsinki beginning in 2016. We investigated the association between the intervention and employee sick leaves using interrupted time series analysis.

DESIGN

Register-based cohort study with a quasi-experimental study design.

SETTING

Employees of the City of Helsinki.

PARTICIPANTS

We analysed individual-level register-based data on all employees who were employed by the city for any length of time between 2013 and 2018 (a total 86 970 employees and 3 014 075 sick leave days). Sick leave days and periods that were OHS-based constituted the intervention time series and the rest of the sick leave days and periods contributed to the comparison time series.

INTERVENTION

Recommendations provided to physicians on managing pain and prescribing sick leave for low back, shoulder and elbow pain.

OUTCOME MEASURES

Number of sick leave days per month and sick leave periods per year.

RESULTS

For all sick leave days prescribed at OHS, there was no immediate change in sick leave days, whereas a gradual change showing decreasing number of OHS-based sick leave days was detected. On average, the intervention was estimated to have saved 2.5 sick leave days per year per employee. For other sick leave days, there was an immediate increase in the level of sick leave days after the intervention and a subsequent gradual trend showing decreasing number of sick leave days.

CONCLUSIONS

The intervention may have reduced employee sick leaves and therefore it is possible that it had led to direct cost savings. However, further evidence for causal inferences is needed.

Effects of stricter drunk-driving laws on alcohol-related road traffic death, injury, and crash rates in South Korea: A synthetic counterfactual approach using Bayesian structural time-series models

In December 2018, new drunk-driving laws were enacted in Korea to impose stricter penalties and standards for driving under the influence of alcohol. This study aimed to estimate the effects of stricter drunk-driving laws on alcohol-related road traffic death, injury, and crash rates in Korea. Using police-reported traffic accident data and registered vehicle data from 2013 to 2020, monthly road traffic outcome rates were calculated: the response series involved alcohol-related rates and the non-equivalent control series involved total and non-alcohol-related rates. Based on a controlled interrupted time-series design using Bayesian structural time-series models, effects of the laws on alcohol-related road traffic outcome rates were evaluated. After implementation of these laws, the alcohol-related road traffic crash rate decreased by 14.3% (95% credible interval [CrI] -26.8% to -1.9%), alcohol-related road traffic injury rate by 17.6% (95% CrI -31.6% to -3.8%), and alcohol-related minor road traffic injury rate by 20.2% (95% CrI -32.4% to -7.7%). Alcohol-related road traffic death and severe injury rates also decreased more than the declining trends in the pre-period, but reduced non-significantly by 15.0% (95% CrI -47.2% to 17.3%) and 9.9% (95% CrI -33.9% to 14.5%), respectively. The mixed effectiveness of Korea's new drunk-driving laws on alcohol-related road traffic outcomes suggests that additional strategies are necessary to consistently and effectively reduce alcohol-related road traffic outcomes. More research is needed on ways to enhance the effectiveness of drunk-driving laws.

Synthetic Control Methods for the Evaluation of Single-Unit Interventions in Epidemiology: A Tutorial

Evaluating the impacts of population-level interventions (e.g., changes to state legislation) can be challenging as conducting randomized experiments is often impractical and inappropriate, especially in settings where the intervention is implemented in a single, aggregate unit (e.g., a country or state). A common nonrandomized alternative is to compare outcomes in the treated unit(s) with unexposed controls both before and after the intervention. However, the validity of these designs depends on the use of controls that closely resemble the treated unit on before-intervention characteristics and trends on the outcome, and suitable controls may be difficult to find because the number of potential control regions is typically limited. The synthetic control method provides a potential solution to these problems by using a data-driven algorithm to identify an optimal weighted control unit—a “synthetic control”—based on data from before the intervention from available control units. While popular in the social sciences, the method has not garnered as much attention in health research, perhaps due to a lack of accessible texts aimed at health researchers. We address this gap by providing a comprehensive, nontechnical tutorial on the synthetic control method, using a worked example evaluating Florida’s “stand your ground” law to illustrate methodological and practical considerations.

Impact of emerging virus pandemics on cause-specific maternal mortality time series: a population-based natural experiment using national vital statistics, Argentina 1980-2017

Background

Emerging pandemic viruses may have multiple deleterious effects on maternal health. This study examines the effects of a pandemic influenza virus on cause-specific maternal mortality time series, using Argentinian vital statistics.

Methods

We conducted a population-based natural experiment from national vital records of maternal deaths between 1980 and 2017. Joinpoint regression models were used to model time series of the maternal mortality ratio (MMR). The sensitivity of the registry to detect the effects of the pandemic H1N1 2009 influenza virus on cause-specific MMR was analysed using a panel of parallel interrupted time series (ITS).

Findings

Over this 38-year study, the MMR decreased by 58·6% (69·5 to 28·8 deaths/100,000 live births), transitioning from direct obstetric causes (67·0 to 21·1/100,000 live births; 68·4% decrease) to indirect causes (2·6 to 7·7/100,000 live births; 196·2% increase). The regression analysis showed an average reduction of -2·2%/year (95% CI: -2·9 to -1·4) with 2 join points in the total trend (1998 and 2009). Parallel ITS analyses revealed the pandemic H1N1 virus had an increasing effect on mortality from the respiratory system- and sepsis-related complications (level change 4·7 and 1·6/100,000 live births respectively), reversing after the outbreak. No effect was found on MMR from hypertensive disorders, haemorrhage, abortive outcomes, other direct obstetric causes, and indirect non-respiratory comorbidities.

Interpretation

The Argentinian maternal death registry appears sensitive to detect different effects of emerging infectious epidemics on maternal health. In a population-based natural experiment, pandemic H1N1 virus impacted maternal mortality almost exclusively from the respiratory system- and sepsis-related complications.

Funding

Supported by FISAR www.fisarchile.org.

A (Flexible) Synthetic Control Method for Count Data and Other Nonnegative Outcomes

The synthetic control method is a covariate balancing method that exploits data from untreated regions to construct a synthetic control that approximates a single, aggregate treatment unit on a time series of preintervention outcomes and covariates. The method is increasingly being used to evaluate population-level interventions in epidemiology. Although the original version can be used with bounded outcomes, it imposes strong constraints on the balancing weights to ensure that the counterfactuals are based solely on interpolation. This feature, while attractive from a causal inference perspective, is sometimes too conservative and can lead to unnecessary bias due to poor covariate balance. Alternatives exist that allow for extrapolation to improve balance but existing procedures may produce negative estimates of the counterfactual outcomes and are therefore inappropriate for count data. We propose an alternative way to allow for extrapolation, although ensuring that the estimated counterfactuals remain nonnegative. Following a related proposal, we add a penalty to the balancing procedure that favors interpolation over extrapolation whenever possible. As we demonstrate theoretically and using empirical examples, our proposal can serve as a useful alternative when existing approaches yield demonstrably poor or unrealistic counterfactuals. Finally, we provide functions to implement the method in R.