Using machine learning to identify structural breaks in single-group interrupted time series designs

Evaluating Impact Using Time-Series Data

Humanity's impact on the environment is increasing, as are strategies to conserve biodiversity, but a lack of understanding about how interventions affect ecological and conservation outcomes hampers decision-making. Time series are often used to assess impacts, but ecologists tend to compare average values from before to after an impact; overlooking the potential for the intervention to elicit a change in trend. Without methods that allow for a range of responses, erroneous conclusions can be drawn, especially for large, multi-time-series datasets, which are increasingly available. Drawing on literature in other disciplines and pioneering work in ecology, we present a standardised framework to robustly assesses how interventions, like natural disasters or conservation policies, affect ecological time series.

Using randomization tests to assess treatment effects in multiple-group interrupted time series analysis

RATIONALE, AIMS, AND OBJECTIVES

Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. However, multiple-group ITSA typically has small sample sizes, and parametric methods for multiple-group ITSA require strong assumptions that are unlikely to be met, possibly resulting in misleading P values. In this paper, randomization tests are introduced as a non-parametric, distribution-free option for computing exact P values.

METHOD

The effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales is evaluated by comparing California (CA) to Montana (MT) and Idaho (ID)-the two best matched control states not exposed to any smoking reduction initiatives. Results from randomization tests are contrasted to those of interrupted time series analysis regression (ITSAREG)-a commonly used parametric approach for evaluating treatment effects in ITSA studies.

RESULTS

Both approaches found ID and MT to be comparable to CA on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the postintervention period (P < 0.01).

CONCLUSIONS

In these data, randomization tests computed P values comparable with ITSAREG, bolstering confidence in the intervention effect. Routinely including randomization tests as a complement, or alternative, to parametric methods is therefore beneficial because randomization tests are free of assumptions regarding sample size and distribution and are extremely flexible in the choice of test statistic.

Using forecast modelling to evaluate treatment effects in single-group interrupted time series analysis

RATIONALE, AIMS, AND OBJECTIVES

Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied serially over time and the intervention is expected to "interrupt" the level and/or trend of that outcome. ITSA is commonly evaluated using methods which may produce biased results if model assumptions are violated. In this paper, treatment effects are alternatively assessed by using forecasting methods to closely fit the preintervention observations and then forecast the post-intervention trend. A treatment effect may be inferred if the actual post-intervention observations diverge from the forecasts by some specified amount.

METHOD

The forecasting approach is demonstrated using the effect of California's Proposition 99 for reducing cigarette sales. Three forecast models are fit to the preintervention series-linear regression (REG), Holt-Winters (HW) non-seasonal smoothing, and autoregressive moving average (ARIMA)-and forecasts are generated into the post-intervention period. The actual observations are then compared with the forecasts to assess intervention effects.

RESULTS

The preintervention data were fit best by HW, followed closely by ARIMA. REG fit the data poorly. The actual post-intervention observations were above the forecasts in HW and ARIMA, suggesting no intervention effect, but below the forecasts in the REG (suggesting a treatment effect), thereby raising doubts about any definitive conclusion of a treatment effect.

CONCLUSIONS

In a single-group ITSA, treatment effects are likely to be biased if the model is misspecified. Therefore, evaluators should consider using forecast models to accurately fit the preintervention data and generate plausible counterfactual forecasts, thereby improving causal inference of treatment effects in single-group ITSA studies.

Using machine learning to evaluate treatment effects in multiple-group interrupted time series analysis

RATIONALE, AIMS, AND OBJECTIVES

Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time, and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. In this paper, we introduce a novel machine learning approach using optimal discriminant analysis (ODA) to evaluate treatment effects in multiple-group ITSA.

METHOD

We evaluate the effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales, by comparing California (CA) to Montana (MT)-the best matching control state not exposed to any smoking reduction initiatives. We contrast results from ODA to those of ITSA regression (ITSAREG)-a commonly used approach for evaluating treatment effects in ITSA studies.

RESULTS

Both approaches found CA and MT to be comparable on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the post-intervention period (P < 0.0001). The ODA model achieved very high effect strength of sensitivity (a measure of classification accuracy) of 91.67%, which remained high (75.00%) after conducting leave-one-out analysis to assess generalizability.

CONCLUSIONS

The ODA framework achieved results comparable to ITSAREG, bolstering confidence in the intervention effect. In addition, ODA confers several advantages over conventional approaches that may make it a better approach to use in multiple group ITSA studies: insensitivity to skewed data, model-free permutation tests to derive P values, identification of the threshold value which best discriminates intervention and control groups, a chance- and maximum-corrected index of classification accuracy, and cross-validation to assess generalizability.

Using group-based trajectory modelling to enhance causal inference in interrupted time series analysis

RATIONALE, AIMS, AND OBJECTIVES

Several enhancements have been proposed for interrupted time series analysis (ITSA) to improve causal inference. Presently, group-based trajectory modelling (GBTM) is introduced as a complement to ITSA. GBTM assumes a certain number of discrete groups in the sample have unique trajectories of the outcome. GBTM is used herein for 2 purposes: (1) to compare outcomes across all trajectory groups via a stand-alone GBTM and (2) to identify comparable non-treated units to serve as controls in the ITSA outcome model. Examples of each are offered.

METHOD

The effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales is evaluated by comparing California to other states not exposed to smoking reduction initiatives. In the stand-alone GBTM, distinct trajectory groups are identified based on cigarette sales for the entire observation period (1970-2000). In the second approach, a GBTM is generated using only baseline period observations (1970-1988), and treatment effects (difference in post-intervention trends) are then estimated for the treatment unit versus non-treated units in the treated unit's trajectory group.

RESULTS

In the stand-alone GBTM, 3 distinct trajectory groups were identified: low-decreasing, medium-decreasing, and high-decreasing (California and 26 other states were in the low-decreasing group). When using baseline data for matching, California and 19 non-treated states comprised the low group. California had a significantly larger decrease in post-intervention cigarette sales than these controls (P < 0.01).

CONCLUSIONS

GBTM enhances ITSA by providing perspective for the outcome trajectory in the treated unit's group versus all others and can identify non-treated units to be used for estimating treatment effects.

Using permutation tests to enhance causal inference in interrupted time series analysis

RATIONALE, AIMS AND OBJECTIVES

Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied serially over time and the intervention is expected to "interrupt" the level and/or trend of that outcome. The internal validity is strengthened considerably when the treated unit is contrasted with a comparable control group. In this paper, we introduce a robustness check based on permutation tests to further improve causal inference.

METHOD

We evaluate the effect of California's Proposition 99 for reducing cigarette sales by iteratively casting each nontreated state into the role of "treated," creating a comparable control group using the ITSAMATCH package in Stata, and then evaluating treatment effects using ITSA regression. If statistically significant "treatment effects" are estimated for pseudotreated states, then any significant changes in the outcome of the actual treatment unit (California) cannot be attributed to the intervention. We perform these analyses setting the cutpoint significance level to P > .40 for identifying balanced matches (the highest threshold possible for which controls could still be found for California) and use the difference in differences of trends as the treatment effect estimator.

RESULTS

Only California attained a statistically significant treatment effect, strengthening confidence in the conclusion that Proposition 99 reduced cigarette sales.

CONCLUSIONS

The proposed permutation testing framework provides an additional robustness check to either support or refute a treatment effect identified in for the true treated unit in ITSA. Given its value and ease of implementation, this framework should be considered as a standard robustness test in all multiple group interrupted time series analyses.

Combining synthetic controls and interrupted time series analysis to improve causal inference in program evaluation

RATIONALE, AIMS AND OBJECTIVES

Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome. The internal validity is strengthened considerably when the treated unit is contrasted with a comparable control group. In this paper, we introduce a robust evaluation framework that combines the synthetic controls method (SYNTH) to generate a comparable control group and ITSA regression to assess covariate balance and estimate treatment effects.

METHODS

We evaluate the effect of California's Proposition 99 for reducing cigarette sales, by comparing California to other states not exposed to smoking reduction initiatives. SYNTH is used to reweight nontreated units to make them comparable to the treated unit. These weights are then used in ITSA regression models to assess covariate balance and estimate treatment effects.

RESULTS

Covariate balance was achieved for all but one covariate. While California experienced a significant decrease in the annual trend of cigarette sales after Proposition 99, there was no statistically significant treatment effect when compared to synthetic controls.

CONCLUSIONS

The advantage of using this framework over regression alone is that it ensures that a comparable control group is generated. Additionally, it offers a common set of statistical measures familiar to investigators, the capability for assessing covariate balance, and enhancement of the evaluation with a comprehensive set of postestimation measures. Therefore, this robust framework should be considered as a primary approach for evaluating treatment effects in multiple group time series analysis.

A matching framework to improve causal inference in interrupted time-series analysis

RATIONALE, AIMS, AND OBJECTIVES

Interrupted time-series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. When ITSA is implemented without a comparison group, the internal validity may be quite poor. Therefore, adding a comparable control group to serve as the counterfactual is always preferred. This paper introduces a novel matching framework, ITSAMATCH, to create a comparable control group by matching directly on covariates and then use these matches in the outcomes model.

METHOD

We evaluate the effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales, by comparing California to other states not exposed to smoking reduction initiatives. We compare ITSAMATCH results to 2 commonly used matching approaches, synthetic controls (SYNTH), and regression adjustment; SYNTH reweights nontreated units to make them comparable to the treated unit, and regression adjusts covariates directly. Methods are compared by assessing covariate balance and treatment effects.

RESULTS

Both ITSAMATCH and SYNTH achieved covariate balance and estimated similar treatment effects. The regression model found no treatment effect and produced inconsistent covariate adjustment.

CONCLUSIONS

While the matching framework achieved results comparable to SYNTH, it has the advantage of being technically less complicated, while producing statistical estimates that are straightforward to interpret. Conversely, regression adjustment may "adjust away" a treatment effect. Given its advantages, ITSAMATCH should be considered as a primary approach for evaluating treatment effects in multiple-group time-series analysis.

Minimizing imbalances on patient characteristics between treatment groups in randomized trials using classification tree analysis

RATIONALE, AIMS, AND OBJECTIVES

Randomization ensures that treatment groups do not differ systematically in their characteristics, thereby reducing threats to validity that may otherwise explain differences in outcomes. Large observed imbalances in patient characteristics may indicate that selection bias is being introduced into the treatment allocation process. We introduce classification tree analysis (CTA) as a novel algorithmic approach for identifying potential imbalances in characteristics and their interactions when provisionally assigning each new participant to one or the other treatment group. The participant is then permanently assigned to the treatment group that elicits either no or less imbalance than if assigned to the alternate group.

METHOD

Using data on participant characteristics from a clinical trial, we compare 3 different treatment allocation approaches: permuted block randomization (the original allocation method), minimization, and CTA. Treatment allocation performance is assessed by examining balance of all 17 patient characteristics between study groups for each of the allocation techniques.

RESULTS

While all 3 treatment allocation techniques achieved excellent balance on main effect variables, Classification tree analysis further identified imbalances on interactions and in the distributions of some of the continuous variables.

CONCLUSIONS

Classification tree analysis offers an algorithmic procedure that may be used with any randomization methodology to identify and then minimize linear, nonlinear, and interactive effects that induce covariate imbalance between groups. Investigators should consider using the CTA approach as a real-time complement to randomization for any clinical trial to safeguard the treatment allocation process against bias.

Smoke-free legislation and paediatric hospitalisations for acute respiratory tract infections: national quasi-experimental study with unexpected findings and important methodological implications

Objectives

We investigated whether Scottish implementation of smoke-free legislation was associated with a reduction in unplanned hospitalisations or deaths (‘events’) due to respiratory tract infections (RTIs) among children.

Design

Interrupted time series (ITS).

Setting/participants

Children aged 0–12 years living in Scotland during 1996–2012.

Intervention

National comprehensive smoke-free legislation (March 2006).

Main outcome measure

Acute RTI events in the Scottish Morbidity Record-01 and/or National Records of Scotland Death Records.

Results

135 134 RTI events were observed over 155 million patient-months. In our prespecified negative binomial regression model accounting for underlying temporal trends, seasonality, sex, age group, region, urbanisation level, socioeconomic status and seven-valent pneumococcal vaccination status, smoke-free legislation was associated with an immediate rise in RTI events (incidence rate ratio (IRR)=1.24, 95% CI 1.20 to 1.28) and an additional gradual increase (IRR=1.05/year, 95% CI 1.05 to 1.06). Given this unanticipated finding, we conducted a number of post hoc exploratory analyses. Among these, automatic break point detection indicated that the rise in RTI events actually preceded the smoke-free law by 16 months. When accounting for this break point, smoke-free legislation was associated with a gradual decrease in acute RTI events: IRR=0.91/year, 95% CI 0.87 to 0.96.

Conclusions

Our prespecified ITS approach suggested that implementation of smoke-free legislation in Scotland was associated with an increase in paediatric RTI events. We were concerned that this result, which contradicted published evidence, was spurious. The association was indeed reversed when accounting for an unanticipated antecedent break point in the temporal trend, suggesting that the legislation may in fact be protective. ITS analyses should be subjected to comprehensive robustness checks to assess consistency.

Using classification tree analysis to generate propensity score weights

RATIONALE, AIMS AND OBJECTIVES

In evaluating non-randomized interventions, propensity scores (PS) estimate the probability of assignment to the treatment group given observed characteristics. Machine learning algorithms have been proposed as an alternative to conventional logistic regression for modelling PS in order to avoid limitations of linear methods. We introduce classification tree analysis (CTA) to generate PS which is a "decision-tree"-like classification model that provides accurate, parsimonious decision rules that are easy to display and interpret, reports P values derived via permutation tests, and evaluates cross-generalizability.

METHOD

Using empirical data, we identify all statistically valid CTA PS models and then use them to compute strata-specific, observation-level PS weights that are subsequently applied in outcomes analyses. We compare findings obtained using this framework to logistic regression and boosted regression, by evaluating covariate balance using standardized differences, model predictive accuracy, and treatment effect estimates obtained using median regression and a weighted CTA outcomes model.

RESULTS

While all models had some imbalanced covariates, main-effects logistic regression yielded the lowest average standardized difference, whereas CTA yielded the greatest predictive accuracy. Nevertheless, treatment effect estimates were generally consistent across all models.

CONCLUSIONS

Assessing standardized differences in means as a test of covariate balance is inappropriate for machine learning algorithms that segment the sample into two or more strata. Because the CTA algorithm identifies all statistically valid PS models for a sample, it is most likely to identify a correctly specified PS model, and should be considered as an alternative approach to modeling the PS.

Persistent threats to validity in single-group interrupted time series analysis with a cross over design

RATIONALE, AIMS AND OBJECTIVES

The basic single-group interrupted time series analysis (ITSA) design has been shown to be susceptible to the most common threat to validity-history-the possibility that some other event caused the observed effect in the time series. A single-group ITSA with a crossover design (in which the intervention is introduced and withdrawn 1 or more times) should be more robust. In this paper, we describe and empirically assess the susceptibility of this design to bias from history.

METHOD

Time series data from 2 natural experiments (the effect of multiple repeals and reinstatements of Louisiana's motorcycle helmet law on motorcycle fatalities and the association between the implementation and withdrawal of Gorbachev's antialcohol campaign with Russia's mortality crisis) are used to illustrate that history remains a threat to ITSA validity, even in a crossover design.

RESULTS

Both empirical examples reveal that the single-group ITSA with a crossover design may be biased because of history. In the case of motorcycle fatalities, helmet laws appeared effective in reducing mortality (while repealing the law increased mortality), but when a control group was added, it was shown that this trend was similar in both groups. In the case of Gorbachev's antialcohol campaign, only when contrasting the results against those of a control group was the withdrawal of the campaign found to be the more likely culprit in explaining the Russian mortality crisis than the collapse of the Soviet Union.

CONCLUSIONS

Even with a robust crossover design, single-group ITSA models remain susceptible to bias from history. Therefore, a comparable control group design should be included, whenever possible.

Challenges to validity in single-group interrupted time series analysis

RATIONALE, AIMS AND OBJECTIVES

Single-group interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single unit of observation is studied; the outcome variable is serially ordered as a time series, and the intervention is expected to "interrupt" the level and/or trend of the time series, subsequent to its introduction. The most common threat to validity is history-the possibility that some other event caused the observed effect in the time series. Although history limits the ability to draw causal inferences from single ITSA models, it can be controlled for by using a comparable control group to serve as the counterfactual.

METHOD

Time series data from 2 natural experiments (effect of Florida's 2000 repeal of its motorcycle helmet law on motorcycle fatalities and California's 1988 Proposition 99 to reduce cigarette sales) are used to illustrate how history biases results of single-group ITSA results-as opposed to when that group's results are contrasted to those of a comparable control group.

RESULTS

In the first example, an external event occurring at the same time as the helmet repeal appeared to be the cause of a rise in motorcycle deaths, but was only revealed when Florida was contrasted with comparable control states. Conversely, in the second example, a decreasing trend in cigarette sales prior to the intervention raised question about a treatment effect attributed to Proposition 99, but was reinforced when California was contrasted with comparable control states.

CONCLUSIONS

Results of single-group ITSA should be considered preliminary, and interpreted with caution, until a more robust study design can be implemented.

Persistent erectile dysfunction in men exposed to the 5α-reductase inhibitors, finasteride, or dutasteride

Importance

Case reports describe persistent erectile dysfunction (PED) associated with exposure to 5α-reductase inhibitors (5α-RIs). Clinical trial reports and the manufacturers’ full prescribing information (FPI) for finasteride and dutasteride state that risk of sexual adverse effects is not increased by longer duration of 5α-RI exposure and that sexual adverse effects of 5α-RIs resolve in men who discontinue exposure.

Objective

Our chief objective was to assess whether longer duration of 5α-RI exposure increases risk of PED, independent of age and other known risk factors. Men with shorter 5α-RI exposure served as a comparison control group for those with longer exposure.

Design

We used a single-group study design and classification tree analysis (CTA) to model PED (lasting ≥90 days after stopping 5α-RI). Covariates included subject attributes, diseases, and drug exposures associated with sexual dysfunction.

Setting

Our data source was the electronic medical record data repository for Northwestern Medicine.

Subjects

The analysis cohorts comprised all men exposed to finasteride or dutasteride or combination products containing one of these drugs, and the subgroup of men 16–42 years old and exposed to finasteride ≤1.25 mg/day.

Main outcome and measures

Our main outcome measure was diagnosis of PED beginning after first 5α-RI exposure, continuing for at least 90 days after stopping 5α-RI, and with contemporaneous treatment with a phosphodiesterase-5 inhibitor (PDE₅I). Other outcome measures were erectile dysfunction (ED) and low libido. PED was determined by manual review of medical narratives for all subjects with ED. Risk of an adverse effect was expressed as number needed to harm (NNH).

Results

Among men with 5α-RI exposure, 167 of 11,909 (1.4%) developed PED (persistence median 1,348 days after stopping 5α-RI, interquartile range (IQR) 631.5–2320.5 days); the multivariable model predicting PED had four variables: prostate disease, duration of 5α-RI exposure, age, and nonsteroidal anti-inflammatory drug (NSAID) use. Of 530 men with new ED, 167 (31.5%) had new PED. Men without prostate disease who combined NSAID use with >208.5 days of 5α-RI exposure had 4.8-fold higher risk of PED than men with shorter exposure (NNH 59.8, all p < 0.002). Among men 16–42 years old and exposed to finasteride ≤1.25 mg/day, 34 of 4,284 (0.8%) developed PED (persistence median 1,534 days, IQR 651–2,351 days); the multivariable model predicting PED had one variable: duration of 5α-RI exposure. Of 103 young men with new ED, 34 (33%) had new PED. Young men with >205 days of finasteride exposure had 4.9-fold higher risk of PED (NNH 108.2, p < 0.004) than men with shorter exposure.

Conclusion and relevance

Risk of PED was higher in men with longer exposure to 5α-RIs. Among young men, longer exposure to finasteride posed a greater risk of PED than all other assessed risk factors.

Using machine learning to model dose-response relationships

RATIONALE, AIMS AND OBJECTIVES

Establishing the relationship between various doses of an exposure and a response variable is integral to many studies in health care. Linear parametric models, widely used for estimating dose-response relationships, have several limitations. This paper employs the optimal discriminant analysis (ODA) machine-learning algorithm to determine the degree to which exposure dose can be distinguished based on the distribution of the response variable. By framing the dose-response relationship as a classification problem, machine learning can provide the same functionality as conventional models, but can additionally make individual-level predictions, which may be helpful in practical applications like establishing responsiveness to prescribed drug regimens.

METHOD

Using data from a study measuring the responses of blood flow in the forearm to the intra-arterial administration of isoproterenol (separately for 9 black and 13 white men, and pooled), we compare the results estimated from a generalized estimating equations (GEE) model with those estimated using ODA.

RESULTS

Generalized estimating equations and ODA both identified many statistically significant dose-response relationships, separately by race and for pooled data. Post hoc comparisons between doses indicated ODA (based on exact P values) was consistently more conservative than GEE (based on estimated P values). Compared with ODA, GEE produced twice as many instances of paradoxical confounding (findings from analysis of pooled data that are inconsistent with findings from analyses stratified by race).

CONCLUSIONS

Given its unique advantages and greater analytic flexibility, maximum-accuracy machine-learning methods like ODA should be considered as the primary analytic approach in dose-response applications.

Enhancing predictive accuracy and reproducibility in clinical evaluation research: Commentary on the special section of the Journal of Evaluation in Clinical Practice

This paper introduces a special section of the current issue of the Journal of Evaluation in Clinical Practice that includes a set of 6 empirical articles showcasing a versatile, new machine-learning statistical method, known as optimal data (or discriminant) analysis (ODA), specifically designed to produce statistical models that maximize predictive accuracy. As this set of papers clearly illustrates, ODA offers numerous important advantages over traditional statistical methods-advantages that enhance the validity and reproducibility of statistical conclusions in empirical research. This issue of the journal also includes a review of a recently published book that provides a comprehensive introduction to the logic, theory, and application of ODA in empirical research. It is argued that researchers have much to gain by using ODA to analyze their data.