Combining machine learning and propensity score weighting to estimate causal effects in multivalued treatments

Individual meropenem epithelial lining fluid and plasma PK/PD target attainment

It is unclear whether plasma is a reliable surrogate for target attainment in the epithelial lining fluid (ELF). The objective of this study was to characterize meropenem target attainment in plasma and ELF using prospective samples. The first 24-hour T>MIC was evaluated vs 1xMIC and 4xMIC targets at the patient (i.e., fixed MIC of 2 mg/L) and population [i.e., cumulative fraction of response (CFR) according to EUCAST MIC distributions] levels for both plasma and ELF. Among 65 patients receiving ≥24 hours of treatment, 40% of patients failed to achieve >50% T>4xMIC in plasma and ELF, and 30% of patients who achieved >50% T>4xMIC in plasma had <50% T>4xMIC in ELF. At 1xMIC and 4xMIC targets, 3% and 25% of patients with >95% T>MIC in plasma had <50% T>MIC in ELF, respectively. Those with a CRCL >115 mL/min were less likely to achieve >50%T>4xMIC in ELF (P < 0.025). In the population, CFR for Escherichia coli at 1xMIC and 4xMIC was >97%. For Pseudomonas aeruginosa, CFR was ≥90% in plasma and ranged 80%-85% in ELF at 1xMIC when a loading dose was applied. CFR was reduced in plasma (range: 75%-81%) and ELF (range: 44%-60%) in the absence of a loading dose at 1xMIC. At 4xMIC, CFR for P. aeruginosa was 60%-86% with a loading dose and 18%-62% without a loading dose. We found that plasma overestimated ELF target attainment inup to 30% of meropenem-treated patients, CRCL >115 mL/min decreased target attainment in ELF, and loading doses increased CFR in the population.

Propensity score analysis with missing data using a multi-task neural network

BACKGROUND

Propensity score analysis is increasingly used to control for confounding factors in observational studies. Unfortunately, unavoidable missing values make estimating propensity scores extremely challenging. We propose a new method for estimating propensity scores in data with missing values.

MATERIALS AND METHODS

Both simulated and real-world datasets are used in our experiments. The simulated datasets were constructed under 2 scenarios, the presence (T = 1) and the absence (T = 0) of the true effect. The real-world dataset comes from LaLonde's employment training program. We construct missing data with varying degrees of missing rates under three missing mechanisms: MAR, MCAR, and MNAR. Then we compare MTNN with 2 other traditional methods in different scenarios. The experiments in each scenario were repeated 20,000 times. Our code is publicly available at https://github.com/ljwa2323/MTNN .

RESULTS

Under the three missing mechanisms of MAR, MCAR and MNAR, the RMSE between the effect and the true effect estimated by our proposed method is the smallest in simulations and in real-world data. Furthermore, the standard deviation of the effect estimated by our method is the smallest. In situations where the missing rate is low, the estimation of our method is more accurate.

CONCLUSIONS

MTNN can perform propensity score estimation and missing value filling at the same time through shared hidden layers and joint learning, which solves the dilemma of traditional methods and is very suitable for estimating true effects in samples with missing values. The method is expected to be broadly generalized and applied to real-world observational studies.

Machine Learning in Causal Inference: Application in Pharmacovigilance

Monitoring adverse drug events or pharmacovigilance has been promoted by the World Health Organization to assure the safety of medicines through a timely and reliable information exchange regarding drug safety issues. We aim to discuss the application of machine learning methods as well as causal inference paradigms in pharmacovigilance. We first reviewed data sources for pharmacovigilance. Then, we examined traditional causal inference paradigms, their applications in pharmacovigilance, and how machine learning methods and causal inference paradigms were integrated to enhance the performance of traditional causal inference paradigms. Finally, we summarized issues with currently mainstream correlation-based machine learning models and how the machine learning community has tried to address these issues by incorporating causal inference paradigms. Our literature search revealed that most existing data sources and tasks for pharmacovigilance were not designed for causal inference. Additionally, pharmacovigilance was lagging in adopting machine learning-causal inference integrated models. We highlight several currently trending directions or gaps to integrate causal inference with machine learning in pharmacovigilance research. Finally, our literature search revealed that the adoption of causal paradigms can mitigate known issues with machine learning models. We foresee that the pharmacovigilance domain can benefit from the progress in the machine learning field.

Matching on poset-based average rank for multiple treatments to compare many unbalanced groups

In this article, we propose an original matching procedure for multiple treatment frameworks based on partially ordered set theory (poset). In our proposal, called matching on poset‐based average rank for multiple treatments (MARMoT), poset theory is used to summarize individuals' confounders and the relative average rank is used to balance confounders and match individuals in different treatment groups. This approach proves to be particularly useful for balancing confounders when the number of treatments considered is high. We apply our approach to the estimation of neighborhood effect on the fractures among older people in Turin (a city in northern Italy).

The role of machine learning in clinical research: transforming the future of evidence generation

Background

Interest in the application of machine learning (ML) to the design, conduct, and analysis of clinical trials has grown, but the evidence base for such applications has not been surveyed. This manuscript reviews the proceedings of a multi-stakeholder conference to discuss the current and future state of ML for clinical research. Key areas of clinical trial methodology in which ML holds particular promise and priority areas for further investigation are presented alongside a narrative review of evidence supporting the use of ML across the clinical trial spectrum.

Results

Conference attendees included stakeholders, such as biomedical and ML researchers, representatives from the US Food and Drug Administration (FDA), artificial intelligence technology and data analytics companies, non-profit organizations, patient advocacy groups, and pharmaceutical companies. ML contributions to clinical research were highlighted in the pre-trial phase, cohort selection and participant management, and data collection and analysis. A particular focus was paid to the operational and philosophical barriers to ML in clinical research. Peer-reviewed evidence was noted to be lacking in several areas.

Conclusions

ML holds great promise for improving the efficiency and quality of clinical research, but substantial barriers remain, the surmounting of which will require addressing significant gaps in evidence.

Artificial intelligence in COVID-19 drug repurposing

Drug repurposing or repositioning is a technique whereby existing drugs are used to treat emerging and challenging diseases, including COVID-19. Drug repurposing has become a promising approach because of the opportunity for reduced development timelines and overall costs. In the big data era, artificial intelligence (AI) and network medicine offer cutting-edge application of information science to defining disease, medicine, therapeutics, and identifying targets with the least error. In this Review, we introduce guidelines on how to use AI for accelerating drug repurposing or repositioning, for which AI approaches are not just formidable but are also necessary. We discuss how to use AI models in precision medicine, and as an example, how AI models can accelerate COVID-19 drug repurposing. Rapidly developing, powerful, and innovative AI and network medicine technologies can expedite therapeutic development. This Review provides a strong rationale for using AI-based assistive tools for drug repurposing medications for human disease, including during the COVID-19 pandemic.

Using machine learning to evaluate treatment effects in multiple-group interrupted time series analysis

RATIONALE, AIMS, AND OBJECTIVES

Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time, and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. In this paper, we introduce a novel machine learning approach using optimal discriminant analysis (ODA) to evaluate treatment effects in multiple-group ITSA.

METHOD

We evaluate the effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales, by comparing California (CA) to Montana (MT)-the best matching control state not exposed to any smoking reduction initiatives. We contrast results from ODA to those of ITSA regression (ITSAREG)-a commonly used approach for evaluating treatment effects in ITSA studies.

RESULTS

Both approaches found CA and MT to be comparable on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the post-intervention period (P < 0.0001). The ODA model achieved very high effect strength of sensitivity (a measure of classification accuracy) of 91.67%, which remained high (75.00%) after conducting leave-one-out analysis to assess generalizability.

CONCLUSIONS

The ODA framework achieved results comparable to ITSAREG, bolstering confidence in the intervention effect. In addition, ODA confers several advantages over conventional approaches that may make it a better approach to use in multiple group ITSA studies: insensitivity to skewed data, model-free permutation tests to derive P values, identification of the threshold value which best discriminates intervention and control groups, a chance- and maximum-corrected index of classification accuracy, and cross-validation to assess generalizability.

Identifying causal mechanisms in health care interventions using classification tree analysis

RATIONALE, AIMS, AND OBJECTIVES

Mediation analysis identifies causal pathways by testing the relationships between the treatment, the outcome, and an intermediate variable that mediates the relationship between the treatment and outcome. This paper introduces classification tree analysis (CTA), a machine-learning procedure, as an alternative to conventional methods for analysing mediation effects.

METHOD

Using data from the JOBS II study, we compare CTA to structural equation models (SEMs) by assessing their consistency in revealing mediation effects on 2 outcomes; reemployment (a binary variable) and depressive symptoms (a continuous variable). Because study participants were not randomized sequentially to both treatment and mediator, an additional model was generated including baseline covariates to strengthen the validity of some key identifying assumptions required of all mediation analyses.

RESULTS

Using SEM, no statistically significant treatment or mediated effects were found for either outcome. In contrast, CTA found a significant treatment effect for reemployment (P = .047) and a mediated pathway for individuals in the treatment group (P = .014). No CTA model could be generated for the reemployment outcome. When covariates were added to the model, CTA found numerous interactions, while SEM found no effects.

CONCLUSIONS

CTA may uncover mediation effects where conventional approaches do not, because CTA does not require any assumptions about the distribution of variables nor of the functional form of the model, and CTA will systematically identify all statistically viable interactions. The versatility of CTA enables the investigator to explore the theorized underlying causal mechanism of an intervention in a much more comprehensive manner than conventional mediation analytic approaches.

Estimating causal effects for survival (time-to-event) outcomes by combining classification tree analysis and propensity score weighting

RATIONALE, AIMS AND OBJECTIVES

A common approach to assessing treatment effects in nonrandomized studies with time-to-event outcomes is to estimate propensity scores and compute weights using logistic regression, test for covariate balance, and then estimate treatment effects using Cox regression. A machine-learning alternative-classification tree analysis (CTA)-used to generate propensity scores and to estimate treatment effects in time-to-event data may identify complex relationships between covariates not found using conventional regression-based approaches.

METHOD

Using empirical data, we identify all statistically valid CTA propensity score models and then use them to compute strata-specific, observation-level propensity score weights that are subsequently applied in outcomes analyses. We compare findings obtained using this framework to the conventional method, by evaluating covariate balance and treatment effect estimates obtained using Cox regression and a weighted CTA outcomes model.

RESULTS

All models had some imbalanced covariates. Nevertheless, treatment effect estimates were generally consistent across all weighted models.

CONCLUSIONS

In the study sample, given that all approaches elicited similar results, using CTA increased confidence that bias could not be reduced any further. Because the CTA algorithm identifies all statistically valid propensity score models for a sample, it is most likely to identify a correctly specified propensity score model-and therefore should be used either to confirm results using traditional methods, or to reveal biases that may be missed by traditional methods. Moreover, given that the true treatment effect is never known in observational data, CTA should be considered for estimating outcomes because no statistical assumptions are required.

Minimizing imbalances on patient characteristics between treatment groups in randomized trials using classification tree analysis

RATIONALE, AIMS, AND OBJECTIVES

Randomization ensures that treatment groups do not differ systematically in their characteristics, thereby reducing threats to validity that may otherwise explain differences in outcomes. Large observed imbalances in patient characteristics may indicate that selection bias is being introduced into the treatment allocation process. We introduce classification tree analysis (CTA) as a novel algorithmic approach for identifying potential imbalances in characteristics and their interactions when provisionally assigning each new participant to one or the other treatment group. The participant is then permanently assigned to the treatment group that elicits either no or less imbalance than if assigned to the alternate group.

METHOD

Using data on participant characteristics from a clinical trial, we compare 3 different treatment allocation approaches: permuted block randomization (the original allocation method), minimization, and CTA. Treatment allocation performance is assessed by examining balance of all 17 patient characteristics between study groups for each of the allocation techniques.

RESULTS

While all 3 treatment allocation techniques achieved excellent balance on main effect variables, Classification tree analysis further identified imbalances on interactions and in the distributions of some of the continuous variables.

CONCLUSIONS

Classification tree analysis offers an algorithmic procedure that may be used with any randomization methodology to identify and then minimize linear, nonlinear, and interactive effects that induce covariate imbalance between groups. Investigators should consider using the CTA approach as a real-time complement to randomization for any clinical trial to safeguard the treatment allocation process against bias.

Modeling time-to-event (survival) data using classification tree analysis

RATIONALE, AIMS, AND OBJECTIVES

Time to the occurrence of an event is often studied in health research. Survival analysis differs from other designs in that follow-up times for individuals who do not experience the event by the end of the study (called censored) are accounted for in the analysis. Cox regression is the standard method for analysing censored data, but the assumptions required of these models are easily violated. In this paper, we introduce classification tree analysis (CTA) as a flexible alternative for modelling censored data. Classification tree analysis is a "decision-tree"-like classification model that provides parsimonious, transparent (ie, easy to visually display and interpret) decision rules that maximize predictive accuracy, derives exact P values via permutation tests, and evaluates model cross-generalizability.

METHOD

Using empirical data, we identify all statistically valid, reproducible, longitudinally consistent, and cross-generalizable CTA survival models and then compare their predictive accuracy to estimates derived via Cox regression and an unadjusted naïve model. Model performance is assessed using integrated Brier scores and a comparison between estimated survival curves.

RESULTS

The Cox regression model best predicts average incidence of the outcome over time, whereas CTA survival models best predict either relatively high, or low, incidence of the outcome over time.

CONCLUSIONS

Classification tree analysis survival models offer many advantages over Cox regression, such as explicit maximization of predictive accuracy, parsimony, statistical robustness, and transparency. Therefore, researchers interested in accurate prognoses and clear decision rules should consider developing models using the CTA-survival framework.

Using classification tree analysis to generate propensity score weights

RATIONALE, AIMS AND OBJECTIVES

In evaluating non-randomized interventions, propensity scores (PS) estimate the probability of assignment to the treatment group given observed characteristics. Machine learning algorithms have been proposed as an alternative to conventional logistic regression for modelling PS in order to avoid limitations of linear methods. We introduce classification tree analysis (CTA) to generate PS which is a "decision-tree"-like classification model that provides accurate, parsimonious decision rules that are easy to display and interpret, reports P values derived via permutation tests, and evaluates cross-generalizability.

METHOD

Using empirical data, we identify all statistically valid CTA PS models and then use them to compute strata-specific, observation-level PS weights that are subsequently applied in outcomes analyses. We compare findings obtained using this framework to logistic regression and boosted regression, by evaluating covariate balance using standardized differences, model predictive accuracy, and treatment effect estimates obtained using median regression and a weighted CTA outcomes model.

RESULTS

While all models had some imbalanced covariates, main-effects logistic regression yielded the lowest average standardized difference, whereas CTA yielded the greatest predictive accuracy. Nevertheless, treatment effect estimates were generally consistent across all models.

CONCLUSIONS

Assessing standardized differences in means as a test of covariate balance is inappropriate for machine learning algorithms that segment the sample into two or more strata. Because the CTA algorithm identifies all statistically valid PS models for a sample, it is most likely to identify a correctly specified PS model, and should be considered as an alternative approach to modeling the PS.

Persistent erectile dysfunction in men exposed to the 5α-reductase inhibitors, finasteride, or dutasteride

Importance

Case reports describe persistent erectile dysfunction (PED) associated with exposure to 5α-reductase inhibitors (5α-RIs). Clinical trial reports and the manufacturers’ full prescribing information (FPI) for finasteride and dutasteride state that risk of sexual adverse effects is not increased by longer duration of 5α-RI exposure and that sexual adverse effects of 5α-RIs resolve in men who discontinue exposure.

Objective

Our chief objective was to assess whether longer duration of 5α-RI exposure increases risk of PED, independent of age and other known risk factors. Men with shorter 5α-RI exposure served as a comparison control group for those with longer exposure.

Design

We used a single-group study design and classification tree analysis (CTA) to model PED (lasting ≥90 days after stopping 5α-RI). Covariates included subject attributes, diseases, and drug exposures associated with sexual dysfunction.

Setting

Our data source was the electronic medical record data repository for Northwestern Medicine.

Subjects

The analysis cohorts comprised all men exposed to finasteride or dutasteride or combination products containing one of these drugs, and the subgroup of men 16–42 years old and exposed to finasteride ≤1.25 mg/day.

Main outcome and measures

Our main outcome measure was diagnosis of PED beginning after first 5α-RI exposure, continuing for at least 90 days after stopping 5α-RI, and with contemporaneous treatment with a phosphodiesterase-5 inhibitor (PDE₅I). Other outcome measures were erectile dysfunction (ED) and low libido. PED was determined by manual review of medical narratives for all subjects with ED. Risk of an adverse effect was expressed as number needed to harm (NNH).

Results

Among men with 5α-RI exposure, 167 of 11,909 (1.4%) developed PED (persistence median 1,348 days after stopping 5α-RI, interquartile range (IQR) 631.5–2320.5 days); the multivariable model predicting PED had four variables: prostate disease, duration of 5α-RI exposure, age, and nonsteroidal anti-inflammatory drug (NSAID) use. Of 530 men with new ED, 167 (31.5%) had new PED. Men without prostate disease who combined NSAID use with >208.5 days of 5α-RI exposure had 4.8-fold higher risk of PED than men with shorter exposure (NNH 59.8, all p < 0.002). Among men 16–42 years old and exposed to finasteride ≤1.25 mg/day, 34 of 4,284 (0.8%) developed PED (persistence median 1,534 days, IQR 651–2,351 days); the multivariable model predicting PED had one variable: duration of 5α-RI exposure. Of 103 young men with new ED, 34 (33%) had new PED. Young men with >205 days of finasteride exposure had 4.9-fold higher risk of PED (NNH 108.2, p < 0.004) than men with shorter exposure.

Conclusion and relevance

Risk of PED was higher in men with longer exposure to 5α-RIs. Among young men, longer exposure to finasteride posed a greater risk of PED than all other assessed risk factors.

Enhancing predictive accuracy and reproducibility in clinical evaluation research: Commentary on the special section of the Journal of Evaluation in Clinical Practice

This paper introduces a special section of the current issue of the Journal of Evaluation in Clinical Practice that includes a set of 6 empirical articles showcasing a versatile, new machine-learning statistical method, known as optimal data (or discriminant) analysis (ODA), specifically designed to produce statistical models that maximize predictive accuracy. As this set of papers clearly illustrates, ODA offers numerous important advantages over traditional statistical methods-advantages that enhance the validity and reproducibility of statistical conclusions in empirical research. This issue of the journal also includes a review of a recently published book that provides a comprehensive introduction to the logic, theory, and application of ODA in empirical research. It is argued that researchers have much to gain by using ODA to analyze their data.