Tell me what you want, what you really really want: Estimands in observational pharmacoepidemiologic comparative effectiveness and safety studies

Directed Acyclic Graphs in Decision-Analytic Modeling: Bridging Causal Inference and Effective Model Design in Medical Decision Making

Decision-analytic models (DAMs) are essentially informative yet complex tools for solving questions in medical decision making. When their complexity grows, the need for causal inference techniques becomes evident as causal relationships between variables become unclear. In this methodological commentary, we argue that graphical representations of assumptions on such relationships, directed acyclic graphs (DAGs), can enhance the transparency of decision models and aid in parameter selection and estimation through visually specifying backdoor paths (i.e., potential biases in parameter estimates) and visually clarifying structural modeling choices of frontdoor paths (i.e., the effect of the model structure on the outcome). This commentary discusses the benefit of integrating DAGs and DAMs in medical decision making and in particular health economics with 2 applications: the first examines statin use for prevention of cardiovascular disease, and the second considers mindfulness-based interventions for students' stress. Despite the potential application of DAGs in the decision science framework, challenges remain, including simplicity, defining the scope of a DAG, unmeasured confounding, noncausal aspects, and limited data availability or quality. Broader adoption of DAGs in decision science requires full-model applications and further debate.HighlightsOur commentary proposes the application of directed acyclic graphs (DAGs) in the design of decision-analytic models, offering researchers a valuable and structured tool to enhance transparency and accuracy by bridging the gap between causal inference and model design in medical decision making.The practical examples in this article showcase the transformative effect DAGs can have on model structure, parameter selection, and the resulting conclusions on effectiveness and cost-effectiveness.This methodological article invites a broader conversation on decision-modeling choices grounded in causal assumptions.

What is the research question? Estimands explained

Although many papers of medical research report on a treatment effect, it is not always clear what is exactly meant by that effect. An estimand is a precise definition of a treatment effect and includes 5 attributes: population, treatment, (outcome) variable, intercurrent events, and summary measure. In this paper, we discuss how the estimand framework helps to align different phases of studies of medical treatments, from research objectives to design, conduct, analysis, and reporting of results.

Methodological challenges using routine clinical care data for real-world evidence: a rapid review utilizing a systematic literature search and focus group discussion

BACKGROUND

The integration of real-world evidence (RWE) from real-world data (RWD) in clinical research is crucial for bridging the gap between clinical trial results and real-world outcomes. Analyzing routinely collected data to generate clinical evidence faces methodological concerns like confounding and bias, similar to prospectively documented observational studies. This study focuses on additional limitations frequently reported in the literature, providing an overview of the challenges and biases inherent to analyzing routine clinical care data, including health claims data (hereafter: routine data).

METHODS

We conducted a literature search on routine data studies in four high-impact journals based on the Journal Citation Reports (JCR) category "Medicine, General & Internal" as of 2022 and three oncology journals, covering articles published from January 2018 to October 2023. Articles were screened and categorized into three scenarios based on their potential to provide meaningful RWE: (1) Burden of Disease, (2) Safety and Risk Group Analysis, and (3) Treatment Comparison. Limitations of this type of data cited in the discussion sections were extracted and classified according to different bias types: main bias categories in non-randomized studies (information bias, reporting bias, selection bias, confounding) and additional routine data-specific challenges (i.e., operationalization, coding, follow-up, missing data, validation, and data quality). These classifications were then ranked by relevance in a focus group meeting of methodological experts. The search was pre-specified and registered in PROSPERO (CRD42023477616).

RESULTS

In October 2023, 227 articles were identified, 69 were assessed for eligibility, and 39 were included in the review: 11 on the burden of disease, 17 on safety and risk group analysis, and 11 on treatment comparison. Besides typical biases in observational studies, we identified additional challenges specific to RWE frequently mentioned in the discussion sections. The focus group had varied opinions on the limitations of Safety and Risk Group Analysis and Treatment Comparison but agreed on the essential limitations for the Burden of Disease category.

CONCLUSION

This review provides a comprehensive overview of potential limitations and biases in analyzing routine data reported in recent high-impact journals. We highlighted key challenges that have high potential to impact analysis results, emphasizing the need for thorough consideration and discussion for meaningful inferences.

Core Concepts: Self-Controlled Designs in Pharmacoepidemiology

One of the key challenges in pharmacoepidemiological studies is that of uncontrolled confounding, which occurs when confounders are poorly measured, unmeasured or unknown. Self-controlled designs can help address this issue, as their key comparison is not between people, but periods of time within the same person. This controls for all time-stable confounders (genetics) and in the absence of time-varying confounding negates the need for an external control group. However, these benefits come at the cost of strong assumptions, not all of which are verifiable. This review briefly introduces the reader to different types of self-controlled study designs, their terminology and highlights key publications through an annotated reference list. We include a practical description of how these designs can be implemented and visualised using recent examples, and finish by discussing recent developments. We hope this review will serve as a starting point for researchers looking to apply self-controlled designs in their own work.

Tailored guidance to apply the Estimand framework to Trials within Cohorts (TwiCs) studies

Objective: The estimand framework offers a structured approach to define the treatment effect to be estimated in a clinical study. Defining the estimand upfront helps formulating the research question and informs study design, data collection and statistical analysis methods. Since the Trials within Cohorts (TwiCs) design has unique characteristics, the objective of this study is to describe considerations and provide guidance for formulating estimands for TwiCs studies. Methods: The key attributes of an estimand are the target population, treatments that are compared, the endpoint, intercurrent events and their handling, and the population-level summary measure. The estimand framework was applied retrospectively to two TwiCs studies: the SPONGE and UMBRELLA Fit trial. The aim is to demonstrate how the estimand framework can be implemented in TwiCs studies, thereby focusing on considerations relevant for defining the estimand. Three estimands were defined for both studies. For the SPONGE trial, estimators were derived. Results: Intercurrent events considered to occur exclusively or more frequently in TwiCs studies compared to conventional randomized trials included intervention refusal after randomization, misalignment of timing of routine cohort measurements and the intervention period, and participants in the control arm initiating treatments similar to the studied intervention. Considerations for handling refusal after randomization related to decisions on whether the target population should include all eligible participants or the subpopulation that would accept (or undergo) the intervention when offered. Considerations for handling treatment initiation in the control arm and misalignments of timing related to decisions on whether such events should be considered part of treatment policy or whether interest is in a hypothetical scenario where such events do not occur. Conclusion: The TwiCs study design has unique features that pose specific considerations when formulating an estimand. The examples in this study can provide guidance in the definition of estimands in future TwiCs studies.

Visualizing the target estimand in comparative effectiveness studies with multiple treatments

Aim: Comparative effectiveness research using real-world data often involves pairwise propensity score matching to adjust for confounding bias. We show that corresponding treatment effect estimates may have limited external validity, and propose two visualization tools to clarify the target estimand. Materials & methods: We conduct a simulation study to demonstrate, with bivariate ellipses and joy plots, that differences in covariate distributions across treatment groups may affect the external validity of treatment effect estimates. We showcase how these visualization tools can facilitate the interpretation of target estimands in a case study comparing the effectiveness of teriflunomide (TERI), dimethyl fumarate (DMF) and natalizumab (NAT) on manual dexterity in patients with multiple sclerosis. Results: In the simulation study, estimates of the treatment effect greatly differed depending on the target population. For example, when comparing treatment B with C, the estimated treatment effect (and respective standard error) varied from -0.27 (0.03) to -0.37 (0.04) in the type of patients initially receiving treatment B and C, respectively. Visualization of the matched samples revealed that covariate distributions vary for each comparison and cannot be used to target one common treatment effect for the three treatment comparisons. In the case study, the bivariate distribution of age and disease duration varied across the population of patients receiving TERI, DMF or NAT. Although results suggest that DMF and NAT improve manual dexterity at 1 year compared with TERI, the effectiveness of DMF versus NAT differs depending on which target estimand is used. Conclusion: Visualization tools may help to clarify the target population in comparative effectiveness studies and resolve ambiguity about the interpretation of estimated treatment effects.

Confounding adjustment in observational studies on cardiothoracic interventions: a systematic review of methodological practice

OBJECTIVES

It is unknown which confounding adjustment methods are currently used in the field of cardiothoracic surgery and whether these are appropriately applied. The aim of this study was to systematically evaluate the quality of conduct and reporting of confounding adjustment methods in observational studies on cardiothoracic interventions.

METHODS

A systematic review was performed, which included all observational studies that compared different interventions and were published between 1 January and 1 July 2022, in 3 European and American cardiothoracic surgery journals. Detailed information on confounding adjustment methods was extracted and subsequently described.

RESULTS

Ninety-two articles were included in the analysis. Outcome regression (n = 49, 53%) and propensity score (PS) matching (n = 44, 48%) were most popular (sometimes used in combination), whereas 11 (12%) studies applied no method at all. The way of selecting confounders was not reported in 42 (46%) of the studies, solely based on previous literature or clinical knowledge in 14 (16%), and (partly) data-driven in 25 (27%). For the studies that applied PS matching, the matched cohorts comprised on average 46% of the entire study population (range 9-82%).

CONCLUSIONS

Current reporting of confounding adjustment methods is insufficient in a large part of observational studies on cardiothoracic interventions, which makes quality judgement difficult. Appropriate application of confounding adjustment methods is crucial for causal inference on optimal treatment strategies for clinical practice. Reporting on these methods is an important aspect of this, which can be improved.

Differences in target estimands between different propensity score-based weights

PURPOSE

Propensity score weighting is a popular approach for estimating treatment effects using observational data. Different sets of propensity score-based weights have been proposed, including inverse probability of treatment weights whose target estimand is the average treatment effect, weights whose target estimand is the average treatment effect in the treated (ATT), and, more recently, matching weights, overlap weights, and entropy weights. These latter three sets of weights focus on estimating the effect of treatment in those subjects for whom there is clinical equipoise. We conducted a series of simulations to explore differences in the value of the target estimands for these five sets of weights when the difference in means is the measure of treatment effect.

METHODS

We considered 648 scenarios defined by different values of the prevalence of treatment, the c-statistic of the propensity score model, the correlation between the linear predictors for treatment selection and the outcome, and by the magnitude of the interaction between treatment status and the linear predictor for the outcome in the absence of treatment.

RESULTS

We found that, when the prevalence of treatment was low or high and the c-statistic of the propensity score model was moderate to high, that matching weights, overlap weights, and entropy weights had target estimands that differed meaningfully from the target estimand of the ATE weights.

CONCLUSIONS

Researchers using matching weights, overlap weights, and entropy weights should not assume that the estimated treatment effect is comparable to the ATE.