Heterogeneity and the Interpretation of Treatment Effect Estimates From Risk Adjustment and Instrumental Variable Methods

Assessing the properties of patient-specific treatment effect estimates from causal forest algorithms under essential heterogeneity

BACKGROUND

Treatment variation from observational data has been used to estimate patient-specific treatment effects. Causal Forest Algorithms (CFAs) developed for this task have unknown properties when treatment effect heterogeneity from unmeasured patient factors influences treatment choice - essential heterogeneity.

METHODS

We simulated eleven populations with identical treatment effect distributions based on patient factors. The populations varied in the extent that treatment effect heterogeneity influenced treatment choice. We used the generalized random forest application (CFA-GRF) to estimate patient-specific treatment effects for each population. Average differences between true and estimated effects for patient subsets were evaluated.

RESULTS

CFA-GRF performed well across the population when treatment effect heterogeneity did not influence treatment choice. Under essential heterogeneity, however, CFA-GRF yielded treatment effect estimates that reflected true treatment effects only for treated patients and were on average greater than true treatment effects for untreated patients.

CONCLUSIONS

Patient-specific estimates produced by CFAs are sensitive to why patients in real-world practice make different treatment choices. Researchers using CFAs should develop conceptual frameworks of treatment choice prior to estimation to guide estimate interpretation ex post.

Generating Practice-Based Evidence in the Use of Guideline-Recommended Combination Therapy for Secondary Prevention of Acute Myocardial Infarction

Background: Clinical guidelines recommend beta-blockers, angiotensin-converting enzyme inhibitors/angiotensin-receptor blockers, and statins for the secondary prevention of acute myocardial infarction (AMI). It is not clear whether variation in real-world practice reflects poor quality-of-care or a balance of outcome tradeoffs across patients. Methods: The study cohort included Medicare fee-for-service beneficiaries hospitalized 2007-2008 for AMI. Treatment within 30-days post-discharge was grouped into one of eight possible combinations for the three drug classes. Outcomes included one-year overall survival, one-year cardiovascular-event-free survival, and 90-day adverse events. Treatment effects were estimated using an Instrumental Variables (IV) approach with instruments based on measures of local-area practice style. Pre-specified data elements were abstracted from hospital medical records for a stratified, random sample to create "unmeasured confounders" (per claims data) and assess model assumptions. Results: Each drug combination was observed in the final sample (N = 124,695), with 35.7% having all three, and 13.5% having none. Higher rates of guideline-recommended treatment were associated with both better survival and more adverse events. Unmeasured confounders were not associated with instrumental variable values. Conclusions: The results from this study suggest that providers consider both treatment benefits and harms in patients with AMIs. The investigation of estimator assumptions support the validity of the estimates.

Assessing the ability of an instrumental variable causal forest algorithm to personalize treatment evidence using observational data: the case of early surgery for shoulder fracture

Background

Comparative effectiveness research (CER) using observational databases has been suggested to obtain personalized evidence of treatment effectiveness. Inferential difficulties remain using traditional CER approaches especially related to designating patients to reference classes a priori. A novel Instrumental Variable Causal Forest Algorithm (IV-CFA) has the potential to provide personalized evidence using observational data without designating reference classes a priori, but the consistency of the evidence when varying key algorithm parameters remains unclear. We investigated the consistency of IV-CFA estimates through application to a database of Medicare beneficiaries with proximal humerus fractures (PHFs) that previously revealed heterogeneity in the effects of early surgery using instrumental variable estimators.

Methods

IV-CFA was used to estimate patient-specific early surgery effects on both beneficial and detrimental outcomes using different combinations of algorithm parameters and estimate variation was assessed for a population of 72,751 fee-for-service Medicare beneficiaries with PHFs in 2011. Classification and regression trees (CART) were applied to these estimates to create ex-post reference classes and the consistency of these classes were assessed. Two-stage least squares (2SLS) estimators were applied to representative ex-post reference classes to scrutinize the estimates relative to known 2SLS properties.

Results

IV-CFA uncovered substantial early surgery effect heterogeneity across PHF patients, but estimates for individual patients varied with algorithm parameters. CART applied to these estimates revealed ex-post reference classes consistent across algorithm parameters. 2SLS estimates showed that ex-post reference classes containing older, frailer patients with more comorbidities, and lower utilizers of healthcare were less likely to benefit and more likely to have detriments from higher rates of early surgery.

Conclusions

IV-CFA provides an illuminating method to uncover ex-post reference classes of patients based on treatment effects using observational data with a strong instrumental variable. Interpretation of treatment effect estimates within each ex-post reference class using traditional CER methods remains conditional on the extent of measured information in the data.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01663-0.

What are the health consequences associated with differences in medical malpractice liability laws? An instrumental variable analysis of surgery effects on health outcomes for proximal humeral facture across states with different liability rules

BACKGROUND

States enacted tort reforms to lower medical malpractice liability, which are associated with higher surgery rates among Medicare patients with shoulder conditions. Surgery in this group often entails tradeoffs between improved health and increased risk of morbidity and mortality. We assessed whether differences in surgery rates across states with different liability rules are associated with surgical outcomes among Medicare patients with proximal humeral fracture.

METHODS

We obtained data for 67,966 Medicare beneficiaries with a diagnosis of proximal humeral fracture in 2011. Outcome measures included adverse events, mortality, and treatment success rates, defined as surviving the treatment period with < $300 in shoulder-related expenditures. We used existing state-level tort reform rules as instruments for surgical treatment and separately as predictors to answer our research question, both for the full cohort and for stratified subgroups based on age and general health status measured by Charlson Comorbidity Index and Function-Related Indicators.

RESULTS

We found a 0.32 percentage-point increase (p < 0.05) in treatment success and a 0.21 percentage-point increase (p < 0.01) in mortality for every 1 percentage-point increase in surgery rates among patients in states with lower liability risk. In subgroup analyses, mortality increased among more vulnerable patients, by 0.29 percentage-point (p < 0.01) for patients with Charlson Comorbidity Index >  = 2 and by 0.45 percentage-point (p < 0.01) among those patients with Function-Related Indicator scores >  = 2. On the other hand, treatment success increased in patients with lower Function-Related Index scores (< 2) by 0.54 percentage-point (p < 0.001). However, younger Medicare patients (< 80 years) experienced an increase in both mortality (0.28 percentage-point, p < 0.01) and treatment success (0.89 percentage-point, p < 0.01). The reduced-form estimates are consistent with our instrumental variable results.

CONCLUSIONS

A tradeoff exists between increased mortality risk and increased treatment success across states with different malpractice risk levels. These results varied across patient subgroups, with more vulnerable patients generally bearing the brunt of the increased mortality and less vulnerable patients enjoying increased success rates. These findings highlight the important risk-reward scenario associated with different liability environments, especially among patients with different health status.

Plant different, eat different? Insights from participatory agricultural research

We examine the association between on-farm production diversity on household dietary diversity in Malawi using microdata collected as part of an environmentally sustainable agricultural intensification program. The program primarily focuses on the integration of legumes into the cropping system through maize-legume intercropping and legume-legume intercropping. Relative to staple cereals such as maize, legumes are rich in micronutrients, contain better-quality protein, and lead to nitrogen fixation. Given the systematic difference we document between program beneficiaries and randomly sampled non-beneficiary (control) households, we employ causal instrumental variables mediation analysis to account for non-random selection and possible simultaneity between production and consumption decisions. We find a significant positive treatment effect on dietary diversity, led by an increase in production diversity. Analysis of potential pathways show that effects on dietary diversity stem mostly from consumption of diverse food items purchased from the market made possible through higher agricultural income. These findings highlight that, while increasing production for markets can enhance dietary diversity through higher income that would make affordable an expanded set of food items, the production of more nutritious crops such as pulses may not necessarily translate into greater own consumption. This may be due to the persistence of dietary habits, tastes, or other local factors that favor consumption of staples such as maize and encourage sales of more profitable and nutritious food items such as pulses. Pulses are a more affordable and environmentally sustainable source of protein than animal source food, and efforts should be made to enhance their nutritional awareness and contribution to sustainable food systems and healthier diets.

Association of Surgical Treatment With Adverse Events and Mortality Among Medicare Beneficiaries With Proximal Humerus Fracture

IMPORTANCE

Meta-analyses of randomized clinical trials suggest that the advantages and risks of surgery compared with conservative management as the initial treatment for proximal humerus fracture (PHF) vary, or are heterogeneous across patients. Substantial geographic variation in surgery rates for PHF suggests that the optimal rate of surgery across the population of patients with PHF is unknown.

OBJECTIVE

To use geographic variation in treatment rates to assess the outcomes associated with higher rates of surgery for patients with PHF.

DESIGN, SETTING, AND PARTICIPANTS

This comparative effectiveness research study analyzed all fee-for-service Medicare beneficiaries with proximal humerus fracture in 2011 who were continuously enrolled in Medicare Parts A and B for the 365-day period before and immediately after their index fracture. Data analysis was performed January through June 2019.

EXPOSURE

Undergoing 1 of the commonly used surgical procedures in the 60 days after an index fracture diagnosis.

MAIN OUTCOMES AND MEASURES

Risk-adjusted area surgery ratios were created for each hospital referral region as a measure of local area practice styles. Instrumental variable approaches were used to assess the association between higher surgery rates and adverse events, mortality risk, and cost at 1 year from Medicare's perspective for patients with PHF in 2011. Instrumental variable models were stratified by age, comorbidities, and frailty. Instrumental variable estimates were compared with estimates from risk-adjusted regression models.

RESULTS

The final cohort included 72 823 patients (mean [SD] age, 80.0 [7.9] years; 13 958 [19.2%] men). The proportion of patients treated surgically ranged from 1.8% to 33.3% across hospital referral regions in the United States. Compared with conservatively managed patients, surgical patients were younger (mean [SD] age, 80.4 [8.1] years vs 78.0 [7.2] years; P < .001) and healthier (Charlson Comorbidity Index score of 0, 14 863 [24.4%] patients vs 3468 [29.1%] patients; Function-Related Indicator score of 0, 20 720 [34.0%] patients vs 4980 [41.8%] patients; P < .001 for both), and a larger proportion were women (49 030 [80.5%] patients vs 9835 [82.5%] patients; P < .001). Instrumental variable analysis showed that higher rates of surgery were associated with increased total costs ($8913) during the treatment period, increased adverse event rates (a 1-percentage point increase in the surgery rate was associated with a 0.19-percentage point increase in the 1-year adverse event rate; β = 0.19; 95% CI, 0.09-0.27; P < .001), and increased mortality risk (a 1-percentage point increase in the surgery rate was associated with a 0.09-percentage point increase in the 1-year mortality rate; β = 0.09; 95% CI, 0.04-0.15; P < .01). Instrumental variable mortality results were even more striking for older patients and those with higher comorbidity burdens and greater frailty. Risk-adjusted estimates suggested that surgical patients had higher costs (increase of $17 278) and more adverse events (a 1-percentage point increase in the surgery rate was associated with a 0.12-percentage point increase in the 1-year adverse event rate; β = 0.12; 95% CI, 0.11 to 0.13; P < .001) but lower risk of mortality after PHF (a 1-percentage point increase in the surgery rate was associated with a 0.01-percentage point decrease in the 1-year mortality rate; β = -0.01; 95% CI, -0.015 to -0.005; P < .001).

CONCLUSIONS AND RELEVANCE

This study found that higher rates of surgery for treatment of patients with PHF were associated with increased costs, adverse event rates, and risk of mortality. Orthopedic surgeons should be aware of the harms of extending the use of surgery to more clinically vulnerable patient subgroups.

Treatment for Rotator Cuff Tear Is Influenced by Demographics and Characteristics of the Area Where Patients Live

Background:

Atraumatic rotator cuff tear is a common orthopaedic complaint for people >60 years of age. Lack of evidence or consensus on appropriate treatment for this type of injury creates the potential for substantial discretion in treatment decisions. To our knowledge, no study has assessed the implications of this discretion on treatment patterns across the United States.

Methods:

All Medicare beneficiaries in the United States with a new magnetic resonance imaging (MRI)-confirmed atraumatic rotator cuff tear were identified with use of 2010 to 2012 Medicare administrative data and were categorized according to initial treatment (surgery, physical therapy, or watchful waiting). Treatment was modeled as a function of the clinical and demographic characteristics of each patient. Variation in treatment rates across hospital referral regions and the presence of area treatment signatures, representing the extent that treatment rates varied across hospital referral regions after controlling for patient characteristics, were assessed. Correlations between measures of area treatment signatures and measures of physician access in hospital referral regions were examined.

Results:

Among patients who were identified as having a new, symptomatic, MRI-confirmed atraumatic rotator cuff tear (n = 32,203), 19.8% were managed with initial surgery; 41.3%, with initial physical therapy; and 38.8%, with watchful waiting. Patients who were older, had more comorbidity, or were female, of non-white race, or dual-eligible for Medicaid were less likely to receive surgery (p < 0.0001). Black, dual-eligible females had 0.42-times (95% confidence interval [CI], 0.34 to 0.50) lower odds of surgery and 2.36-times (95% CI, 2.02 to 2.70) greater odds of watchful waiting. Covariate-adjusted odds of surgery varied dramatically across hospital referral regions; unadjusted surgery and physical therapy rates varied from 0% to 73% and from 6% to 74%, respectively. On average, patients in high-surgery areas were 62% more likely to receive surgery than the average patient with identical measured characteristics, and patients in low-surgery areas were half as likely to receive surgery than the average comparable patient. The supply of orthopaedic surgeons and the supply of physical therapists were associated with greater use of initial surgery and physical therapy, respectively.

Conclusions:

Patient characteristics had a significant influence on treatment for atraumatic rotator cuff tear but did not explain the wide-ranging variation in treatment rates across areas. Local-area physician supply and specialty mix were correlated with treatment, independent of the patient’s measured characteristics.

Comparative Effectiveness of Surgical Options for Patients with Ductal Carcinoma In Situ: An Instrumental Variable Approach

CONTEXT

Many patients with ductal carcinoma in situ (DCIS) receive treatment that is too extensive.

OBJECTIVE

To take a holistic approach in comparing the effectiveness in cancer prevention between mastectomy and breast-conserving surgery (BCS) for patients with DCIS.

DESIGN

Female Kaiser Permanente Southern California members who underwent surgery for treatment of single primary DCIS from 2004 to 2014 were identified by the Kaiser Permanente Southern California cancer registry and HealthConnect database.

METHOD

Two-stage residual inclusion with the surgeon's preference of surgical procedure type as the instrumental variable was used to examine the effect of surgical choice on DCIS recurrence, breast cancer progression, and other cancer progression. Traditional Cox proportional hazards models were used for comparison.

RESULTS

Of qualified subjects, 72.2% underwent BCS and 27.8% underwent mastectomy. Patients were likelier to receive BCS if their surgeon preferred to perform BCS in the past 5 years (odds ratio = 1.02, 95% confidence interval = 1.02-1.03). Although traditional Cox proportional hazards models suggested an association between BCS and higher risk of DCIS recurrence, no significant effect was observed when we adjusted for endogeneity. Neither model showed significant differences between mastectomy and BCS in progression of any cancer.

CONCLUSION

No significant benefit was observed with a more aggressive surgical procedure in preventing DCIS recurrence or cancer progression in a diverse population. Many patients with DCIS could benefit from BCS with preservation of their body image. Breast conservation followed-up with cancer surveillance is a rational approach to ensure affordable, effective care for patients with DCIS.

Understanding Treatment Effect Estimates When Treatment Effects Are Heterogeneous for More Than One Outcome

BACKGROUND

Patient-centred care requires evidence of treatment effects across many outcomes. Outcomes can be beneficial (e.g. increased survival or cure rates) or detrimental (e.g. adverse events, pain associated with treatment, treatment costs, time required for treatment). Treatment effects may also be heterogeneous across outcomes and across patients. Randomized controlled trials are usually insufficient to supply evidence across outcomes. Observational data analysis is an alternative, with the caveat that the treatments observed are choices. Real-world treatment choice often involves complex assessment of expected effects across the array of outcomes. Failure to account for this complexity when interpreting treatment effect estimates could lead to clinical and policy mistakes.

OBJECTIVE

Our objective was to assess the properties of treatment effect estimates based on choice when treatments have heterogeneous effects on both beneficial and detrimental outcomes across patients.

METHODS

Simulation methods were used to highlight the sensitivity of treatment effect estimates to the distributions of treatment effects across patients across outcomes. Scenarios with alternative correlations between benefit and detriment treatment effects across patients were used. Regression and instrumental variable estimators were applied to the simulated data for both outcomes.

RESULTS

True treatment effect parameters are sensitive to the relationships of treatment effectiveness across outcomes in each study population. In each simulation scenario, treatment effect estimate interpretations for each outcome are aligned with results shown previously in single outcome models, but these estimates vary across simulated populations with the correlations of treatment effects across patients across outcomes.

CONCLUSIONS

If estimator assumptions are valid, estimates across outcomes can be used to assess the optimality of treatment rates in a study population. However, because true treatment effect parameters are sensitive to correlations of treatment effects across outcomes, decision makers should be cautious about generalizing estimates to other populations.

Treatment Effect Estimation Using Nonlinear Two-Stage Instrumental Variable Estimators: Another Cautionary Note

OBJECTIVE

To examine the settings of simulation evidence supporting use of nonlinear two-stage residual inclusion (2SRI) instrumental variable (IV) methods for estimating average treatment effects (ATE) using observational data and investigate potential bias of 2SRI across alternative scenarios of essential heterogeneity and uniqueness of marginal patients.

STUDY DESIGN

Potential bias of linear and nonlinear IV methods for ATE and local average treatment effects (LATE) is assessed using simulation models with a binary outcome and binary endogenous treatment across settings varying by the relationship between treatment effectiveness and treatment choice.

PRINCIPAL FINDINGS

Results show that nonlinear 2SRI models produce estimates of ATE and LATE that are substantially biased when the relationships between treatment and outcome for marginal patients are unique from relationships for the full population. Bias of linear IV estimates for LATE was low across all scenarios.

CONCLUSIONS

Researchers are increasingly opting for nonlinear 2SRI to estimate treatment effects in models with binary and otherwise inherently nonlinear dependent variables, believing that it produces generally unbiased and consistent estimates. This research shows that positive properties of nonlinear 2SRI rely on assumptions about the relationships between treatment effect heterogeneity and choice.

Validation of disability status, a claims-based measure of functional status for cancer treatment and outcomes studies

BACKGROUND

In prior research, we developed a claims-based prediction model for poor patient disability status (DS), a proxy measure for performance status, commonly used by oncologists to summarize patient functional status and assess ability of a patient to tolerate aggressive treatment. In this study, we implemented and validated the DS measure in 4 cohorts of cancer patients: early and advanced non-small cell lung cancers (NSCLC), stage IV estrogen receptor-negative (ER-) breast cancer, and myelodysplastic syndromes (MDS).

DATA AND METHODS

SEER-Medicare data (1999-2007) for the 4 cohorts of cancer patients. Bivariate and multivariate logistic regression tested the association of the DS measure with designated cancer-directed treatments: early NSCLC (surgery), advanced NSCLC (chemotherapy), stage IV ER- breast cancer (chemotherapy), and MDS (erythropoiesis-stimulating agents). Treatment model fit was compared across model iterations.

RESULTS

In both unadjusted and adjusted results, predicted poor DS was strongly associated with a lower likelihood of cancer treatment receipt in all 4 cohorts [early NSCLC (N=20,280), advanced NSCLC (N=31,341), stage IV ER- breast cancer (N=1519), and MDS (N=6058)] independent of other patient, contextual, and disease characteristics, as well as the Charlson Comorbidity Index. Inclusion of the DS measure into models already controlling for other variables did not significantly improve model fit across the cohorts.

CONCLUSIONS

The DS measure is a significant independent predictor of cancer-directed treatment. Small changes in model fit associated with both DS and the Charlson Comorbidity Index suggest that unobserved factors continue to play a role in determining cancer treatments.

What is the effect of area size when using local area practice style as an instrument?

OBJECTIVES

Discuss the tradeoffs inherent in choosing a local area size when using a measure of local area practice style as an instrument in instrumental variable estimation when assessing treatment effectiveness.

STUDY DESIGN

Assess the effectiveness of angiotensin converting-enzyme inhibitors and angiotensin receptor blockers on survival after acute myocardial infarction for Medicare beneficiaries using practice style instruments based on different-sized local areas around patients. We contrasted treatment effect estimates using different local area sizes in terms of the strength of the relationship between local area practice styles and individual patient treatment choices; and indirect assessments of the assumption violations.

RESULTS

Using smaller local areas to measure practice styles exploits more treatment variation and results in smaller standard errors. However, if treatment effects are heterogeneous, the use of smaller local areas may increase the risk that local practice style measures are dominated by differences in average treatment effectiveness across areas and bias results toward greater effectiveness.

CONCLUSION

Local area practice style measures can be useful instruments in instrumental variable analysis, but the use of smaller local area sizes to generate greater treatment variation may result in treatment effect estimates that are biased toward higher effectiveness. Assessment of whether ecological bias can be mitigated by changing local area size requires the use of outside data sources.

Methods in comparative effectiveness research

Comparative effectiveness research (CER) seeks to assist consumers, clinicians, purchasers, and policy makers to make informed decisions to improve health care at both the individual and population levels. CER includes evidence generation and evidence synthesis. Randomized controlled trials are central to CER because of the lack of selection bias, with the recent development of adaptive and pragmatic trials increasing their relevance to real-world decision making. Observational studies comprise a growing proportion of CER because of their efficiency, generalizability to clinical practice, and ability to examine differences in effectiveness across patient subgroups. Concerns about selection bias in observational studies can be mitigated by measuring potential confounders and analytic approaches, including multivariable regression, propensity score analysis, and instrumental variable analysis. Evidence synthesis methods include systematic reviews and decision models. Systematic reviews are a major component of evidence-based medicine and can be adapted to CER by broadening the types of studies included and examining the full range of benefits and harms of alternative interventions. Decision models are particularly suited to CER, because they make quantitative estimates of expected outcomes based on data from a range of sources. These estimates can be tailored to patient characteristics and can include economic outcomes to assess cost effectiveness. The choice of method for CER is driven by the relative weight placed on concerns about selection bias and generalizability, as well as pragmatic concerns related to data availability and timing. Value of information methods can identify priority areas for investigation and inform research methods.

Survival implications associated with variation in mastectomy rates for early-staged breast cancer

Despite a 20-year-old guideline from the National Institutes of Health (NIH) Consensus Development Conference recommending breast conserving surgery with radiation (BCSR) over mastectomy for woman with early-stage breast cancer (ESBC) because it preserves the breast, recent evidence shows mastectomy rates increasing and higher-staged ESBC patients are more likely to receive mastectomy. These observations suggest that some patients and their providers believe that mastectomy has advantages over BCSR and these advantages increase with stage. These beliefs may persist because the randomized controlled trials (RCTs) that served as the basis for the NIH guideline were populated mainly with lower-staged patients. Our objective is to assess the survival implications associated with mastectomy choice by patient alignment with the RCT populations. We used instrumental variable methods to estimate the relationship between surgery choice and survival for ESBC patients based on variation in local area surgery styles. We find results consistent with the RCTs for patients closely aligned to the RCT populations. However, for patients unlike those in the RCTs, our results suggest that higher mastectomy rates are associated with reduced survival. We are careful to interpret our estimates in terms of limitations of our estimation approach.

Comparison of instrumental variable analysis using a new instrument with risk adjustment methods to reduce confounding by indication

Confounding by indication is a vexing problem, especially in evaluating treatment effects using observational data, since treatment decisions are often related to disease severity, prognosis, and frailty. To compare the ability of the instrumental variable (IV) approach with a new instrument based on the local-area practice style and risk adjustment methods, including conventional multivariate regression and propensity score adjustment, to reduce confounding by indication, the authors investigated the effects of long-term control (LTC) therapy on the occurrence of acute asthma exacerbation events among children and young adults with incident and uncontrolled persistent asthma, using Iowa Medicaid claims files from 1997-1999. Established evidence from clinical trials has demonstrated the protective benefits of LTC therapy for persistent asthma. Among patients identified (n = 4,275), those with higher asthma severity at baseline were more likely to receive LTC therapy. The multivariate regression and propensity score adjustment methods suggested that LTC therapy had no effect on the occurrence of acute exacerbation events. Estimates from the new IV approach showed that LTC therapy significantly decreased the occurrence of acute exacerbation events, which is consistent with established clinical evidence. The authors discuss how to interpret estimates from the risk adjustment and IV methods when the treatment effect is heterogeneous.

Apples and oranges? Interpretations of risk adjustment and instrumental variable estimates of intended treatment effects using observational data

Instrumental variable (IV) and risk adjustment (RA) estimators, including propensity score adjustments, are both used to alleviate confounding problems in nonexperimental studies on treatment effects, but it is not clear how estimates based on these 2 approaches compare. Methodological considerations have shown that IV and RA estimators yield estimates of distinct types of causal treatment effects regardless of confounding problems. Many investigators have neglected these distinctions. In this paper, the authors use 3 schematic models to explain visually the relations between IV and RA estimates of intended treatment effects as demonstrated in the methodological studies. When treatment effects are homogeneous across a study population or when treatment effects are heterogeneous across the study population but treatment decisions are unrelated to the treatment effects, RA and IV estimates should be equivalent when the respective assumptions are met. In contrast, when treatment effects are heterogeneous and treatment decisions are related to the treatment effects, RA estimates of treatment effect can asymptotically differ from IV estimates, but both are correct even when the respective assumptions are met. Appropriate interpretations of IV or RA estimates can be facilitated by developing conceptual models related to treatment choice and treatment effect heterogeneity prior to analyses.

Some cautions on the use of instrumental variables estimators in outcomes research: how bias in instrumental variables estimators is affected by instrument strength, instrument contamination, and sample size

OBJECTIVES

To examine the performance of instrumental variables (IV) and ordinary least squares (OLS) regression under a range of conditions likely to be encountered in empirical research.

METHODS

A series of simulation analyses are carried out to compare estimation error between OLS and IV when the independent variable of interest is endogenous. The simulations account for a range of situations that may be encountered by researchers in actual practice-varying degrees of endogeneity, instrument strength, instrument contamination, and sample size. The intent of this article is to provide researchers with more intuition with respect to how important these factors are from an empirical standpoint.

RESULTS

Notably, the simulations indicate a greater potential for inferential error when using IV than OLS in all but the most ideal circumstances.

CONCLUSIONS

Researchers should be cautious when using IV methods. These methods are valuable in testing for the presence of endogeneity but only under the most ideal circumstances are they likely to produce estimates with less estimation error than OLS.

The effect of dental care on cardiovascular disease outcomes: an application of instrumental variables in the presence of heterogeneity and self-selection

Studies show a relationship between oral inflammatory processes and cardiovascular risk factors, suggesting that dental care may reduce the risk of cardiovascular disease (CVD) events. However, due to the differences between men and women in the development and presentation of CVD, such effects may vary by sex. We use a valid set of instrumental variables to evaluate these issues and include a test of essential heterogeneity. CVD events include new occurrences of heart attack (including death from heart attack), stroke (including death from stroke), angina, and congestive heart failure. Controls include age, race, education, marital status, foreign birthplace, and cardiovascular risk factors (health status, body mass index, alcohol use, smoking status, diabetes status, high-blood-pressure status, physical activity, and depression). Our analysis finds no evidence of essential heterogeneity. We find the minimum average treatment effect for women to be -0.01, but find no treatment effect for men. This suggests that women who receive dental care may reduce their risk of future CVD events by at least one-third. The findings may only apply to married middle-aged and older individuals as the data set is only representative for this group.

Comparative effectiveness of prostate cancer treatments: evaluating statistical adjustments for confounding in observational data

BACKGROUND

Using observational data to assess the relative effectiveness of alternative cancer treatments is limited by patient selection into treatment, which often biases interpretation of outcomes. We evaluated methods for addressing confounding in treatment and survival of patients with early-stage prostate cancer in observational data and compared findings with those from a benchmark randomized clinical trial.

METHODS

We selected 14 302 early-stage prostate cancer patients who were aged 66-74 years and had been treated with radical prostatectomy or conservative management from linked Surveillance, Epidemiology, and End Results-Medicare data from January 1, 1995, through December 31, 2003. Eligibility criteria were similar to those from a clinical trial used to benchmark our analyses. Survival was measured through December 31, 2007, by use of Cox proportional hazards models. We compared results from the benchmark trial with results from models with observational data by use of traditional multivariable survival analysis, propensity score adjustment, and instrumental variable analysis.

RESULTS

Prostate cancer patients receiving conservative management were more likely to be older, nonwhite, and single and to have more advanced disease than patients receiving radical prostatectomy. In a multivariable survival analysis, conservative management was associated with greater risk of prostate cancer-specific mortality (hazard ratio [HR] = 1.59, 95% confidence interval [CI] = 1.27 to 2.00) and all-cause mortality (HR = 1.47, 95% CI = 1.35 to 1.59) than radical prostatectomy. Propensity score adjustments resulted in similar patient characteristics across treatment groups, although survival results were similar to traditional multivariable survival analyses. Results for the same comparison from the instrumental variable approach, which theoretically equalizes both observed and unobserved patient characteristics across treatment groups, differed from the traditional multivariable and propensity score results but were consistent with findings from the subset of elderly patient with early-stage disease in the trial (ie, conservative management vs radical prostatectomy: for prostate cancer-specific mortality, HR = 0.73, 95% CI = 0.08 to 6.73; for all-cause mortality, HR = 1.09, 95% CI = 0.46 to 2.59).

CONCLUSION

Instrumental variable analysis may be a useful technique in comparative effectiveness studies of cancer treatments if an acceptable instrument can be identified.

Instrumental variable methods in comparative safety and effectiveness research

Instrumental variable (IV) methods have been proposed as a potential approach to the common problem of uncontrolled confounding in comparative studies of medical interventions, but IV methods are unfamiliar to many researchers. The goal of this article is to provide a non-technical, practical introduction to IV methods for comparative safety and effectiveness research. We outline the principles and basic assumptions necessary for valid IV estimation, discuss how to interpret the results of an IV study, provide a review of instruments that have been used in comparative effectiveness research, and suggest some minimal reporting standards for an IV analysis. Finally, we offer our perspective of the role of IV estimation vis-à-vis more traditional approaches based on statistical modeling of the exposure or outcome. We anticipate that IV methods will be often underpowered for drug safety studies of very rare outcomes, but may be potentially useful in studies of intended effects where uncontrolled confounding may be substantial.

Good research practices for comparative effectiveness research: analytic methods to improve causal inference from nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report--Part III

OBJECTIVES

Most contemporary epidemiologic studies require complex analytical methods to adjust for bias and confounding. New methods are constantly being developed, and older more established methods are yet appropriate. Careful application of statistical analysis techniques can improve causal inference of comparative treatment effects from nonrandomized studies using secondary databases. A Task Force was formed to offer a review of the more recent developments in statistical control of confounding.

METHODS

The Task Force was commissioned and a chair was selected by the ISPOR Board of Directors in October 2007. This Report, the third in this issue of the journal, addressed methods to improve causal inference of treatment effects for nonrandomized studies.

RESULTS

The Task Force Report recommends general analytic techniques and specific best practices where consensus is reached including: use of stratification analysis before multivariable modeling, multivariable regression including model performance and diagnostic testing, propensity scoring, instrumental variable, and structural modeling techniques including marginal structural models, where appropriate for secondary data. Sensitivity analyses and discussion of extent of residual confounding are discussed.

CONCLUSIONS

Valid findings of causal therapeutic benefits can be produced from nonrandomized studies using an array of state-of-the-art analytic techniques. Improving the quality and uniformity of these studies will improve the value to patients, physicians, and policymakers worldwide.

Interpreting treatment-effect estimates with heterogeneity and choice: simulation model results

BACKGROUND

Researchers using observational data in health-services research use various treatment-effect estimators to reduce the bias associated with unmeasured confounding variables and have focused on estimate differences to indicate the relative ability of these estimators to mitigate bias. However, available estimators may identify different treatment-effect concepts; if treatment effects are heterogeneous across patients and treatment choice reflects "sorting on the gain," then treatment-effect estimates should differ regardless of confounding. Risk-adjustment approaches yield estimates of the average treatment effect on the treated (ATT), whereas instrumental variable approaches yield estimates of a local average treatment effect (LATE).

OBJECTIVE

The goal of this article was to use simulation methods to illustrate the treatment-effect concepts that are identified using observational data with various estimators.

METHODS

We simulated patient treatment choices based on expected treatment valuation to observe estimates of both ATT and LATE. Different model scenarios were run to isolate the effects of both treatment-effect heterogeneity and unmeasured confounding on treatment-effect concept estimation. Models were estimated using standard linear and nonlinear estimation methods.

RESULTS

We show that the true values of the underlying treatment concepts differ if patients (with the help of their health care providers) make treatment choices based on expected gains, and that distinct estimators produce estimates of distinct concepts. In scenarios without unmeasured confounding, both linear and nonlinear estimation models produced estimates close to the true value of the concept identified by each estimator. However, nonlinear models suggested additional treatment-effect heterogeneity that does not exist in these scenarios.

CONCLUSIONS

Our results suggest that, to ensure clarity and correctness of treatment-effect estimate interpretation, it is important for researchers to state the treatment-effect concept that they are trying to identify before beginning estimation. In addition, theoretical models of treatment choice are needed to provide the foundation linking treatment-effect estimates to treatment-effect concepts and to justify instrument selection.

Instrumental variables I: instrumental variables exploit natural variation in nonexperimental data to estimate causal relationships

The gold standard of study design for treatment evaluation is widely acknowledged to be the randomized controlled trial (RCT). Trials allow for the estimation of causal effect by randomly assigning participants either to an intervention or comparison group; through the assumption of "exchangeability" between groups, comparing the outcomes will yield an estimate of causal effect. In the many cases where RCTs are impractical or unethical, instrumental variable (IV) analysis offers a nonexperimental alternative based on many of the same principles. IV analysis relies on finding a naturally varying phenomenon, related to treatment but not to outcome except through the effect of treatment itself, and then using this phenomenon as a proxy for the confounded treatment variable. This article demonstrates how IV analysis arises from an analogous but potentially impossible RCT design, and outlines the assumptions necessary for valid estimation. It gives examples of instruments used in clinical epidemiology and concludes with an outline on estimation of effects.

Emerging Methods in Comparative Effectiveness and Safety

BACKGROUND

Interest in new methods for comparative effectiveness, drug and patient safety, and related studies is burgeoning. The advent of Medicare Part D for outpatient prescription drugs has drawn significant attention to the need for efficient ways to monitor the potential benefits and harms of pharmaceuticals. These trends prompted the Effective Health Care program at the Agency for Healthcare Research and Quality and its DEcIDE (Developing Evidence to Inform Decisions about Effectiveness) network to examine innovative approaches for such investigations through an invitational symposium in June 2006.

RESULTS

Conference papers covered numerous points about ways to structure both interventional and database-oriented studies, particularly those concerned with adverse drug events, to avoid bias in those studies, and to apply advanced statistical tools to exploit the information from these studies to their fullest. Of particular importance are: (1) using new types of experimental designs, including cluster randomization, delayed designs, pragmatic trials, and practice-based investigations that incorporate the natural variation of data from routine clinical practice; (2) finding efficient ways to use different types of databases-eg, Department of Veterans Affairs files, Centers for Disease Control and Prevention surveillance files, Medicaid claims data, and state hospital data-for examining initiation, persistence, and adherence, and the benefits and adverse events of pharmaceutical use; and (3) inventing or refining ways to decrease the threats to validity of analyses relying on administrative or other observational data, particularly through propensity scoring, inverse probability weighting, risk adjustment, and direct or indirect methods for synthesizing comparative effectiveness information.