What is the best way to estimate hospital quality outcomes? A simulation approach

Improving target price calculations in Medicare bundled payment programs

OBJECTIVE

To compare the predictive accuracy of two approaches to target price calculations under Bundled Payments for Care Improvement-Advanced (BPCI-A): the traditional Centers for Medicare and Medicaid Services (CMS) methodology and an empirical Bayes approach designed to mitigate the effects of regression to the mean.

DATA SOURCES

Medicare fee-for-service claims for beneficiaries discharged from acute care hospitals between 2010 and 2016.

STUDY DESIGN

We used data from a baseline period (discharges between January 1, 2010 and September 30, 2013) to predict spending in a performance period (discharges between October 1, 2015 and June 30, 2016). For 23 clinical episode types in BPCI-A, we compared the average prediction error across hospitals associated with each statistical approach. We also calculated an average across all clinical episode types and explored differences by hospital size.

DATA COLLECTION/EXTRACTION METHODS

We used a 20% sample of Medicare claims, excluding hospitals and episode types with small numbers of observations.

PRINCIPAL FINDINGS

The empirical Bayes approach resulted in significantly more accurate episode spending predictions for 19 of 23 clinical episode types. Across all episode types, prediction error averaged $8456 for the CMS approach versus $7521 for the empirical Bayes approach. Greater improvements in accuracy were observed with increasing hospital size.

CONCLUSIONS

CMS should consider using empirical Bayes methods to calculate target prices for BPCI-A.

Ranking hospitals when performance and risk factors are correlated: A simulation-based comparison of risk adjustment approaches for binary outcomes

Background

The conceptualization of hospital quality indicators usually includes some form of risk adjustment to account for hospital differences in case mix. For binary outcome variables like in-hospital mortality, frequently utilized risk adjusted measures include the standardized mortality ratio (SMR), the risk standardized mortality rate (RSMR), and excess risk (ER). All of these measures require the estimation of expected hospital mortality, which is often based on logistic regression models. In this context, an issue that is often neglected is correlation between hospital performance (e.g. care quality) and patient-specific risk factors. The objective of this study was to investigate the impact of such correlation on the adequacy of hospital rankings based on different measures and methods.

Methods

Using Monte Carlo simulation, the impact of correlation between hospital care quality and patient-specific risk factors on the adequacy of hospital rankings was assessed for SMR/RSMR, and ER based on logistic regression and random effects logistic regression. As an alternative method, fixed effects logistic regression with Firth correction was considered. The adequacies of the resulting hospital rankings were assessed by the shares of hospitals correctly classified into quintiles according to their true (unobserved) care qualities.

Results

The performance of risk adjustment approaches based on logistic regression and random effects logistic regression declined when correlation between care quality and a risk factor was induced. In contrast, fixed-effects-based estimations proved to be more robust. This was particularly true for fixed-effects-logistic-regression-based ER. In the absence of correlation between risk factors and care quality, all approaches showed similar performance.

Conclusions

Correlation between risk factors and hospital performance may severely bias hospital rankings based on logistic regression and random effects logistic regression. ER based on fixed effects logistic regression with Firth correction should be considered as an alternative approach to assess hospital performance.

Association Between 30-Day Mortality After Percutaneous Coronary Intervention and Education and Certification Variables for New York State Interventional Cardiologists

BACKGROUND

Patients and other providers have access to few publicly available physician attributes that identify interventional cardiologists with better postprocedural outcomes, particularly in states without public reporting of outcomes. Interventional cardiology board certification, maintenance of certification, graduation from a US medical school, medical school ranking, and length of practice represent such publicly available attributes. Previous studies on these measures have shown mixed results.

METHODS AND RESULTS

We included interventional cardiologists practicing in New York State in the years 2011 to 2013. The primary outcome was 30-day risk-standardized mortality rate (RSMR) after percutaneous coronary intervention. Hierarchical regression modeling was used to analyze the physician attributes and was adjusted for provider caseload. A total of 356 providers were studied. The average 30-day RSMR was 1.1 (SD=0.1) deaths per 100 cases for all percutaneous coronary interventions and 0.7 (SD=0.1) deaths per 100 cases for nonemergent procedures. The primary outcome was slightly lower among providers with interventional cardiology board certification compared with noncertified providers (1.06 [SD=0.14] versus 1.14 [SD=0.14] deaths per 100 cases; P<0.001). In multivariable hierarchical regression modeling, after adjusting for provider caseload, none of the physician attributes were associated with the primary outcome. Provider caseload was significantly associated with 30-day RSMR independent of the other attributes.

CONCLUSIONS

Interventional cardiology board-certified providers had a modestly lower 30-day RSMR before accounting for caseload. However, after adjusting for provider caseload, none of the examined publicly available physician attributes, including interventional cardiology board certification, were independently associated with 30-day RSMR.

Observed to expected or logistic regression to identify hospitals with high or low 30-day mortality?

INTRODUCTION

A common quality indicator for monitoring and comparing hospitals is based on death within 30 days of admission. An important use is to determine whether a hospital has higher or lower mortality than other hospitals. Thus, the ability to identify such outliers correctly is essential. Two approaches for detection are: 1) calculating the ratio of observed to expected number of deaths (OE) per hospital and 2) including all hospitals in a logistic regression (LR) comparing each hospital to a form of average over all hospitals. The aim of this study was to compare OE and LR with respect to correctly identifying 30-day mortality outliers. Modifications of the methods, i.e., variance corrected approach of OE (OE-Faris), bias corrected LR (LR-Firth), and trimmed mean variants of LR and LR-Firth were also studied.

MATERIALS AND METHODS

To study the properties of OE and LR and their variants, we performed a simulation study by generating patient data from hospitals with known outlier status (low mortality, high mortality, non-outlier). Data from simulated scenarios with varying number of hospitals, hospital volume, and mortality outlier status, were analysed by the different methods and compared by level of significance (ability to falsely claim an outlier) and power (ability to reveal an outlier). Moreover, administrative data for patients with acute myocardial infarction (AMI), stroke, and hip fracture from Norwegian hospitals for 2012-2014 were analysed.

RESULTS

None of the methods achieved the nominal (test) level of significance for both low and high mortality outliers. For low mortality outliers, the levels of significance were increased four- to fivefold for OE and OE-Faris. For high mortality outliers, OE and OE-Faris, LR 25% trimmed and LR-Firth 10% and 25% trimmed maintained approximately the nominal level. The methods agreed with respect to outlier status for 94.1% of the AMI hospitals, 98.0% of the stroke, and 97.8% of the hip fracture hospitals.

CONCLUSION

We recommend, on the balance, LR-Firth 10% or 25% trimmed for detection of both low and high mortality outliers.

Impact on hospital ranking of basing readmission measures on a composite endpoint of death or readmission versus readmissions alone

BACKGROUND

Readmission penalties are central to the Centers for Medicare and Medicaid Services (CMS) efforts to improve patient outcomes and reduce health care spending. However, many clinicians believe that readmission metrics may unfairly penalize low-mortality hospitals because mortality and readmission are competing risks. The objective of this study is to compare hospital ranking based on a composite outcome of death or readmission versus readmission alone.

METHODS

We performed a retrospective observational study of 344,565 admissions for acute myocardial infarction (AMI), congestive heart failure (CHF), or pneumoniae (PNEU) using population-based data from the New York State Inpatient Database (NY SID) between 2011 and 2013. Hierarchical logistic regression modeling was used to estimate separate risk-adjustment models for the (1) composite outcome (in-hospital death or readmission within 7-days), and (2) 7-day readmission. Hospital rankings based on the composite measure and the readmission measure were compared using the intraclass correlation coefficient and kappa analysis.

RESULTS

Using data from all AMI, CHF, and PNEU admissions, there was substantial agreement between hospital adjusted odds ratio (AOR) based on the composite outcome versus the readmission outcome (intraclass correlation coefficient [ICC] 0.67; 95% CI: 0.56, 0.75). For patients admitted with AMI, there was moderate agreement (ICC 0.53; 95% CI: 0.41, 0.62); for CHF, substantial agreement (ICC 0.72; 95% CI: 0.66, 0.78); and for PNEU, substantial agreement (ICC 0.71; 95% CI: 0.61, 0.78). There was moderate agreement when the composite and readmission metrics were used to classify hospitals as high, average, and low-performance hospitals (κ = 0.54, SE = 0.050). For patients admitted with AMI, there was slight agreement (κ = 0.14, SE = 0.037) between the two metrics.

CONCLUSIONS

Hospital performance on readmissions is significantly different from hospital performance on a composite metric based on readmissions and mortality. CMS and policy makers should consider re-assessing the use of readmission metrics for measuring hospital performance.

Quantifying Geographic Variation in Health Care Outcomes in the United States before and after Risk-Adjustment

BACKGROUND

Despite numerous studies of geographic variation in healthcare cost and utilization at the local, regional, and state levels across the U.S., a comprehensive characterization of geographic variation in outcomes has not been published. Our objective was to quantify variation in US health outcomes in an all-payer population before and after risk-adjustment.

METHODS AND FINDINGS

We used information from 16 independent data sources, including 22 million all-payer inpatient admissions from the Healthcare Cost and Utilization Project (which covers regions where 50% of the U.S. population lives) to analyze 24 inpatient mortality, inpatient safety, and prevention outcomes. We compared outcome variation at state, hospital referral region, hospital service area, county, and hospital levels. Risk-adjusted outcomes were calculated after adjusting for population factors, co-morbidities, and health system factors. Even after risk-adjustment, there exists large geographical variation in outcomes. The variation in healthcare outcomes exceeds the well publicized variation in US healthcare costs. On average, we observed a 2.1-fold difference in risk-adjusted mortality outcomes between top- and bottom-decile hospitals. For example, we observed a 2.3-fold difference for risk-adjusted acute myocardial infarction inpatient mortality. On average a 10.2-fold difference in risk-adjusted patient safety outcomes exists between top and bottom-decile hospitals, including an 18.3-fold difference for risk-adjusted Central Venous Catheter Bloodstream Infection rates. A 3.0-fold difference in prevention outcomes exists between top- and bottom-decile counties on average; including a 2.2-fold difference for risk-adjusted congestive heart failure admission rates. The population, co-morbidity, and health system factors accounted for a range of R2 between 18-64% of variability in mortality outcomes, 3-39% of variability in patient safety outcomes, and 22-70% of variability in prevention outcomes.

CONCLUSION

The amount of variability in health outcomes in the U.S. is large even after accounting for differences in population, co-morbidities, and health system factors. These findings suggest that: 1) additional examination of regional and local variation in risk-adjusted outcomes should be a priority; 2) assumptions of uniform hospital quality that underpin rationale for policy choices (such as narrow insurance networks or antitrust enforcement) should be challenged; and 3) there exists substantial opportunity for outcomes improvement in the US healthcare system.

Modern Management Practices and Hospital Admissions

We investigate whether the modern management practices and publicly reported performance measures are associated with choice of hospital for patients with acute myocardial infarction (AMI). We define and measure management practices at approximately half of US cardiac care units using a novel survey approach. A patient's choice of a hospital is modeled as a function of the hospital's performance on publicly reported quality measures and the quality of its management. The estimates, based on a grouped conditional logit specification, reveal that higher management scores and better performance on publicly reported quality measures are positively associated with hospital choice. Management practices appear to have a direct correlation with admissions for AMI--potentially through reputational effects--and indirect association, through better performance on publicly reported measures. Overall, a one standard deviation change in management practice scores is associated with an 8% increase in AMI admissions.

Measuring Returns to Hospital Care: Evidence from Ambulance Referral Patterns

Medicare spending exceeds 4% of GDP in the US each year, and there are concerns that moral hazard problems have led to overspending. This paper considers whether hospitals that treat patients more aggressively and receive higher payments from Medicare improve health outcomes for their patients. An innovation is a new lens to compare hospital performance for emergency patients: plausibly exogenous variation in ambulance-company assignment among patients who live near one another. Using Medicare data from 2002-2010, we show that ambulance company assignment importantly affects hospital choice for patients in the same ZIP code. Using data for New York State from 2000-2006 that matches exact patient addresses to hospital discharge records, we show that patients who live very near each other but on either side of ambulance service area boundaries go to different types of hospitals. Both identification strategies show that higher-cost hospitals achieve better patient outcomes for a variety of emergency conditions. Using our Medicare sample, the estimates imply that a one standard deviation increase in Medicare reimbursement leads to a 4 percentage point reduction in mortality (10% compared to the mean). Taking into account one-year spending after the health shock, the implied cost per at least one year of life saved is approximately $80,000. These results are found across different types of hospitals and patients, as well across both identification strategies.

HIV quality report cards: impact of case-mix adjustment and statistical methods

BACKGROUND

There will be increasing pressure to publicly report and rank the performance of healthcare systems on human immunodeficiency virus (HIV) quality measures. To inform discussion of public reporting, we evaluated the influence of case-mix adjustment when ranking individual care systems on the viral control quality measure.

METHODS

We used data from the Veterans Health Administration (VHA) HIV Clinical Case Registry and administrative databases to estimate case-mix adjusted viral control for 91 local systems caring for 12 368 patients. We compared results using 2 adjustment methods, the observed-to-expected estimator and the risk-standardized ratio.

RESULTS

Overall, 10 913 patients (88.2%) achieved viral control (viral load ≤400 copies/mL). Prior to case-mix adjustment, system-level viral control ranged from 51% to 100%. Seventeen (19%) systems were labeled as low outliers (performance significantly below the overall mean) and 11 (12%) as high outliers. Adjustment for case mix (patient demographics, comorbidity, CD4 nadir, time on therapy, and income from VHA administrative databases) reduced the number of low outliers by approximately one-third, but results differed by method. The adjustment model had moderate discrimination (c statistic = 0.66), suggesting potential for unadjusted risk when using administrative data to measure case mix.

CONCLUSIONS

Case-mix adjustment affects rankings of care systems on the viral control quality measure. Given the sensitivity of rankings to selection of case-mix adjustment methods-and potential for unadjusted risk when using variables limited to current administrative databases-the HIV care community should explore optimal methods for case-mix adjustment before moving forward with public reporting.

Measurement and risk adjustment of prelabor cesarean rates in a large sample of California hospitals

OBJECTIVE

Prelabor cesareans in women without a prior cesarean is an important quality measure, yet one that is seldom tracked. We estimated patient-level risks and calculated how sensitive hospital rankings on this proposed quality metric were to risk adjustment.

STUDY DESIGN

This retrospective cohort study linked Californian patient data from the Agency for Healthcare Research and Quality with hospital-level operational and financial data. Using the outcome of primary prelabor cesarean, we estimated patient-level logistic regressions in progressively more detailed models. We assessed incremental fit and discrimination, and aggregated the predicted patient-level event probabilities to construct hospital-level rankings.

RESULTS

Of 408,355 deliveries by women without prior cesareans at 254 hospitals, 11.0% were prelabor cesareans. Including age, ethnicity, race, insurance, weekend and unscheduled admission, and 12 well-known patient risk factors yielded a model c-statistic of 0.83. Further maternal comorbidities, and hospital and obstetric unit characteristics only marginally improved fit. Risk adjusting hospital rankings led to a median absolute change in rank of 44 places compared to rankings based on observed rates. Of the 48 (49) hospitals identified as in the best (worst) quintile on observed rates, only 23 (18) were so identified by the risk-adjusted model.

CONCLUSION

Models predict primary prelabor cesareans with good discrimination. Systematic hospital-level variation in patient risk factors requires risk adjustment to avoid considerably different classification of hospitals by outcome performance. An opportunity exists to define this metric and report such risk-adjusted outcomes to stakeholders.

Composite quality measures for common inpatient medical conditions

BACKGROUND

Public reporting on quality aims to help patients select better hospitals. However, individual quality measures are suboptimal in identifying superior and inferior hospitals based on outcome performance.

OBJECTIVE

To combine structure, process, and outcome measures into an empirically derived composite quality measure for heart failure (HF), acute myocardial infarction (AMI), and pneumonia (PNA). To assess how well the composite measure predicts future high and low performers, and explains variance in future hospital mortality.

RESEARCH DESIGN

Using national Medicare data, we created a cohort of older patients treated at an acute care hospital for HF (n=1,203,595), AMI (n=625,595), or PNA (n=1,234,299). We ranked hospitals on the basis of their July 2005 to June 2008 performance on the composite. We then estimated the odds of future (July to December 2009) 30-day, risk-adjusted mortality at the worst versus best quintile of hospitals. We repeated this analysis using 2005-2008 performance on existing quality indicators, including mortality.

RESULTS

The composite (vs. Hospital Compare) explained 68% (vs. 39%) of variation in future AMI mortality rates. In 2009, if an AMI patient had chosen a hospital in the worst versus best quintile of performance using 2005-2008 composite (vs. Hospital Compare) rankings, he or she would have had 1.61 (vs. 1.39) times the odds of dying in 30 days (P-value for difference <0.001). Results were similar for HF and PNA.

CONCLUSIONS

Composite measures of quality for HF, AMI, and PNA performed better than existing measures at explaining variation in future mortality and predicting future high and low performers.

Profiling provider outcome quality for pay-for-performance in the presence of missing data: a simulation approach

OBJECTIVE

Provider profiling of outcome performance has become increasingly common in pay-for-performance programs. For chronic conditions, a substantial proportion of patients eligible for outcome measures may be lost to follow-up, potentially compromising outcome profiling. In the context of primary care depression treatment, we assess the implications of missing data for the accuracy of alternative approaches to provider outcome profiling.

DATA

We used data from the Improving Mood-Promoting Access to Collaborative Treatment trial and the Depression Improvement across Minnesota, Offering a New Direction initiative to generate parameters for a Monte Carlo simulation experiment.

STUDY DESIGN

The patient outcome of interest is the rate of remission of depressive symptoms at 6 months among a panel of patients with major depression at baseline. We considered two alternative approaches to profiling this outcome: (1) a relative, or tournament style threshold, set at the 80th percentile of remission rate among all providers, and (2) an absolute threshold, evaluating whether providers exceed a specified remission rate (30 percent). We performed a Monte Carlo simulation experiment to evaluate the total error rate (proportion of providers who were incorrectly classified) under each profiling approach. The total error rate was partitioned into error from random sampling variability and error resulting from missing data. We then evaluated the accuracy of alternative profiling approaches under different assumptions about the relationship between missing data and depression remission.

PRINCIPAL FINDINGS

Over a range of scenarios, relative profiling approaches had total error rates that were approximately 20 percent lower than absolute profiling approaches, and error due to missing data was approximately 50 percent lower for relative profiling. Most of the profiling error in the simulations was a result of random sampling variability, not missing data: between 11 and 21 percent of total error was attributable to missing data for relative profiling, while between 16 and 33 percent of total error was attributable to missing data for absolute profiling. Finally, compared with relative profiling, absolute profiling was much more sensitive to missing data that was correlated with the remission outcome.

CONCLUSIONS

Relative profiling approaches for pay-for-performance were more accurate and more robust to missing data than absolute profiling approaches.