The Hospital Compare mortality model and the volume-outcome relationship

Fixed effects modelling for provider mortality outcomes: Analysis of the Australia and New Zealand Intensive Care Society (ANZICS) Adult Patient Data-base

BACKGROUND

Risk adjusted mortality for intensive care units (ICU) is usually estimated via logistic regression. Random effects (RE) or hierarchical models have been advocated to estimate provider risk-adjusted mortality on the basis that standard estimators increase false outlier classification. The utility of fixed effects (FE) estimators (separate ICU-specific intercepts) has not been fully explored.

METHODS

Using a cohort from the Australian and New Zealand Intensive Care Society Adult Patient Database, 2009-2010, the model fit of different logistic estimators (FE, random-intercept and random-coefficient) was characterised: Bayesian Information Criterion (BIC; lower values better), receiver-operator characteristic curve area (AUC) and Hosmer-Lemeshow (H-L) statistic. ICU standardised hospital mortality ratios (SMR) and 95%CI were compared between models. ICU site performance (FE), relative to the grand observation-weighted mean (GO-WM) on odds ratio (OR), risk ratio (RR) and probability scales were assessed using model-based average marginal effects (AME).

RESULTS

The data set consisted of 145355 patients in 128 ICUs, years 2009 (47.5%) & 2010 (52.5%), with mean(SD) age 60.9(18.8) years, 56% male and ICU and hospital mortalities of 7.0% and 10.9% respectively. The FE model had a BIC = 64058, AUC = 0.90 and an H-L statistic P-value = 0.22. The best-fitting random-intercept model had a BIC = 64457, AUC = 0.90 and H-L statistic P-value = 0.32 and random-coefficient model, BIC = 64556, AUC = 0.90 and H-L statistic P-value = 0.28. Across ICUs and over years no outliers (SMR 95% CI excluding null-value = 1) were identified and no model difference in SMR spread or 95%CI span was demonstrated. Using AME (OR and RR scale), ICU site-specific estimates diverged from the GO-WM, and the effect spread decreased over calendar years. On the probability scale, a majority of ICUs demonstrated calendar year decrease, but in the for-profit sector, this trend was reversed.

CONCLUSIONS

The FE estimator had model advantage compared with conventional RE models. Using AME, between and over-year ICU site-effects were easily characterised.

HIV quality report cards: impact of case-mix adjustment and statistical methods

BACKGROUND

There will be increasing pressure to publicly report and rank the performance of healthcare systems on human immunodeficiency virus (HIV) quality measures. To inform discussion of public reporting, we evaluated the influence of case-mix adjustment when ranking individual care systems on the viral control quality measure.

METHODS

We used data from the Veterans Health Administration (VHA) HIV Clinical Case Registry and administrative databases to estimate case-mix adjusted viral control for 91 local systems caring for 12 368 patients. We compared results using 2 adjustment methods, the observed-to-expected estimator and the risk-standardized ratio.

RESULTS

Overall, 10 913 patients (88.2%) achieved viral control (viral load ≤400 copies/mL). Prior to case-mix adjustment, system-level viral control ranged from 51% to 100%. Seventeen (19%) systems were labeled as low outliers (performance significantly below the overall mean) and 11 (12%) as high outliers. Adjustment for case mix (patient demographics, comorbidity, CD4 nadir, time on therapy, and income from VHA administrative databases) reduced the number of low outliers by approximately one-third, but results differed by method. The adjustment model had moderate discrimination (c statistic = 0.66), suggesting potential for unadjusted risk when using administrative data to measure case mix.

CONCLUSIONS

Case-mix adjustment affects rankings of care systems on the viral control quality measure. Given the sensitivity of rankings to selection of case-mix adjustment methods-and potential for unadjusted risk when using variables limited to current administrative databases-the HIV care community should explore optimal methods for case-mix adjustment before moving forward with public reporting.

A comparison of clinical outcomes from carotid artery stenting among US hospitals

BACKGROUND

The Centers for Medicare and Medicaid Services require hospitals performing carotid artery stenting (CAS) to recertify the quality of their programs every 2 years, but currently this involves no explicit comparisons of postprocedure mortality across hospitals. Hence, the current recertification process may fail to identify hospitals that are performing poorly in relation to peer institutions. Our objective was to compare risk-standardized procedural outcomes across US hospitals that performed CAS and to identify hospitals with statistically high postprocedure mortality rates.

METHODS AND RESULTS

We conducted a retrospective cohort study of Medicare beneficiaries who underwent CAS from July 2009 to June 2011 at 927 US hospitals. Thirty-day risk-standardized mortality rates were calculated using the Hospital Compare statistical method, a well-validated hierarchical generalized linear model that included both patient-level and hospital-level predictors. Claims were examined from 22 708 patients undergoing CAS, with a crude 30-day mortality rate of 2.0%. Risk-standardized 30-day mortality rates after CAS varied from 1.1% to 5.1% (P<0.001 for the difference). Thirteen hospitals had risk-standardized mortality rates that were statistically (P<0.05) higher than the national mean. Conversely, 5 hospitals had risk-standardized mortality rates that were statistically (P<0.05) lower than the national mean.

CONCLUSIONS

We used administrative claims to identify several CAS hospitals with excessively high 30-day mortality after carotid stenting. When combined with information currently used by Medicare for CAS recertification, such as clinical registry data and program reports, clinical outcomes comparisons could enhance Medicare's ability to identify hospitals that are questionable candidates for recertification.

Measurement and risk adjustment of prelabor cesarean rates in a large sample of California hospitals

OBJECTIVE

Prelabor cesareans in women without a prior cesarean is an important quality measure, yet one that is seldom tracked. We estimated patient-level risks and calculated how sensitive hospital rankings on this proposed quality metric were to risk adjustment.

STUDY DESIGN

This retrospective cohort study linked Californian patient data from the Agency for Healthcare Research and Quality with hospital-level operational and financial data. Using the outcome of primary prelabor cesarean, we estimated patient-level logistic regressions in progressively more detailed models. We assessed incremental fit and discrimination, and aggregated the predicted patient-level event probabilities to construct hospital-level rankings.

RESULTS

Of 408,355 deliveries by women without prior cesareans at 254 hospitals, 11.0% were prelabor cesareans. Including age, ethnicity, race, insurance, weekend and unscheduled admission, and 12 well-known patient risk factors yielded a model c-statistic of 0.83. Further maternal comorbidities, and hospital and obstetric unit characteristics only marginally improved fit. Risk adjusting hospital rankings led to a median absolute change in rank of 44 places compared to rankings based on observed rates. Of the 48 (49) hospitals identified as in the best (worst) quintile on observed rates, only 23 (18) were so identified by the risk-adjusted model.

CONCLUSION

Models predict primary prelabor cesareans with good discrimination. Systematic hospital-level variation in patient risk factors requires risk adjustment to avoid considerably different classification of hospitals by outcome performance. An opportunity exists to define this metric and report such risk-adjusted outcomes to stakeholders.

Composite measures for profiling hospitals on bariatric surgery performance

IMPORTANCE

The optimal approach for profiling hospital performance with bariatric surgery is unclear.

OBJECTIVE

To develop a novel composite measure for profiling hospital performance with bariatric surgery.

DESIGN, SETTING, AND PARTICIPANTS

Using clinical registry data from the Michigan Bariatric Surgery Collaborative, we studied all patients undergoing bariatric surgery from January 1, 2008, through December 31, 2010. For laparoscopic gastric bypass surgery, we used empirical Bayes techniques to create a composite measure by combining several measures, including serious complications, reoperations, and readmissions; hospital and surgeon volume; and outcomes with other related procedures. Hospitals were ranked for 2008 through 2009 and placed in 1 of 3 groups: 3-star (top 20%), 2-star (middle 60%), and 1-star (bottom 20%). We assessed how well these ratings predicted outcomes in the next year (2010) compared with other widely used measures.

MAIN OUTCOMES AND MEASURES

Risk-adjusted serious complications.

RESULTS

Composite measures explained a larger proportion of hospital-level variation in serious complication rates with laparoscopic gastric bypass than other measures. For example, the composite measure explained 89% of the variation compared with only 28% for risk-adjusted complication rates alone. Composite measures also appeared better at predicting future performance compared with individual measures. When ranked on the composite measure, 1-star hospitals had 2-fold higher serious complication rates (4.6% vs 2.4%; odds ratio, 2.0; 95% CI, 1.1-3.5) compared with 3-star hospitals. Differences in serious complication rates between 1- and 3-star hospitals were much smaller when hospitals were ranked using serious complications (4.0% vs 2.7%; odds ratio, 1.6; 95% CI, 0.8-2.9) and hospital volume (3.3% vs 3.2%; odds ratio, 0.85; 95% CI, 0.4-1.7).

CONCLUSIONS AND RELEVANCE

Composite measures are much better at explaining hospital-level variation in serious complications and predicting future performance than other approaches. In this preliminary study, it appears that such composite measures may be better than existing alternatives for profiling hospital performance with bariatric surgery.

Limits of readmission rates in measuring hospital quality suggest the need for added metrics

Recent national policies use risk-standardized readmission rates to measure hospital performance on the theory that readmissions reflect dimensions of the quality of patient care that are influenced by hospitals. In this article our objective was to assess readmission rates as a hospital quality measure. First we compared quartile rankings of hospitals based on readmission rates in 2009 and 2011 to see whether hospitals maintained their relative performance or whether shifts occurred that suggested either changes in quality or random variation. Next we examined the relationship between readmission rates and several commonly used hospital quality indicators, including risk-standardized mortality rates, volume, teaching status, and process-measure performance. We found that quartile rankings fluctuated and that readmission rates for lower-performing hospitals in 2009 tended to improve by 2011, while readmission rates for higher-performing hospitals tended to worsen. Regression to the mean (a form of statistical noise) accounted for a portion of the changes in hospital performance. We also found that readmission rates were higher in teaching hospitals and were weakly correlated with the other indicators of hospital quality. Policy makers should consider augmenting the use of readmission rates with other measures of hospital performance during care transitions and should build on current efforts that take a communitywide approach to the readmissions issue.

Statistical benchmarks for health care provider performance assessment: a comparison of standard approaches to a hierarchical Bayesian histogram-based method

OBJECTIVE

Examine how widely used statistical benchmarks of health care provider performance compare with histogram-based statistical benchmarks obtained via hierarchical Bayesian modeling.

DATA SOURCES

Publicly available data from 3,240 hospitals during April 2009-March 2010 on two process-of-care measures reported on the Medicare Hospital Compare website.

STUDY DESIGN

Secondary data analyses of two process-of-care measures comparing statistical benchmark estimates and threshold exceedance determinations under various combinations of hospital performance measure estimates and benchmarking approaches.

PRINCIPAL FINDINGS

Statistical benchmarking approaches for determining top 10 percent performance varied with respect to which hospitals exceeded the performance benchmark; such differences were not found at the 50 percent threshold. Benchmarks derived from the histogram of provider performance under hierarchical Bayesian modeling provide a compromise between benchmarks based on direct (raw) estimates, which are overdispersed relative to the true distribution of provider performance and prone to high variance for small providers, and posterior mean provider performance, for which over-shrinkage and under-dispersion relative to the true provider performance distribution is a concern.

CONCLUSIONS

Given the rewards and penalties associated with characterizing top performance, the ability of statistical benchmarks to summarize key features of the provider performance distribution should be examined.

Composite quality measures for common inpatient medical conditions

BACKGROUND

Public reporting on quality aims to help patients select better hospitals. However, individual quality measures are suboptimal in identifying superior and inferior hospitals based on outcome performance.

OBJECTIVE

To combine structure, process, and outcome measures into an empirically derived composite quality measure for heart failure (HF), acute myocardial infarction (AMI), and pneumonia (PNA). To assess how well the composite measure predicts future high and low performers, and explains variance in future hospital mortality.

RESEARCH DESIGN

Using national Medicare data, we created a cohort of older patients treated at an acute care hospital for HF (n=1,203,595), AMI (n=625,595), or PNA (n=1,234,299). We ranked hospitals on the basis of their July 2005 to June 2008 performance on the composite. We then estimated the odds of future (July to December 2009) 30-day, risk-adjusted mortality at the worst versus best quintile of hospitals. We repeated this analysis using 2005-2008 performance on existing quality indicators, including mortality.

RESULTS

The composite (vs. Hospital Compare) explained 68% (vs. 39%) of variation in future AMI mortality rates. In 2009, if an AMI patient had chosen a hospital in the worst versus best quintile of performance using 2005-2008 composite (vs. Hospital Compare) rankings, he or she would have had 1.61 (vs. 1.39) times the odds of dying in 30 days (P-value for difference <0.001). Results were similar for HF and PNA.

CONCLUSIONS

Composite measures of quality for HF, AMI, and PNA performed better than existing measures at explaining variation in future mortality and predicting future high and low performers.

Based on key measures, care quality for Medicare enrollees at safety-net and non-safety-net hospitals was almost equal

Safety-net hospitals, which include urban hospitals serving large numbers of low-income, uninsured, and otherwise vulnerable populations, have historically faced greater financial strains than hospitals that serve more affluent populations. These strains can affect hospitals' quality of care, perhaps resulting in worse outcomes that are commonly used as indicators of care quality-mortality and readmission rates. We compared risk-standardized rates of both of these clinical outcomes among fee-for-service Medicare beneficiaries admitted for acute myocardial infarction, heart failure, or pneumonia. These beneficiaries were admitted to urban hospitals within Metropolitan Statistical Areas that contained at least one safety-net and at least one non-safety-net hospital. We found that outcomes varied across the urban areas for both safety-net and non-safety-net hospitals for all three conditions. However, mortality and readmission rates were broadly similar, with non-safety-net hospitals outperforming safety-net hospitals on average by less than one percentage point across most conditions. For heart failure mortality, there was no difference between safety-net and non-safety-net hospitals. These findings suggest that safety-net hospitals are performing better than many would have expected.

Medicare's public reporting initiative on hospital quality had modest or no impact on mortality from three key conditions

Hospital Compare, Medicare's public reporting initiative, began reporting measures of hospital quality for almost all US acute care hospitals in 2005. The impact of this public reporting initiative on patient mortality is unknown. We used Medicare claims data from the period 2000-08 to estimate the effect of Hospital Compare on thirty-day mortality for heart attack, heart failure, and pneumonia. Our analysis indicates that the fact that hospitals had to report quality data under Hospital Compare led to no reductions in mortality beyond existing trends for heart attack and pneumonia and led to a modest reduction in mortality for heart failure. We conclude that Medicare's public reporting initiative for hospitals has had a minimal impact on patient mortality.

Hospital and surgeon variation in complications and repeat surgery following incident lumbar fusion for common degenerative diagnoses

OBJECTIVE

To identify factors that account for variation in complication rates across hospitals and surgeons performing lumbar spinal fusion surgery.

DATA SOURCES

Discharge registry including all nonfederal hospitals in Washington State from 2004 to 2007.

STUDY DESIGN

We identified adults (n = 6,091) undergoing an initial inpatient lumbar fusion for degenerative conditions. We identified whether each patient had a subsequent complication within 90 days. Logistic regression models with hospital and surgeon random effects were used to examine complications, controlling for patient characteristics and comorbidity.

PRINCIPAL FINDINGS

Complications within 90 days of a fusion occurred in 4.8 percent of patients, and 2.2 percent had a reoperation. Hospital effects accounted for 8.8 percent of the total variability, and surgeon effects account for 14.4 percent. Surgeon factors account for 54.5 percent of the variation in hospital reoperation rates, and 47.2 percent of the variation in hospital complication rates. The discretionary use of operative features, such as the inclusion of bone morphogenetic proteins, accounted for 30 and 50 percent of the variation in surgeons' reoperation and complication rates, respectively.

CONCLUSIONS

To improve the safety of lumbar spinal fusion surgery, quality improvement efforts that focus on surgeons' discretionary use of operative techniques may be more effective than those that target hospitals.

Composite measures for rating hospital quality with major surgery

OBJECTIVE

To assess the value of a novel composite measure for identifying the best hospitals for major procedures.

DATA SOURCE

We used national Medicare data for patients undergoing five high-risk surgical procedures between 2005 and 2008.

STUDY DESIGN

For each procedure, we used empirical Bayes techniques to create a composite measure combining hospital volume, risk-adjusted mortality with the procedure of interest, risk-adjusted mortality with other related procedures, and other variables. Hospitals were ranked based on 2005-2006 data and placed in one of three groups: 1-star (bottom 20 percent), 2-star (middle 60 percent), and 3-star (top 20 percent). We assessed how well these ratings forecasted risk-adjusted mortality rates in the next 2 years (2007-2008), compared to other measures.

PRINCIPAL FINDINGS

For all five procedures, the composite measures based on 2005-2006 data performed well in predicting future hospital performance. Compared to 1-star hospitals, risk-adjusted mortality was much lower at 3-star hospitals for esophagectomy (6.7 versus 14.4 percent), pancreatectomy (4.7 versus 9.2 percent), coronary artery bypass surgery (2.6 versus 5.0 percent), aortic valve replacement (4.5 versus 8.5 percent), and percutaneous coronary interventions (2.4 versus 4.1 percent). Compared to individual surgical quality measures, the composite measures were better at forecasting future risk-adjusted mortality. These measures also outperformed the Center for Medicare and Medicaid Services (CMS) Hospital Compare ratings.

CONCLUSION

Composite measures of surgical quality are very effective at predicting hospital mortality rates with major procedures. Such measures would be more informative than existing quality indicators in helping patients and payers identify high-quality hospitals with specific procedures.

What is the best way to estimate hospital quality outcomes? A simulation approach

OBJECTIVE

To test the accuracy of alternative estimators of hospital mortality quality using a Monte Carlo simulation experiment.

DATA SOURCES

Data are simulated to create an admission-level analytic dataset. The simulated data are validated by comparing distributional parameters (e.g., mean and standard deviation of 30-day mortality rate, hospital sample size) with the same parameters observed in Medicare data for acute myocardial infarction (AMI) inpatient admissions.

STUDY DESIGN

We perform a Monte Carlo simulation experiment in which true quality is known to test the accuracy of the Observed-over-Expected estimator, the Risk Standardized Mortality Rate (RSMR), the Dimick and Staiger (DS) estimator, the Hierarchical Poisson estimator, and the Moving Average estimator using hospital 30-day mortality for AMI as the outcome. Estimator accuracy is evaluated for all hospitals and for small, medium, and large hospitals.

DATA EXTRACTION METHODS

Data are simulated.

PRINCIPAL FINDINGS

Significant and substantial variation is observed in the accuracy of the tested outcome estimators. The DS estimator is the most accurate for all hospitals and for small hospitals using both accuracy criteria (root mean squared error and proportion of hospitals correctly classified into quintiles).

CONCLUSIONS

The mortality estimator currently in use by Medicare for public quality reporting, the RSMR, has been shown to be less accurate than the DS estimator, although the magnitude of the difference is not large. Pending testing and validation of our findings using current hospital data, CMS should reconsider the decision to publicly report mortality rates using the RSMR.

Medication administration quality and health information technology: a national study of US hospitals

OBJECTIVE

To determine whether the use of computerized physician order entry (CPOE) and electronic medication administration records (eMAR) is associated with better quality of medication administration at medium-to-large acute-care hospitals. DATA/STUDY SETTING: A retrospective cross-sectional analysis of data from three sources: CPOE/eMAR usage from HIMSS Analytics (2010), medication quality scores from CMS Hospital Compare (2010), and hospital characteristics from CMS Acute Inpatient Prospective Payment System (2009). The analysis focused on 11 quality indicators (January-December 2009) at 2603 medium-to-large (≥ 100 beds), non-federal acute-care hospitals measuring proportion of eligible patients given (or prescribed) recommended medications for conditions, including acute myocardial infarction, heart failure, and pneumonia, and surgical care improvement. Using technology adoption by 2008 as reference, hospitals were coded: (1) eMAR-only adopters (n=986); (2) CPOE-only adopters (n=115); and (3) adopters of both technologies (n=804); with non-adopters of both technologies as reference group (n=698). Hospitals were also coded for duration of use in 2-year increments since technology adoption. Hospital characteristics, historical measure-specific patient volume, and propensity scores for technology adoption were used to control for confounding factors. The analysis was performed using a generalized linear model (logit link and binomial family).

PRINCIPAL FINDINGS

Relative to non-adopters of both eMAR and CPOE, the odds of adherence to all measures (except one) were higher by 14-29% for eMAR-only hospitals and by 13-38% for hospitals with both technologies, translating to a marginal increase of 0.4-2.0 percentage points. Further, each additional 2 years of technology use was associated with 6-15% higher odds of compliance on all medication measures for eMAR-only hospitals and users of both technologies.

CONCLUSIONS

Implementation and duration of use of health information technologies are associated with improved adherence to medication guidelines at US hospitals. The benefits are evident for adoption of eMAR systems alone and in combination with CPOE.

Predictors of cesarean delivery for periviable neonates

OBJECTIVE

To test for racial or ethnic disparities or both in periviable cesarean delivery and describe sociodemographic and clinical characteristics associated with periviable cesarean delivery.

METHODS

This was a retrospective cohort study of state-level maternal and neonatal hospital discharge data linked to vital statistics data for deliveries occurring between 23 0/7 and 24 6/7 [corrected] weeks of gestation in California, Missouri, and Pennsylvania from 1995 to 2005 (N=8,290).

RESULTS

Approximately 79% of the population was aged 18-35 years, and almost half were nulliparous. Almost 20% of the women were African American, 36.4% were Hispanic, and 33.6% were white. Overall, 33.6% of periviable neonates were delivered by cesarean. In multivariable analyses adjusting for sociodemographic and clinical characteristics, cesarean delivery did not differ among African American and Hispanic women compared with white women (odds ratio [OR] 0.89, 95% confidence interval [CI] 0.76-1.05; and OR 0.95, 95% CI 0.83-1.09, respectively). Women presenting with preterm labor were significantly less likely to undergo cesarean delivery (OR 0.84, 95% CI 0.73-0.96), whereas women presenting with preterm premature rupture of membranes (OR 1.29, 95% CI 1.14-1.45) or abruption (OR 2.43, 95% CI 2.09-2.81) were more likely to have cesarean deliveries. The strongest predictor of periviable cesarean delivery was pregnancy-induced hypertension (OR 15.6.4, 95% CI 12.3-19.7).

CONCLUSION

Unlike disparities observed at later gestational ages, cesarean delivery did not differ by race and ethnicity among this periviable cohort. Instead, medical indications such as pregnancy-induced hypertension, preterm premature rupture of membranes, or abruption were associated with a higher likelihood of cesarean delivery. Periviable deliveries represent a subset of deliveries, wherein race and ethnicity do not influence mode of delivery; the acuity of the clinical encounter dictates the course of care.