Statistical Methods for Profiling Providers of Medical Care: Issues and Applications

Variation in receipt of radiation therapy after breast-conserving surgery: assessing the impact of physicians and geographic regions

BACKGROUND

Among older women with early-stage breast cancer, patients with a short life expectancy (LE) are much less likely to benefit from adjuvant radiation therapy (RT). Little is known about the impact of physicians and regional factors on the use of RT across LE groups.

OBJECTIVE

To determine the relative contribution of patient, physician, and regional factors on the use of RT.

DESIGN

Retrospective cohort.

SUBJECTS

Women aged 67-94 years diagnosed with stage I breast cancer between 1998 and 2007 receiving breast-conserving surgery.

MEASURES

We evaluated patient, physician, and regional factors for their association with RT across strata of LE using a 3-level hierarchical logistic regression model. Risk-standardized treatment rates (RSTRs) for the receipt of radiation were calculated according to primary surgeon and region.

RESULTS

Approximately 43.6% of the 2253 women with a short LE received RT, compared with 90.8% of the 11,027 women with a long LE. Among women with a short LE, the probability of receiving RT varied substantially across primary surgeons; RSTRs ranged from 27.7% to 67.3% (mean, 43.9%). There was less variability across geographic regions; RSTRs ranged from 42.0% to 45.2% (mean, 43.6%). Short LE patients were more likely to receive RT in areas with high radiation oncologist density (odds ratio, 1.59; 95% confidence interval, 1.07-2.36).

CONCLUSIONS

Although there is a wide variation across geographic regions in the use of RT among women with breast cancer and short LE, the regional variation was substantially diminished after accounting for the operating surgeon.

Evaluating the performance of Australian and New Zealand intensive care units in 2009 and 2010

The Australian and New Zealand Intensive Care Society Adult Patient Database (ANZICS APD) is one of the largest databases of its kind in the world and collects individual admissions' data from intensive care units (ICUs) around Australia and New Zealand. Use of this database for monitoring and comparing the performance of ICUs, quantified by the standardised mortality ratio, poses several theoretical and computational challenges, which are addressed in this paper. In particular, the expected number of deaths must be appropriately estimated, the ICU casemix adjustment must be adequate, statistical variation must be fully accounted for, and appropriate adjustment for multiple comparisons must be made. Typically, one or more of these issues have been neglected in ICU comparison studies. Our approach to the analysis proceeds by fitting a random coefficient hierarchical logistic regression model for the inhospital death of each patient, with patients clustered within ICUs. We anticipate the majority of ICUs will be estimated as performing 'usually' after adjusting for important clinical covariates. We take as a starting point the ideas in Ohlssen et al and estimate an appropriate null model that we expect these ICUs to follow, taking a frequentist rather than a Bayesian approach. This methodology allows us to rigorously account for the aforementioned statistical issues and to determine if there are any ICUs contributing to the Australian and New Zealand Intensive Care Society database that have comparatively unusual performance. In addition to investigating the yearly performance of the ICUs, we also estimate changes in individual ICU performance between 2009 and 2010 by adjusting for regression-to-the-mean.

Refining physician quality indicators for screening mammography in older women: distinguishing appropriate use from overuse

OBJECTIVES

To assess the feasibility of refining physician quality indicators of screening mammography use based on patient life expectancy.

DESIGN

Retrospective population-based cohort study.

SETTING

Texas.

PARTICIPANTS

Three thousand five hundred ninety-five usual care providers (UCPs) with at least 10 female patients aged 67 and older on January 1, 2008, with an estimated life expectancy of 7 years or more (222,584 women) and at least 10 women with an estimated life expectancy of less than 7 years (90,903 women), based on age and comorbidity.

MEASUREMENTS

Screening mammography use in 2008-09 by each provider in each population.

RESULTS

The average adjusted mammography screening rates for UCPs were 31.1% for women with a life expectancy of less than 7 years and 55.2% for women with a life expectancy of 7 years or longer. For women with limited life expectancy, 3.7% of UCPs had significantly lower and 9.2% had significantly higher than average adjusted mammography screening rates. For women with longer life expectancy, 16.7% of UCPs had significantly lower and 19.7% had significantly higher than average rates. UCP adjusted screening rates were stable over time (2006-07 vs 2008-09, correlation coefficient (r) = 0.65, P < .001). There was a strong correlation between UCP screening rates for their female patients with a life expectancy of less than 7 years and 7 years or longer (r = 0.67, P < .001). Most physician characteristics associated with higher screening rates (e.g., being female and foreign trained) in women with longer life expectancy were also associated with higher screening rates in women with limited life expectancy.

CONCLUSION

Providers with high mammography screening rates for women with longer life expectancy also tend to screen women with limited life expectancy. Quality indicators for screening practice can be improved by distinguishing appropriate use from overuse based on patient life expectancy.

Shrinkage estimators for a composite measure of quality conceptualized as a formative construct

OBJECTIVE

To demonstrate the value of shrinkage estimators when calculating a composite quality measure as the weighted average of a set of individual quality indicators.

DATA SOURCES

Rates of 28 quality indicators (QIs) calculated from the minimum dataset from residents of 112 Veterans Health Administration nursing homes in fiscal years 2005-2008.

STUDY DESIGN

We compared composite scores calculated from the 28 QIs using both observed rates and shrunken rates derived from a Bayesian multivariate normal-binomial model.

PRINCIPAL FINDINGS

Shrunken-rate composite scores, because they take into account unreliability of estimates from small samples and the correlation among QIs, have more intuitive appeal than observed-rate composite scores. Facilities can be profiled based on more policy-relevant measures than point estimates of composite scores, and interval estimates can be calculated without assuming the QIs are independent. Usually, shrunken-rate composite scores in 1 year are better able to predict the observed total number of QI events or the observed-rate composite scores in the following year than the initial year observed-rate composite scores.

CONCLUSION

Shrinkage estimators can be useful when a composite measure is conceptualized as a formative construct.

Comparison of four methods for deriving hospital standardised mortality ratios from a single hierarchical logistic regression model

There is interest in deriving case-mix adjusted standardised mortality ratios so that comparisons between healthcare providers, such as hospitals, can be undertaken in the controversial belief that variability in standardised mortality ratios reflects quality of care. Typically standardised mortality ratios are derived using a fixed effects logistic regression model, without a hospital term in the model. This fails to account for the hierarchical structure of the data - patients nested within hospitals - and so a hierarchical logistic regression model is more appropriate. However, four methods have been advocated for deriving standardised mortality ratios from a hierarchical logistic regression model, but their agreement is not known and neither do we know which is to be preferred. We found significant differences between the four types of standardised mortality ratios because they reflect a range of underlying conceptual issues. The most subtle issue is the distinction between asking how an average patient fares in different hospitals versus how patients at a given hospital fare at an average hospital. Since the answers to these questions are not the same and since the choice between these two approaches is not obvious, the extent to which profiling hospitals on mortality can be undertaken safely and reliably, without resolving these methodological issues, remains questionable.

Profiling hospitals based on emergency readmission: a multilevel transition modelling approach

Emergency readmission is seen as an important part of the United Kingdom government policy to improve the quality of care that patients receive. In this context, patients and the public have the right to know how well different health organizations are performing. Most methods for profiling estimate the expected numbers of adverse outcomes (e.g. readmission, mortality) for each organization. A number of statistical concerns have been raised, such as the differences in hospital sizes and the unavailability of relevant data for risk adjustment. Having recognized these statistical concerns, a new framework known as the multilevel transition model is developed. Hospital specific propensities of the first, second and further readmissions are considered to be measures of performance, where these measures are used to define a new performance index. During the period 1997 and 2004, the national (English) hospital episodes statistics dataset comprise more than 5 million patient readmissions. Implementing a multilevel model using the complete population dataset could possibly take weeks to estimate the parameters. To resolve the problem, we extract 1000 random samples from the original data, where each random sample is likely to lead to differing hospital performance measures. For computational efficiency a Grid implementation of the model is developed. Analysing the output from the full 1000 sample, we noticed that 4 out of the 5 worst performing hospitals treating cancer patients were in London. These hospitals are known to be the leading NHS Trusts in England, providing diverse range of services to complex patients, and therefore it is inevitable to expect higher numbers of emergency readmissions.

Hospital-level variation in the use of intensive care

OBJECTIVE

To determine the extent to which hospitals vary in the use of intensive care, and the proportion of variation attributable to differences in hospital practice that is independent of known patient and hospital factors.

DATA SOURCE

Hospital discharge data in the State Inpatient Database for Maryland and Washington States in 2006.

STUDY DESIGN

Cross-sectional analysis of 90 short-term, acute care hospitals with critical care capabilities. DATA COLLECTION/METHODS: We quantified the proportion of variation in intensive care use attributable to hospitals using intraclass correlation coefficients derived from mixed-effects logistic regression models after successive adjustment for known patient and hospital factors.

PRINCIPAL FINDINGS

The proportion of hospitalized patients admitted to an intensive care unit (ICU) across hospitals ranged from 3 to 55 percent (median 12 percent; IQR: 9, 17 percent). After adjustment for patient factors, 19.7 percent (95 percent CI: 15.1, 24.4) of total variation in ICU use across hospitals was attributable to hospitals. When observed hospital characteristics were added, the proportion of total variation in intensive care use attributable to unmeasured hospital factors decreased by 26-14.6 percent (95 percent CI: 11, 18.3 percent).

CONCLUSIONS

Wide variability exists in the use of intensive care across hospitals, not attributable to known patient or hospital factors, and may be a target to improve efficiency and quality of critical care.

Variation in the adoption of patient safety practices among New Zealand district health boards

OBJECTIVE

To investigate the adoption and impact of quality improvement measures in New Zealand hospitals.

METHOD

Structured interviews with quality and safety managers of District Health Boards (DHBs). Correlation of use of measures with adjusted 30-day mortality data.

RESULTS

Eighteen of New Zealand's 21 DHBs participated in the survey. Structural or policy measures to improve patient safety, such as credentialing and event reporting procedures, had been introduced into all DHBs, whereas changes to general clinical processes such as medicine reconciliation, falls prevention interventions and disease-specific management guidelines were less consistently used. There was no meaningful correlation between risk-adjusted mortality rates for three common medical conditions and related quality measures.

CONCLUSION

Widespread variation exists among New Zealand DHBs in their adoption of quality and safety practices, especially in relation to clinical processes of care.

Estimating and reporting on the quality of inpatient stroke care by Veterans Health Administration Medical Centers

BACKGROUND

Reporting of quality indicators (QIs) in Veterans Health Administration Medical Centers is complicated by estimation error caused by small numbers of eligible patients per facility. We applied multilevel modeling and empirical Bayes (EB) estimation in addressing this issue in performance reporting of stroke care quality in the Medical Centers.

METHODS AND RESULTS

We studied a retrospective cohort of 3812 veterans admitted to 106 Medical Centers with ischemic stroke during fiscal year 2007. The median number of study patients per facility was 34 (range, 12-105). Inpatient stroke care quality was measured with 13 evidence-based QIs. Eligible patients could either pass or fail each indicator. Multilevel modeling of a patient's pass/fail on individual QIs was used to produce facility-level EB-estimated QI pass rates and confidence intervals. The EB estimation reduced interfacility variation in QI rates. Small facilities and those with exceptionally high or low rates were most affected. We recommended 8 of the 13 QIs for performance reporting: dysphagia screening, National Institutes of Health Stroke Scale documentation, early ambulation, fall risk assessment, pressure ulcer risk assessment, Functional Independence Measure documentation, lipid management, and deep vein thrombosis prophylaxis. These QIs displayed sufficient variation across facilities, had room for improvement, and identified sites with performance that was significantly above or below the population average. The remaining 5 QIs were not recommended because of too few eligible patients or high pass rates with little variation.

CONCLUSIONS

Considerations of statistical uncertainty should inform the choice of QIs and their application to performance reporting.

Hospital-wide mortality as a quality metric: conceptual and methodological challenges

Hospital-wide mortality rates are used as a measure of overall hospital quality. However, their parsimony and apparent simplicity belie significant conceptual and methodological concerns. For many diagnoses included in hospital-wide mortality, the association between short-term mortality and quality of care is not well established. Furthermore, compared with condition-specific or procedure-specific mortality, hospital-wide mortality rates pose greater methodological challenges (ie, eligibility and exclusion criteria, risk adjustment, statistical techniques for aggregating across diagnoses, usability). Many of these result from substantial interprovider heterogeneity in diagnosis frequency, sample sizes, and patient severity. Hospital-wide mortality is problematic as a quality metric for public reporting, although hospitals may elect to use such measures for other purposes. Potential alternative approaches include multidimensional composite metrics or mortality measurement limited to selected conditions and procedures for which the link between hospital mortality and quality is clear, legitimate exclusions are uncommon, and sample sizes, end points, and risk adjustment are adequate.

Correlation of inpatient and outpatient measures of stroke care quality within veterans health administration hospitals

BACKGROUND AND PURPOSE

Quality of care delivered in the inpatient and ambulatory settings may be correlated within an integrated health system such as the Veterans Health Administration. We examined the correlation between stroke care quality at hospital discharge and within 6 months postdischarge.

METHODS

We conducted a cross-sectional hospital-level correlation analyses of chart-abstracted data for 3467 veterans discharged alive after an acute ischemic stroke from 108 Veterans Health Administration medical centers and 2380 veterans with postdischarge follow-up within 6 months in fiscal year 2007. Four risk-standardized processes of care represented discharge care quality: prescription of antithrombotic and antilipidmic therapy, anticoagulation for atrial fibrillation, and tobacco cessation counseling along with a composite measure of defect-free care. Five risk-standardized intermediate outcomes represented postdischarge care quality: achievement of blood pressure, low-density lipoprotein, international normalized ratio, and glycosylated hemoglobin target levels, and delivery of appropriate treatment for poststroke depression along with a composite measure of achieved outcomes.

RESULTS

Median risk-standardized composite rate of defect-free care at discharge was 79%. Median risk-standardized postdischarge rates of achieving goal were 56% for blood pressure, 36% for low-density lipoprotein, 41% for international normalized ratio, 40% for glycosylated hemoglobin, and 39% for depression management and the median risk-standardized composite 6-month outcome rate was 44%. The hospital composite rate of defect-free care at discharge was correlated with meeting the low-density lipoprotein goal (r=0.31; P=0.007) and depression management (r=0.27; P=0.03) goal but was not correlated with blood pressure, international normalized ratio, glycosylated hemoglobin goals, nor with the composite measure of achieved postdischarge outcomes (probability values >0.13).

CONCLUSIONS

Hospital discharge care quality was not consistently correlated with ambulatory care quality.

A methodological approach for register-based evaluation of cost and outcomes in health care

INTRODUCTION

In health care, measures of performance are needed at producer level for improving the treatment processes and at system level for steering purposes. In addition, measures that enable reliable comparisons of producers with respect to each other should encourage them to develop their treatment processes to attain better positioning in benchmarking.

METHODS

The main innovation of the Performance, Effectiveness, and Costs of Treatment episodes (PERFECT) project is to measure performance using existing linkable information available from registers within well-defined care episodes in a whole population. Finnish health care and related registers are used for constructing the disease-specific databases, with rich content on treatment processes and complete follow-up data.

RESULTS

The PERFECT project has developed numerous performance indicators that can be used to evaluate health policy actions as well as to create regional and hospital-level benchmarking data. In PERFECT, the idea is to eliminate individual-level variation from the performance indicators by using individual-level data and proper risk adjustment methods. The focus of our interest is in the variation at the producer or regional level.

CONCLUSIONS

Our experience shows that the utilization of population-level health care registers with an episode-of-care approach enables a continual system and producer-level performance measurement.

An administrative claims model for profiling hospital 30-day mortality rates for pneumonia patients

BACKGROUND

Outcome measures for patients hospitalized with pneumonia may complement process measures in characterizing quality of care. We sought to develop and validate a hierarchical regression model using Medicare claims data that produces hospital-level, risk-standardized 30-day mortality rates useful for public reporting for patients hospitalized with pneumonia.

METHODOLOGY/PRINCIPAL FINDINGS

Retrospective study of fee-for-service Medicare beneficiaries age 66 years and older with a principal discharge diagnosis of pneumonia. Candidate risk-adjustment variables included patient demographics, administrative diagnosis codes from the index hospitalization, and all inpatient and outpatient encounters from the year before admission. The model derivation cohort included 224,608 pneumonia cases admitted to 4,664 hospitals in 2000, and validation cohorts included cases from each of years 1998-2003. We compared model-derived state-level standardized mortality estimates with medical record-derived state-level standardized mortality estimates using data from the Medicare National Pneumonia Project on 50,858 patients hospitalized from 1998-2001. The final model included 31 variables and had an area under the Receiver Operating Characteristic curve of 0.72. In each administrative claims validation cohort, model fit was similar to the derivation cohort. The distribution of standardized mortality rates among hospitals ranged from 13.0% to 23.7%, with 25(th), 50(th), and 75(th) percentiles of 16.5%, 17.4%, and 18.3%, respectively. Comparing model-derived risk-standardized state mortality rates with medical record-derived estimates, the correlation coefficient was 0.86 (Standard Error = 0.032).

CONCLUSIONS/SIGNIFICANCE

An administrative claims-based model for profiling hospitals for pneumonia mortality performs consistently over several years and produces hospital estimates close to those using a medical record model.

Impact of Independent Data Adjudication on Hospital-Specific Estimates of Risk-Adjusted Mortality Following Percutaneous Coronary Interventions in Massachusetts

Background—

As part of state-mandated public reporting of outcomes after percutaneous coronary interventions (PCIs) in Massachusetts, procedural and clinical data were prospectively collected. Variables associated with higher mortality were audited to ensure accuracy of coding. We examined the impact of adjudication on identifying hospitals with possible deficiencies in the quality of PCI care.

Methods and Results—

From October 2005 to September 2006, 15 721 admissions for PCI occurred in 21 hospitals. Of the 864 high-risk variables from 822 patients audited by committee, 201 were changed, with reassignment to lower acuities in 97 (30%) of the 321 shock cases, 24 (43%) of the 56 salvage cases, and 73 (15%) of the 478 emergent cases. Logistic regression models were used to predict patient-specific in-hospital mortality. Of 241 (1.5%) patients who died after PCI, 30 (12.4%) had a lower predicted mortality with adjudicated than with unadjudicated data. Model accuracy was excellent with either adjudicated or unadjudicated data. Hospital-specific risk-standardized mortality rates were estimated using both adjudicated and unadjudicated data through hierarchical logistic regression. Although adjudication reduced between-hospital variation by one third, risk-standardized mortality rates were similar using unadjudicated and adjudicated data. None of the hospitals were identified as statistical outliers. However, cross-validated posterior-predicted P values calculated with adjudicated data increased the number of borderline hospital outliers compared with unadjudicated data.

Conclusions—

Independent adjudication of site-reported high-risk features may increase the ability to identify hospitals with higher risk-adjusted mortality after PCI despite having little impact on the accuracy of risk prediction for the entire population.

Development and validation of a disease-specific risk adjustment system using automated clinical data

OBJECTIVE

To develop and validate a disease-specific automated inpatient mortality risk adjustment system primarily using computerized numerical laboratory data and supplementing them with administrative data. To assess the values of additional manually abstracted data.

METHODS

Using 1,271,663 discharges in 2000-2001, we derived 39 disease-specific automated clinical models with demographics, laboratory findings on admission, ICD-9 principal diagnosis subgroups, and secondary diagnosis-based chronic conditions. We then added manually abstracted clinical data to the automated clinical models (manual clinical models). We compared model discrimination, calibration, and relative contribution of each group of variables. We validated these 39 models using 1,178,561 discharges in 2004-2005.

RESULTS

The overall mortality was 4.6 percent (n = 58,300) and 4.0 percent (n = 47,279) for derivation and validation cohorts, respectively. Common mortality predictors included age, albumin, blood urea nitrogen or creatinine, arterial pH, white blood counts, glucose, sodium, hemoglobin, and metastatic cancer. The average c-statistic for the automated clinical models was 0.83. Adding manually abstracted variables increased the average c-statistic to 0.85 with better calibration. Laboratory results displayed the highest relative contribution in predicting mortality.

CONCLUSIONS

A small number of numerical laboratory results and administrative data provided excellent risk adjustment for inpatient mortality for a wide range of clinical conditions.

Ranking hospitals on surgical mortality: the importance of reliability adjustment

OBJECTIVE

We examined the implications of reliability adjustment on hospital mortality with surgery.

DATA SOURCE

We used national Medicare data (2003-2006) for three surgical procedures: coronary artery bypass grafting (CABG), abdominal aortic aneurysm (AAA) repair, and pancreatic resection.

STUDY DESIGN

We conducted an observational study to evaluate the impact of reliability adjustment on hospital mortality rankings. Using hierarchical modeling, we adjusted hospital mortality for reliability using empirical Bayes techniques. We assessed the implication of this adjustment on the apparent variation across hospitals and the ability of historical hospital mortality rates (2003-2004) to forecast future mortality (2005-2006).

PRINCIPAL FINDINGS

The net effect of reliability adjustment was to greatly diminish apparent variation for all three operations. Reliability adjustment was also particularly important for identifying hospitals with the lowest future mortality. Without reliability adjustment, hospitals in the "best" quintile (2003-2004) with pancreatic resection had a mortality of 7.6 percent in 2005-2006; with reliability adjustment, the "best" hospital quintile had a mortality of 2.7 percent in 2005-2006. For AAA repair, reliability adjustment also improved the ability to identify hospitals with lower future mortality. For CABG, the benefits of reliability adjustment were limited to the lowest volume hospitals.

CONCLUSION

Reliability adjustment results in more stable estimates of mortality that better forecast future performance. This statistical technique is crucial for helping patients select the best hospitals for specific procedures, particularly uncommon ones, and should be used for public reporting of hospital mortality.

Variation in hospital mortality rates for patients with acute myocardial infarction

Hospitals vary by twofold in their hospital-specific 30-day risk-stratified mortality rates (RSMRs) for Medicare beneficiaries with acute myocardial infarction (AMI). However, we lack a comprehensive investigation of hospital characteristics associated with 30-day RSMRs and the degree to which the variation in 30-day RSMRs is accounted for by these characteristics, including the socioeconomic status (SES) profile of hospital patient populations. We conducted a cross-sectional national study of hospitals with ≥15 AMI discharges from July 1, 2005 to June 20, 2008. We estimated a multivariable weighted regression using Medicare claims data for hospital-specific 30-day RSMRs, American Hospital Association Survey of Hospitals for hospital characteristics, and the United States Census data reported by Neilsen Claritas, Inc., for zip-code level estimates of SES status. Analysis included 2,908 hospitals with 513,202 AMI discharges. Mean hospital 30-day RSMR was 16.5% (SD 1.7 percentage points). Our multivariable model explained 17.1% of the variation in hospital-specific 30-day RSMRs. Teaching status, number of hospital beds, AMI volume, cardiac facilities available, urban/rural location, geographic region, ownership type, and SES profile of patients were significantly (p < 0.05) associated with 30-day RSMRs. In conclusion, substantial variation in hospital outcomes for patients with AMI remains unexplained by measurements of hospital characteristics including SES patient profile.

The Hospital Compare mortality model and the volume-outcome relationship

OBJECTIVE

We ask whether Medicare's Hospital Compare random effects model correctly assesses acute myocardial infarction (AMI) hospital mortality rates when there is a volume-outcome relationship.

DATA SOURCES/STUDY SETTING

Medicare claims on 208,157 AMI patients admitted in 3,629 acute care hospitals throughout the United States.

STUDY DESIGN

We compared average-adjusted mortality using logistic regression with average adjusted mortality based on the Hospital Compare random effects model. We then fit random effects models with the same patient variables as in Medicare's Hospital Compare mortality model but also included terms for hospital Medicare AMI volume and another model that additionally included other hospital characteristics.

PRINCIPAL FINDINGS

Hospital Compare's average adjusted mortality significantly underestimates average observed death rates in small volume hospitals. Placing hospital volume in the Hospital Compare model significantly improved predictions.

CONCLUSIONS

The Hospital Compare random effects model underestimates the typically poorer performance of low-volume hospitals. Placing hospital volume in the Hospital Compare model, and possibly other important hospital characteristics, appears indicated when using a random effects model to predict outcomes. Care must be taken to insure the proper method of reporting such models, especially if hospital characteristics are included in the random effects model.

Modelling multivariate disease rates with a latent structure mixture model

There has been considerable recent interest in multivariate modelling of the geographical distribution of morbidity or mortality rates for potentially related diseases. The motivations for this include investigation of similarities or dissimilarities in the risk distribution for the different diseases, as well as ‘borrowing strength’ across disease rates to shrink the uncertainty in geographical risk assessment for any particular disease. A number of approaches to such multivariate modelling have been suggested and this paper proposes an extension to these which may provide a richer range of dependency structures than those encompassed so far. We develop a model which incorporates a discrete mixture of latent structures and argue that this provides potential to represent an enhanced range of correlation structures between diseases at the same time as implicitly allowing for less restrictive spatial correlation structures between geographical units. We compare and contrast our approach to other commonly used multivariate disease models and demonstrate comparative results using data taken from cancer registries on four carcinomas in some 300 geographical units in England, Scotland and Wales.

Modeling the volume-effectiveness relationship in the case of hip fracture treatment in Finland

Background

A common argument in the recent health policy debate is that treatment is more effective among care providers with large volumes. It is challenging, however, to examine the volume-effectiveness relationship empirically. Several suggestions have recently been made for methodological improvements in the examination of the volume-effectiveness relationship. The aim of this study is to develop an extended methodology for examining the volume-effectiveness relationship and demonstrate it for the case of hip fracture treatment.

Methods

Data consisting of 22,857 hip fracture patients from 52 hospitals in Finland in 1998-2001 were extracted from the administrative registers. The relationship between hospital and rehabilitation unit volumes and effectiveness was examined using a statistical model that allowed risk adjustments and hierarchical modeling of volume trends, developed for the purposes of this study. Four-month mortality and the alternative register-based measure of maintainability were used as effectiveness indicators.

Results

No clear relationship was found between hospital volume and the effectiveness of hip fracture treatment, but a novel result showing an association between the rehabilitation unit volume and effectiveness was detected. The face validity of the maintainability indicator seemed to be acceptable.

Conclusions

The methodological ideas presented allow for improved examination of the volume-effectiveness relationship. There are no indications that patients with hip fractures should only be treated in high-volume hospitals, though it may be beneficial to centralize the rehabilitation of hip fracture patients to specialized units.

How well do hospital mortality rates reported in the New York State CABG report card predict subsequent hospital performance?

BACKGROUND

The use of mortality report cards as the basis for hospital choice assumes that a hospital's current performance is predicted by its past performance.

OBJECTIVE

To assess the accuracy of hospital risk-adjusted mortality rates reported in the New York State (NYS) coronary artery bypass graft (CABG) report card for predicting subsequent hospital mortality.

METHODS

We performed a retrospective study based on hospital mortality measures for CABG surgery (n = 37 hospitals) in NYS, which are publicly reported by the NYS Department of Health. Feasible generalized least squares was used to examine the association between a hospital's past quality ranking (high-quality, intermediate-quality, low-quality) and its subsequent performance, as measured using the ratio of the observed-to-expected mortality rate (O-to-E ratio).

RESULTS

Hospitals identified as low-mortality hospitals using 2-year-old data had subsequent O-to-E ratios that were 16.8% lower (95% confidence interval, 8.9-24.8; P < 0.001) than average-mortality hospitals, whereas hospitals identified as high-mortality hospitals had subsequent O-to-E ratios that were 31.8% higher (95% confidence interval, 3.69-59.9; P < 0.05) compared with average-mortality hospitals. Hospitals identified as high-mortality hospitals using 3-year-old data were indistinguishable from average-mortality hospitals.

CONCLUSION

Hospital ranking based on 2-year-old data is a strong predictor of future performance. Report cards based on 3-year-old data may not be useful for identifying low-performance hospitals. We recommend that the CABG report cards in NYS should be based on 2-year-old data, as opposed to the current practice of basing them on either 2- or 3-year-old data.

Outlier detection for a hierarchical Bayes model in a study of hospital variation in surgical procedures

One of the most important aspects of profiling healthcare providers or services is constructing a model that is flexible enough to allow for random variation. At the same time, we wish to identify those institutions that clearly deviate from the usual standard of care. Here, we propose a hierarchical Bayes model to study the choice of surgical procedure for rectal cancer using data previously analysed by Simons et al.(1) Using hospitals as random effects, we construct a computationally simple graphical method for determining hospitals that are outliers; that is, they differ significantly from other hospitals of the same type in terms of surgical choice.

US emergency department performance on wait time and length of visit

STUDY OBJECTIVE

Prolonged emergency department (ED) wait time and length of visit reduce quality of care and increase adverse events. Previous studies have not examined hospital-level performance on ED wait time and visit length in the United States. The purpose of this study is to describe hospital-level performance on ED wait time and visit length.

METHODS

We conducted a retrospective cross-sectional study of a stratified random sampling of 35,849 patient visits to 364 nonfederal US hospital EDs in 2006, weighted to represent 119,191,528 visits to 4,654 EDs. Measures included EDs' median wait times and visit lengths, EDs' median proportion of patients treated by a physician within the time recommended at triage, and EDs' median proportion of patients dispositioned within 4 or 6 hours.

RESULTS

In the median ED, 78% (interquartile range [IQR], 63% to 90%) of all patients and 67% (IQR, 52% to 82%) of patients who were triaged to be treated within 1 hour were treated by a physician within the target triage time. A total of 31% of EDs achieved the triage target for more than 90% of their patients; 14% of EDs achieved the triage target for 90% or more of patients triaged to be treated within an hour. In the median ED, 76% (IQR 54% to 94%) of patients were admitted within 6 hours. A total of 48% of EDs admitted more than 90% of their patients within 6 hours, but only 25% of EDs admitted more than 90% of their patients within 4 hours.

CONCLUSION

A minority of hospitals consistently achieved recommended wait times for all ED patients, and fewer than half of hospitals consistently admitted their ED patients within 6 hours.

Using a hierarchical model to estimate risk-adjusted mortality for hospitals not included in the reference sample

OBJECTIVE

To provide a method for any hospital to evaluate patient mortality using a hierarchical risk-adjustment equation derived from a reference sample.

DATA SOURCE

American College of Surgeons National Trauma Data Bank (NTDB).

STUDY DESIGN

Hierarchical logistic regression models predicting mortality were estimated from NTDB data. Risk-adjusted hospital effects obtained directly from models using standard software were compared with approximations derived from a summary equation and data from each individual hospital.

PRINCIPAL FINDINGS

Theoretical approximations were similar to results using standard software.

CONCLUSIONS

To allow independent verification, agencies using reference databases for hospital mortality "report cards" should publish their risk-adjustment equations. Similar hospitals not in the reference database may also use the published equations along with the approximations described to evaluate their own outcomes using their own data.

Measuring quality for public reporting of health provider quality: making it meaningful to patients

Public quality reports of hospitals, health plans, and physicians are being used to promote efficiency and quality in the health care system. Shrinkage estimators have been proposed as superior measures of quality to be used in these reports because they offer more conservative and stable quality ranking of providers than traditional, nonshrinkage estimators. Adopting the perspective of a patient faced with choosing a local provider on the basis of publicly provided information, we examine the advantages and disadvantages of shrinkage and nonshrinkage estimators and contrast the information made available by them. We demonstrate that 2 properties of shrinkage estimators make them less useful than nonshrinkage estimators for patients making choices in their area of residence.

Understanding the factors associated with differences in caesarean section rates at hospital level: the case of Latin America

As in many other regions of the world, caesarean section (CS) rates in Latin America are increasing. Studies elsewhere have shown that providing feedback to caregivers regarding their own performance relative to their peers can significantly reduce the rates. Our objectives are to calculate risk-adjusted CS rates for hospitals in Latin America and to identify factors associated with differences among risk-adjusted rates. We included 120 randomly selected institutions in eight countries of Latin America, representing 97 095 pregnancies. We used random-effects models to calculate a risk-adjusted rate for each hospital and to identify hospitals significantly higher or lower than a benchmark rate. We conducted a regression analysis to identify characteristics of hospitals associated with differences among risk-adjusted rates. The overall CS rate was 35%, ranging from 0% to 85%. Risk-adjusted CS rates ranged from 11% to 78%. Three-quarters of hospitals had risk-adjusted rates significantly above the previously identified benchmark of 20%. Characteristics of institutions explained 48% of the variability among risk-adjusted rates, including being a private as opposed to a public institution, having some economic incentive for CS as opposed to no incentive, and having > or = 50 maternity beds. Strategies to halt further increases in CS rates and reduce rates to levels that reflect the best quality of care, are urgently needed worldwide. The involvement of local quality control departments is an essential component in achieving success. Our results can be used to identify institutions that can be targets for further interventions to reduce CS rates.

Are (the log-odds of) hospital mortality rates normally distributed? Implications for studying variations in outcomes of medical care

RATIONALE

Hierarchical regression models are increasingly being used to examine variations in outcomes following the provision of medical care across providers. These models frequently assume a normal distribution for the provider-specific random effects. The appropriateness of this assumption for examining variations in health care outcomes has never been explicitly tested.

AIMS AND OBJECTIVES

To compare hierarchical logistic regression models in which the provider-specific random effects were either a normal distribution or a mixture of three normal distributions.

METHODS

We used data on 18,825 patients admitted to 109 hospitals in Ontario with a diagnosis of acute myocardial infarction. We used the Deviance Information Criterion, Bayes factors and predictive distributions to compare the evidence between the two competing models.

RESULTS

There was strong evidence that the distribution of hospital-specific log-odds of mortality was a mixture of three normal distributions compared to the evidence that it was normal. In some scenarios, the hospital-specific posterior tail probabilities of unacceptably high mortality were lower when a logistic-normal model was fit compared to when a logistic-mixture of normal distributions model was fit. Additionally, in these same scenarios, fewer hospitals were classified as having higher than acceptable mortality when the logistic-mixture of three normal distributions was used.

CONCLUSIONS

These findings have important consequences for those who use hierarchical models to examine variations in outcomes of medical care across providers since the mixture of three normal distributions model indicated that variations in outcomes across providers was greater than indicated by the logistic-normal model.

Composite measures for hospital quality using quality-adjusted life years

Developing clinically meaningful summary measures of health-care quality is key to inferring quality of care. Current summary measures use a number of different approaches to weight their individual measures but rarely use weights based on clinical 'importance'. Such an approach would help to focus quality improvement efforts on areas likely to have the largest impact on health outcomes. Using coronary artery bypass graft (CABG) surgery as a case study, we weight and combine 11 process, complication, and survival measures to summarize differences in quality-adjusted life expectancy 1 year following surgery for a sample of hospitals. We use a fully Bayesian analysis to estimate 1-year survival outcomes using a hierarchical exponential survival model. We then estimate the expected utility of the year following surgery for each patient using complication probabilities fitted from hierarchical models and utility values from the literature. We estimate quality-adjusted life years (QALYs) for each hospital as the utility-weighted average 1-year survival probability and then estimate 'incremental QALYs' by taking the difference in QALYs for each hospital relative to a comparison group that reflects the average performance of all hospitals in the state. We illustrate our framework by estimating incremental QALYs for 14 hospitals performing CABG surgery in Massachusetts in 2003 and find that a composite measure based on QALYs can change the classification of quality outliers relative to conventional mortality measures.

Empirically derived composite measures of surgical performance

BACKGROUND

Individual quality measures have significant limitations for assessing surgical performance. Despite growing interest in composite measures, empirically-based methods for combining multiple domains of surgical quality are not well established.

OBJECTIVE

To develop and validate a composite measure of surgical performance that best describes variation in hospital mortality rates and forecasts future performance.

RESEARCH DESIGN

Using the national Medicare claims database, we identified all patients undergoing aortic valve replacement in 2000 to 2001 (n = 53,120). To serve as input variables, we identified hospital-level predictors of mortality with aortic valve replacement, including hospital volume, complication rates, and mortality with other procedures. Hospital-specific predicted mortality rates were then determined using Bayesian-derived modeling techniques and assessed against subsequent hospital mortality (2002-2003).

RESULTS

Our composite measure explained 78% of the variation in aortic valve replacement mortality rates (2000-2001). The most important input variables were hospital volume, mortality with aortic valve replacement, and mortality for other high-risk cardiac procedures. The composite measure forecasted 70% of future hospital-level variation in mortality rates (2002-2003), and was substantially better in this regard than individual measures. Hospitals scoring in the bottom quintile on the composite measure in 2000 to 2001 had 2-fold higher mortality rates in 2002 to 2003 than hospitals in the top quintile (adjusted odds ratio, 1.97; 95% CI, 1.73-2.23).

CONCLUSIONS

Compared with individual surgical quality indicators, empirically derived composite measures are superior in explaining variation in hospital mortality rates and in forecasting future performance. Such measures could be useful for public reporting, value-based purchasing, or benchmarking for quality improvement purposes.

Analyzing center specific outcomes in hematopoietic cell transplantation

Reporting transplant center-specific survival rates after hematopoietic cell transplantation is required in the United States. We describe a method to report 1-year survival outcomes by center, as well as to quantify center performance relative to the transplant center network average, which can be reliably used with censored data and for small center sizes. Each center's observed 1-year survival outcome is compared to a predicted survival outcome adjusted for patient characteristics using a pseudovalue regression technique. A 95% prediction interval for 1-year survival assuming no center effect is computed for each center by bootstrapping the scaled residuals from the regression model, and the observed 1-year survival is compared to this prediction interval to determine center performance. We illustrate the technique using a recent center specific analysis performed by the Center for International Blood and Marrow Transplant Research, and study the performance of this method using simulation.

Improving quality indicator report cards through Bayesian modeling

Background

The National Database for Nursing Quality Indicators^® (NDNQI^®) was established in 1998 to assist hospitals in monitoring indicators of nursing quality (eg, falls and pressure ulcers). Hospitals participating in NDNQI transmit data from nursing units to an NDNQI data repository. Data are summarized and published in reports that allow participating facilities to compare the results for their units with those from other units across the nation. A disadvantage of this reporting scheme is that the sampling variability is not explicit. For example, suppose a small nursing unit that has 2 out of 10 (rate of 20%) patients with pressure ulcers. Should the nursing unit immediately undertake a quality improvement plan because of the rate difference from the national average (7%)?

Methods

In this paper, we propose approximating 95% credible intervals (CrIs) for unit-level data using statistical models that account for the variability in unit rates for report cards.

Results

Bayesian CrIs communicate the level of uncertainty of estimates more clearly to decision makers than other significance tests.

Conclusion

A benefit of this approach is that nursing units would be better able to distinguish problematic or beneficial trends from fluctuations likely due to chance.

Mixed-effects Poisson regression analysis of adverse event reports: the relationship between antidepressants and suicide

A new statistical methodology is developed for the analysis of spontaneous adverse event (AE) reports from post-marketing drug surveillance data. The method involves both empirical Bayes (EB) and fully Bayes estimation of rate multipliers for each drug within a class of drugs, for a particular AE, based on a mixed-effects Poisson regression model. Both parametric and semiparametric models for the random-effect distribution are examined. The method is applied to data from Food and Drug Administration (FDA)'s Adverse Event Reporting System (AERS) on the relationship between antidepressants and suicide. We obtain point estimates and 95 per cent confidence (posterior) intervals for the rate multiplier for each drug (e.g. antidepressants), which can be used to determine whether a particular drug has an increased risk of association with a particular AE (e.g. suicide). Confidence (posterior) intervals that do not include 1.0 provide evidence for either significant protective or harmful associations of the drug and the adverse effect. We also examine EB, parametric Bayes, and semiparametric Bayes estimators of the rate multipliers and associated confidence (posterior) intervals. Results of our analysis of the FDA AERS data revealed that newer antidepressants are associated with lower rates of suicide adverse event reports compared with older antidepressants. We recommend improvements to the existing AERS system, which are likely to improve its public health value as an early warning system.

Ranking USRDS provider specific SMRs from 1998-2001

Provider profiling (ranking/percentiling) is prevalent in health services research. Bayesian models coupled with optimizing a loss function provide an effective framework for computing non-standard inferences such as ranks. Inferences depend on the posterior distribution and should be guided by inferential goals. However, even optimal methods might not lead to definitive results and ranks should be accompanied by valid uncertainty assessments. We outline the Bayesian approach and use estimated Standardized Mortality Ratios (SMRs) in 1998-2001 from the United States Renal Data System (USRDS) as a platform to identify issues and demonstrate approaches. Our analyses extend Liu et al. (2004) by computing estimates developed by Lin et al. (2006) that minimize errors in classifying providers above or below a percentile cut-point, by combining evidence over multiple years via a first-order, autoregressive model on log(SMR), and by use of a nonparametric prior. Results show that ranks/percentiles based on maximum likelihood estimates of the SMRs and those based on testing whether an SMR = 1 substantially under-perform the optimal estimates. Combining evidence over the four years using the autoregressive model reduces uncertainty, improving performance over percentiles based on only one year. Furthermore, percentiles based on posterior probabilities of exceeding a properly chosen SMR threshold are essentially identical to those produced by minimizing classification loss. Uncertainty measures effectively calibrate performance, showing that considerable uncertainty remains even when using optimal methods. Findings highlight the importance of using loss function guided percentiles and the necessity of accompanying estimates with uncertainty assessments.

Benchmarking physical therapy clinic performance: statistical methods to enhance internal validity when using observational data

Many clinics and payers are beginning programs to collect and interpret outcomes related to quality of care and provider performance (ie, benchmarking).

OUTCOMES

assessment is commonly done using observational research designs, which makes it important for those involved in these endeavors to appreciate the underlying challenges and limitations of these designs. This perspective article discusses the advantages and limitations of using observational research to evaluate quality of care and provider performance in order to inform clinicians, researchers, administrators, and policy makers who want to use data to guide practice and policy or critically appraise observational studies and benchmarking efforts. Threats to internal validity, including potential confounding, patient selection bias, and missing data, are discussed along with statistical methods commonly used to address these limitations. An example is given from a recent study comparing physical therapy clinic performance in terms of patient outcomes and service utilization with and without the use of these methods. The authors demonstrate that crude differences in clinic outcomes and service utilization tend to be inflated compared with the differences that are statistically adjusted for selected threats to internal validity. The authors conclude that quality of care measurement and ranking procedures that do not use similar methods may produce findings that may be misleading.

Estimating a composite measure of hospital quality from the Hospital Compare database: differences when using a Bayesian hierarchical latent variable model versus denominator-based weights

BACKGROUND

A single composite measure calculated from individual quality indicators (QIs) is a useful measure of hospital performance and can be justified conceptually even when the indicators are not highly correlated with one another.

OBJECTIVE

To compare 2 basic approaches for calculating a composite measure: an extension of the most widely-used approach, which weights individual indicators based on the number of people eligible for the indicator (referred to as denominator-based weights, DBWs), and a Bayesian hierarchical latent variable model (BLVM).

METHODS

Using data for 15 QIs from 3275 hospitals in the Hospital Compare database, we calculated hospital ranks using several versions of DBWs and 2 BLVMs. Estimates in 1 BLVM were driven by differences in variances of the QIs (BLVM1) and estimates in the other by differences in the signal-to-noise ratios of the QIs (BLVM2).

RESULTS

There was a high correlation in ranks among all of the DBW approaches and between those approaches and BLVM1. However, a high correlation does not necessarily mean that the same hospitals were ranked in the top or bottom quality deciles. In general, large hospitals were ranked in higher quality deciles by all of the approaches, though the effect was most apparent using BLVM2.

CONCLUSIONS

Both conceptually and practically, hospital-specific DBWs are a reasonable approach for calculating a composite measure. However, this approach fails to take into account differences in the reliability of estimates from hospitals of different sizes, a big advantage of the Bayesian models.

Statistical methodology for classifying units on the basis of multiple-related measures

Both the private and public sectors have begun giving financial incentives to healthcare providers, such as hospitals, delivering superior 'quality of care'. Quality of care is assessed through a set of disease-specific measures that characterize the performance of healthcare providers. These measures are then combined into a unidimensional composite score. Most of the programs that reward superior performance use raw averages of the measures as the composite score. The scores based on raw averages fail to take into account typical characteristics of data used for performance evaluation, such as within-patient and within-hospital correlations, variable number of measures available in different hospitals, and missing data. In this paper, we contrast two different versions of composites based on raw average scores with a model-based score constructed using a latent variable model. We also present two methods to identify hospitals with superior performance. The methods are illustrated using national data collected to evaluate quality of care delivered by the U.S. acute care hospitals.