Judging hospitals by severity-adjusted mortality rates: the influence of the severity-adjustment method

Risk-adjusting acute myocardial infarction mortality: are APR-DRGs the right tool?

OBJECTIVE

To determine if a widely used proprietary risk-adjustment system, APR-DRGs, misadjusts for severity of illness and misclassifies provider performance.

DATA SOURCES

(1) Discharge abstracts for 116,174 noninstitutionalized adults with acute myocardial infarction (AMI) admitted to nonfederal California hospitals in 1991-1993; (2) inpatient medical records for a stratified probability sample of 974 patients with AMIs admitted to 30 California hospitals between July 31, 1990 and May 31, 1991.

STUDY DESIGN

Using the 1991-1993 data set, we evaluated the predictive performance of APR-DRGs Version 12. Using the 1990/1991 validation sample, we assessed the effect of assigning APR-DRGs based on different sources of ICD-9-CM data.

DATA COLLECTION/EXTRACTION METHODS

Trained, blinded coders reabstracted all ICD-9-CM diagnoses and procedures, and established the timing of each diagnosis. APR-DRG Risk of Mortality and Severity of Illness classes were assigned based on (1) all hospital-reported diagnoses, (2) all reabstracted diagnoses, and (3) reabstracted diagnoses present at admission. The outcome variables were 30-day mortality in the 1991-1993 data set and 30-day inpatient mortality in the 1990/1991 validation sample.

PRINCIPAL FINDINGS

The APR-DRG Risk of Mortality class was a strong predictor of death (c = .831-.847), but was further enhanced by adding age and sex. Reabstracting diagnoses improved the apparent performance of APR-DRGs (c = .93 versus c = .87), while using only the diagnoses present at admission decreased apparent performance (c = .74). Reabstracting diagnoses had less effect on hospitals' expected mortality rates (r = .83-.85) than using diagnoses present at admission instead of all reabstracted diagnoses (r = .72-.77). There was fair agreement in classifying hospital performance based on these three sets of diagnostic data (K = 0.35-0.38).

CONCLUSIONS

The APR-DRG Risk of Mortality system is a powerful risk-adjustment tool, largely because it includes all relevant diagnoses, regardless of timing. Although some late diagnoses may not be preventable, APR-DRGs appear suitable only if one assumes that none is preventable.

Risk-adjusted in-hospital death rates for peer hospitals in rural and urban regions

The purpose of this research project was to compare inpatient mortality rates for rural hospitals with mortality rates of urban hospitals of given sizes and ranges of service. Statistical adjustments for risk were made in the probability of death during hospitalization for 43,000 patients across 166 hospitals by age, gender, principal diagnosis, principal surgical procedure, characteristics of the secondary diagnoses, and whether or not cancer was a secondary diagnosis. Eighty-three small hospitals that had a relatively unspecialized range of services constituted the study group. Patient characteristics of this study group were moderately representative of the national population. A standardized score was calculated for each hospital using a formula based on the actual hospital death rate and the death rate expected for a given hospital with patients of the same demographic and medical characteristics. Patients admitted to hospitals in nonmetropolitan areas had a mortality rate of 0.41 percent compared with a mortality rate of 0.66 percent in peer hospitals in metropolitan areas. After mortality rates were risk-adjusted and converted to z scores, nonmetropolitan areas had an average z of +0.16, and metropolitan areas had an average z of -0.25, where positive z scores reflect a lower-than-average adjusted mortality rate. The metropolitan-nonmetropolitan (urban-rural) difference was not statistically significant, but it is meaningful in that rural hospitals tended to have a lower adjusted mortality rate than urban hospitals of the same size and type, indicating that rural hospitals had the same or lower adjusted mortality rates. The possibility of urban hospitals having riskier patients was minimized but could not be definitively ruled out. Taken together with other studies, the data are consistent with the view that small rural hospitals generally make appropriate transfer decisions for severely ill patients and provide quality care for retained patients.

Performance of the '100 top hospitals': what does the report card report?

We examine whether Medicare patients with acute myocardial infarction (AMI) admitted to one of HCIA-Mercer's "100 top hospitals" received better care or had better outcomes than patients treated in other hospitals. Among four hospital peer groups, the top 100 hospitals had similar thirty-day mortality and use of aspirin, beta-blockers, and reperfusion compared with their peers, but lower lengths-of-stay and in-hospital costs, with similar or lower readmission rates. Our findings suggest that the 100 Top Hospitals study may be better suited for identifying hospitals with higher performance on financial and operating measures than superior clinical performance in treating elderly AMI patients. However, there was no evidence that quality was sacrificed for increased financial efficiency among the top 100 hospitals.

Comparing AMI mortality among hospitals in patients 65 years of age and older: evaluating methods of risk adjustment

BACKGROUND

Interest in the reporting of risk-adjusted outcomes for patients with acute myocardial infarction is growing. A useful risk-adjustment model must balance parsimony and ease of data collection with predictive ability.

METHODS AND RESULTS

From our analysis of 82 359 patients >/=65 years of age admitted with acute myocardial infarction to 2401 hospitals, we derived a parsimonious model that predicts 30-day mortality. The model was validated on a similar group of 78 699 patients from 2386 hospitals. Of the 73 candidate predictor variables examined, 7 variables describing patient characteristics on arrival were selected for inclusion in the final model: age, cardiac arrest, anterior or lateral location of myocardial infarction, systolic blood pressure, white blood cell count, serum creatinine, and congestive heart failure. The area under the receiver-operating characteristic curve for the final model was 0.77 in the derivation cohort and 0.77 in the validation cohort. The rankings of hospitals by performance (in deciles) with this model were most similar to a comprehensive 27-variable model based on medical chart review and least similar to models based on administrative billing codes.

CONCLUSIONS

A simple 7-variable risk model performs as well as more complex models in comparing hospital outcomes for acute myocardial infarction. Although there is a continuing need to improve methods of risk adjustment, our results provide a basis for hospitals to develop a simple approach to compare outcomes.

Explaining differences in English hospital death rates using routinely collected data

OBJECTIVES

To ascertain hospital inpatient mortality in England and to determine which factors best explain variation in standardised hospital death ratios.

DESIGN

Weighted linear regression analysis of routinely collected data over four years, with hospital standardised mortality ratios as the dependent variable.

SETTING

England.

SUBJECTS

Eight million discharges from NHS hospitals when the primary diagnosis was one of the diagnoses accounting for 80% of inpatient deaths.

MAIN OUTCOME MEASURES

Hospital standardised mortality ratios and predictors of variations in these ratios.

RESULTS

The four year crude death rates varied across hospitals from 3.4% to 13.6% (average for England 8.5%), and standardised hospital mortality ratios ranged from 53 to 137 (average for England 100). The percentage of cases that were emergency admissions (60% of total hospital admissions) was the best predictor of this variation in mortality, with the ratio of hospital doctors to beds and general practitioners to head of population the next best predictors. When analyses were restricted to emergency admissions (which covered 93% of all patient deaths analysed) number of doctors per bed was the best predictor.

CONCLUSION

Analysis of hospital episode statistics reveals wide variation in standardised hospital mortality ratios in England. The percentage of total admissions classified as emergencies is the most powerful predictor of variation in mortality. The ratios of doctors to head of population served, both in hospital and in general practice, seem to be critical determinants of standardised hospital death rates; the higher these ratios, the lower the death rates in both cases.

The importance of surgeon volume and training in outcomes for vascular surgical procedures

PURPOSE

Mortality and morbidity rates after vascular surgical procedures have been related to hospital volume. Hospitals in which greater volumes of vascular surgical procedures are performed tend to have statistically lower mortality rates than those hospitals in which fewer procedures are performed. Only a few studies have directly assessed the impact of the surgeon's volume on outcome. Therefore, the purpose of this study was to review a large state data set to determine the impact of surgeon volume on outcome after carotid endarterectomy (CEA), lower extremity bypass grafting (LEAB), and abdominal aortic aneurysm repair (AAA).

METHODS

The Florida Agency for Health Care Administration state admission data from 1992 to 1996 were obtained. The data included all nonfederal hospital admissions. Frequencies were calculated from first-listed International Classification of Diseases-9 codes. Multiple logistic regression was used to test the significance on outcome of surgeon volume, American Board of Surgery certification for added qualifications in general vascular surgery, hospital size, hospital volume, patient age, and gender.

RESULTS

During this interval, there were 31,172 LEABs, 45,744 CEAs, and 13,415 AAAs performed. The in-hospital mortality rate increased with age. A doubling of surgeon volume was associated with a 4% reduction in risk for adverse outcome for CEA (P =.006), an 8% reduction for LEAB, and an 11% reduction for AAA ( P =.0002). However, although hospital volume was significant in predicting better outcomes for CEA and AAA procedures, it was not associated with better outcomes for LEAB. Certification for added qualifications in general vascular surgery was a significant predictor of better outcomes for CEA and AAA. Certified vascular surgeons had a 15% lower risk rate of death or complications after CEA (P =.002) and a 24% lower risk rate of a similar outcome after AAA (P =.009). However, for LEAB, certification was not significant.

CONCLUSION

Surgeon volume and certification are significantly related to better patient outcomes for patients who undergo CEA and AAA. In addition, surgeons with high volumes demonstrated consistently lower mortality and morbidity rates than did surgeons with low volumes. Hospital volume for a given procedure also is correlated with better outcomes.

Critical care transport: outcome evaluation after interfacility transfer and hospitalization

STUDY OBJECTIVE

To test the hypothesis that interfacility transfer is not associated with increased mortality, duration of stay, or readmission within 7 days.

METHODS

We matched 3,298 patients who were hospitalized for chest pain or related complaints in Kaiser Permanente medical centers after transfer from the emergency department of a nonplan hospital (transported patients) with 3,298 patients of the same gender and age (+/-5 years) and with the same principal diagnosis who were hospitalized within 6 months without transfer in the same Kaiser Permanente medical center (directly admitted patients). Patients were compared in terms of outcome measures: in-hospital deaths, continued care in another facility, readmission within 7 days, in-patient length of stay (LOS), and LOS in special care units.

RESULTS

The adjusted odds ratios for in-hospital mortality and readmission within 7 days were 1.0 (95% confidence interval,.8 to 1.4) and.9 (95% confidence interval,.7 to 1.2), respectively. The adjusted mean difference in LOS was -.1 days (95% confidence interval, -.2 to.1). Transported and directly admitted cardiac patients were also compared for all examined outcome measures at each of 10 medical centers. At a few medical centers, we observed significant difference in LOS, special care LOS, and continued care in another facility. However, all these differences were small, and most were probably random errors.

CONCLUSION

Conservative patient selection criteria, pretransfer stabilization, and the use of appropriate equipment and medical personnel have resulted in the interfacility transfer program's achieving its goal of transferring high-risk patients without adverse impact on clinical outcomes or resource use.

A new casemix adjustment index for hospital mortality among patients with congestive heart failure

OBJECTIVES

Comparative analysis of hospital outcomes requires reliable adjustment for casemix. Although congestive heart failure is one of the most common indications for hospitalization, congestive heart failure casemix adjustment has not been widely studied. The purposes of this study were (1) to describe and validate a new congestive heart failure-specific casemix adjustment index to predict in-hospital mortality and (2) to compare its performance to the Charlson comorbidity index.

METHODS

Data from all 4,608 admissions to the Massachusetts General Hospital from January 1990 to July 1996 with a principal ICD-9-CM discharge diagnosis of congestive heart failure were evaluated. Massachusetts General Hospital patients were randomly divided in a derivation and a validation set. By logistic regression, odds ratios for in-hospital death were computed and weights were assigned to construct a new predictive index in the derivation set. The performance of the index was tested in an internal Massachusetts General Hospital validation set and in a non-Massachusetts General Hospital external validation set incorporating data from all 1995 New York state hospital discharges with a primary discharge diagnosis of congestive heart failure.

RESULTS

Overall in-hospital mortality was 6.4%. Based on the new index, patients were assigned to six categories with incrementally increasing hospital mortality rates ranging from 0.5% to 31%. By logistic regression, "c" statistics of the congestive heart failure-specific index (0.83 and 0.78, derivation and validation set) were significantly superior to the Charlson index (0.66). Similar incrementally increasing hospital mortality rates were observed in the New York database with the congestive heart failure-specific index ("c" statistics 0.75).

CONCLUSION

In an administrative database, this congestive heart failure-specific index may be a more adequate casemix adjustment tool to predict hospital mortality in patients hospitalized for congestive heart failure.

Using outcomes to make inferences about nursing home quality

Recent concerns about containing the growth of public expenditures on nursing home care and the development of prospective and casemix reimbursement systems with incentives for cost containment have increased the importance of monitoring quality in nursing homes. The current view is that quality assurance systems should include more outcome measures to improve quality. This article discusses why it is difficult to develop facility-level outcome measures that can be used to evaluate and compare the quality of care of nursing homes. The article places the current interest in outcomes measures in its historical policy context and reviews important conceptual and methodological issues associated with outcome-based quality assessment. The authors discuss the difficulty in isolating the facility effect when studying nursing home outcomes and implications of using different estimation approaches. In conclusion, they discuss the need to integrate research with outcome-based quality assurance systems to allow ongoing evaluation and quality improvement.

Commentary: inaccurate data on the quality of care may do more harm than good--an alternative approach is required

Recently the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) announced that it would integrate the use of clinical outcomes and other performance measures into the accreditation process through its new "ORYX" program. This JCAHO initiative represents a significant new development that will include more than 100 different performance measurement systems, most of which are available through commercial firms and outside organizations. However, we see some potential problems with this new initiative. This is because some indicators recommended by JCAHO may be questionable due to the fact they are based on flawed methodologies that could result in biased and confounded data. To illustrate some of the potential adverse effects that could result from using such data to compare health care providers and facilities, we discuss some common problems associated with several widely available performance measurement systems. We then suggest an alternative approach that could potentially avoid many of these problems in the future.

Statistical and study design issues in assessing the quality and outcomes of care in rheumatic diseases

As we have pointed out in this article, a health care study should have a well-defined intent that is matched to the study type, population of interest, and outcome. Care must be taken to collect samples in a meaningful way, so that results can be generalized to larger populations. Outcomes must be selected carefully to ensure that they will be sensitive to the types of care being considered, and only a few main outcomes should be selected so as to preserve the level of statistical significance of the research results. Sample sizes should be sufficient to detect an effect of reasonable size, after accounting for attrition in longitudinal studies and rates of occurrence with dichotomous outcomes. If the study purpose is to compare multiple institutions or health care providers, statistical adjustments for case mix will generally be required. Outcome research is an expanding area of development for rheumatology care and for medical care in general. It offers the promise of the use of administrative data bases to answer questions that are important both to arthritis researchers and to consumers of rheumatology care. As with all areas of clinical research, we must maintain appropriate levels of statistical rigor to protect the integrity of the results. Inadequate attention to the design and analysis of data can compromise research results before a study even gets started, and health care research studies have as many potential statistical pitfalls as other types of clinical research.

Studying outcomes of organizational change in health services

OBJECTIVES

The rapidly changing organizational context within which health care is delivered is altering provider-patient relations and processes of clinical decision-making, with significant implications for patient outcomes. Yet definitive research on such effects is lacking. The authors seek to underscore the contribution of organizational research to studies of clinical outcomes and demonstrates several approaches to further such efforts.

METHODS

The authors present a theoretical framework of the operant mechanisms linking organizational attributes and patient outcomes. They use case examples from their ongoing research on hospitals to illustrate strategies for measuring these mechanisms and for overcoming some of the feasibility issues inherent in organizational research.

RESULTS

Several methodological issues are explored: (1) exploiting "targets of opportunity" and "natural experiments" is a promising strategy for studying patient outcomes related to organizational reform; (2) indices of organizational traits, constructed from individual survey responses, can illuminate the operant mechanisms by which structure affects outcomes; and (3) secondary data sources and innovative statistical matching procedures provide a feasible strategy for constructing study comparison groups. Extending the organizational outcomes research strategy to new areas of inquiry offers an opportunity to enhance our understanding of how nursing organization affects outcomes.

CONCLUSIONS

Improving the effectiveness of medical care in a health-care system undergoing fundamental restructuring requires greater understanding of how organizational context affects clinical outcomes. A higher priority should be placed on organizational outcomes research by researchers and funding agencies.

Differences in procedure use, in-hospital mortality, and illness severity by gender for acute myocardial infarction patients: are answers affected by data source and severity measure?

OBJECTIVES

According to some studies, women with heart disease receive fewer procedures and have higher in-hospital death rates than men. These studies vary by data source (hospital discharge abstract versus detailed clinical information) and severity measurement methods. The authors examined whether evaluations of gender differences for acute myocardial infarction patients vary by data source and severity measure.

METHODS

The authors considered 10 severity measures: four using clinical medical record data and six using discharge abstracts (diagnosis and procedure codes). The authors studied all 14,083 patients admitted in 1991 for acute myocardial infarction to 100 hospitals nationwide, examining in-hospital death and use of coronary angiography, coronary artery bypass graft surgery (CABG), and percutaneous transluminal coronary angioplasty (PTCA). Logistic regression was used to calculate odds ratios for death and procedure use for women compared with men, controlling for age and each of the severity scores.

RESULTS

After adjusting only for age, women were significantly more likely than men to die and less likely to receive CABG and coronary angiography. Severity measures provided different assessments of whether women were sicker than men; for all cases, clinical data-based MedisGroups rated women's severity compared with men's, whereas four code-based severity measures viewed women as sicker. After adjusting for severity and age, women were significantly more likely than men to die in-hospital and less likely to receive coronary angiography and CABG; women and men had relatively equal adjusted odds ratios of receiving PTCA. Odds ratios reflecting gender differences in procedure use and death rates were similar across severity measures.

CONCLUSIONS

Comparisons of severity-adjusted in-hospital death rates and invasive procedure use between men and women yielded similar findings regardless of data source and severity measure.