Report cards on cardiac surgeons. Assessing New York State's approach

Performance of the '100 top hospitals': what does the report card report?

We examine whether Medicare patients with acute myocardial infarction (AMI) admitted to one of HCIA-Mercer's "100 top hospitals" received better care or had better outcomes than patients treated in other hospitals. Among four hospital peer groups, the top 100 hospitals had similar thirty-day mortality and use of aspirin, beta-blockers, and reperfusion compared with their peers, but lower lengths-of-stay and in-hospital costs, with similar or lower readmission rates. Our findings suggest that the 100 Top Hospitals study may be better suited for identifying hospitals with higher performance on financial and operating measures than superior clinical performance in treating elderly AMI patients. However, there was no evidence that quality was sacrificed for increased financial efficiency among the top 100 hospitals.

Ready-made, recalibrated, or Remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery

BACKGROUND

Risk indexes for operative mortality after cardiac surgery are used for comparative profiling of surgeons or centers. We examined whether clinicians and managers should use an existing index without modification, recalibrate it for their populations, or derive a new model altogether.

METHODS AND RESULTS

Drawing on 7491 consecutive patients who underwent isolated CABG at 2 Toronto teaching hospitals between 1993 and 1996, we compared 3 strategies: (1) using a ready-made model originally derived and validated in our jurisdiction; (2) recalibrating the ready-made model to better fit the population; and (3) deriving a new model with additional risk factors. We assessed statistical accuracy, ie, area under a receiver-operator characteristic curve (ROC); precision, ie, statistical goodness-of-fit; and actual impact on both risk-adjusted operative mortalities (RAOM) and performance rankings for 14 surgeons. The new model was slightly more accurate than the ready-made model (ROC, 0.78 versus 0.76; P<0.05), albeit not different from the recalibrated model (ROC, 0.77). The ready-made model showed poor fit between the predicted and observed results (P<0.001), leading to significant underestimation of RAOM (1.6+/-0. 2%) compared with the other strategies (2.5+/-0.2%; P=0.048). Remodeling also changed the performance rankings among half the surgeons with higher RAOM.

CONCLUSIONS

Poorly calibrated risk algorithms can bias the calculation of RAOM and alter the results of surgeon-specific profiles. Any existing index used for risk assessment in cardiac surgery should be episodically recalibrated or compared with new models derived from local subjects to ensure that its performance remains optimal.

Comparing physician-specific two-year patient outcomes after coronary angiography: methodologic issues and results

OBJECTIVES

We sought to evaluate methodologies to compare physician-related long-term patient outcomes appropriately.

BACKGROUND

Evaluation of physicians on the basis of short-term patient outcome is becoming widely practiced. These analyses fail to consider the importance of long-term outcome, and methods appropriate to such an analysis are poorly defined.

METHODS

All patients undergoing coronary angiography between 1992 and 1994 who received all of their cardiac care at our institution were followed for 27+/-13 months (mean+/-SD). Patients (n = 754) were cared for by one or more of 17 staff physicians. Risk-adjusted models were developed for four candidate clinical end points and cost. Physicians were then evaluated for each outcome measure.

RESULTS

Of the clinical end points, death could be modeled most accurately (c-statistic = 0.83). The c-statistics for other end points ranged from 0.63 to 0.70. Physicians with outcomes statistically different (p < 0.05) from other physicians were identified more commonly than would be expected from the play of chance (p = 0.005). However, improvement in the c-statistics by the addition of physician identifiers was very modest. Physician's evaluations by the four measures of clinical outcome were variably correlated (r = .00 to .85). Graphic display of clinical and cost results for each physician did identify certain physicians who might be judged to provide more cost-effective care than others.

CONCLUSIONS

Although comparisons of groups of physicians on the basis of long-term patient outcomes may have merit, individual physician-to-physician comparisons will be more difficult, owing to 1) multiple physicians contributing care to individual patients; 2) the poor predictive capacity of models other than that for survival; and 3) the modest apparent impact of differences in physician providers on long-term patient outcome. With these caveats in mind, modeling to compare patient outcomes of individual physicians with homogeneous patient populations or to identify gross outliers (good or bad) may be practicable in some patient-care systems, but may be inappropriate in others.

Grading the graders: how hospitals in California and New York perceive and interpret their report cards

BACKGROUND

Concerns about quality of care are increasing as hospitals struggle to lower costs. Hospital report cards are controversial, but little is known about their impact.

OBJECTIVES

To determine whether recent hospital report cards are viewed more favorably than pioneering federal efforts; whether a report based on clinical data is viewed more favorably than one based on administrative data; and whether attitudes toward report cards are related to hospital characteristics.

DESIGN

Mailed survey of chief executives at 374 California hospitals and 31 New York hospitals listed in report cards on myocardial infarction and coronary bypass mortality.

SUBJECTS

Two-hundred-and-seventy-four hospitals in California (73.3% response) and 27 in New York (87.1% response). California hospitals were categorized on ownership, size, occupancy, risk-adjusted mortality, teaching status, patient volume, and surgical capability.

MEASURES

Number of hospital units that received or discussed the report card, ratings of its quality, perceptions of its usefulness, and knowledge of its methods.

RESULTS

In both states, report cards were widely disseminated within hospitals. The mean quality rating was higher (P = 0.0074) in New York than in California; New York respondents appeared to be more knowledgeable about key methods. One or more hospital characteristics was associated with each outcome measure. Leaders at high-mortality hospitals were especially critical and did not find the report useful, despite limited understanding of its methods.

CONCLUSIONS

Recent hospital report cards were rated better than pioneering federal efforts. A report based on clinical data was rated better, understood better, and disseminated more often to key staff than one that was based on administrative data. Barriers to constructive use of outcomes data persist, especially at high mortality hospitals.

Professional competence in medicine

Professional competence in medicine is under the microscope following a year of government and media attention directed at the performance of doctors and outcomes for patients. The ability of the profession to self-regulate has been questioned and the roles of the state, the universities, the Royal colleges and the postgraduate deans are shifting. This paper provides a context for considering these important changes.

Assessment of cardiovascular risks and overall risks for noncardiac surgery

Appropriate care of the elderly patient requires a concerted multi-disciplinary approach before, during, and after surgery to optimize functional outcomes, with the principal focus placed on improving quality of life and strategies for risk reduction. Perioperative physicians must be able to assess the biologic, not the chronologic, age of geriatric patients and their capacity for independent function. Physicians need to understand alterations in the physiology of elderly patients attributable to the normal aging process as well as the prevalence of concurrent pathologic conditions that necessitate special precautions. Maintaining autonomy and function as a result of an acute surgical intervention may be the most important outcome to the elderly patient. Most of the data available and guidelines promulgated do not specifically address the elderly population. It is important to collect data prospectively and use sophisticated methods for analyses to develop better management algorithms for these (often complicated) clinical issues in the elderly.

Accuracy of risk-adjusted mortality rate as a measure of hospital quality of care

OBJECTIVES

Reports on hospital quality performance are being produced with increasing frequency by state agencies, commercial data vendors, and health care purchasers. Risk-adjusted mortality rate is the most commonly used measure of quality in these reports. The purpose of this study was to determine whether risk-adjusted mortality rates are valid indicators of hospital quality performance.

METHODS

Based on an analytical model of random measurement error, sensitivity and predictive error of mortality rate indicators of hospital performance were estimated.

RESULTS

The following six parameters were shown to determine accuracy: (1) mortality risks of patients who receive good quality care and (2) of those who receive poor quality care, (3) proportion of patients (across all hospitals) who receive poor quality care, (4) proportion of hospitals considered to be "poor quality," (5) patients' relative risk of receiving poor quality care in "good quality" and in "poor quality" hospitals, and (6) number of patients treated per hospital. Using best available values for model parameters, analyses demonstrated that in nearly all situations, even with perfect risk adjustment, identifying poor quality hospitals on the basis of mortality rate performance is highly inaccurate. Of hospitals that delivered poor quality care, fewer than 12% were identified as high mortality rate outliers, and more than 60% of outliers were actually good quality hospitals.

CONCLUSIONS

Under virtually all realistic assumptions for model parameter values, sensitivity was less than 20% and predictive error was greater than 50%. Reports that measure quality using risk-adjusted mortality rates misinform the public about hospital performance.

Research evidence on the validity of risk-adjusted mortality rate as a measure of hospital quality of care

For more than 10 years, reports comparing quality of care in hospitals have been disseminated to the public. The most commonly used measure in these reports is hospital mortality rate. Despite the resources devoted to analyzing and disseminating mortality data, little attention has been given to the question of their validity as a quality measure. In this article, the authors synthesize findings from 18 articles identified as providing information relevant to this issue. From this review, the authors find evidence that poor quality care increases patients' risk of mortality and that, on average, quality of care provided in hospitals identified as high—mortality rate outliers is poorer than that provided in low—mortality rate outlier hospitals. Nevertheless, a clear conclusion from these studies is that when used as a measure of quality for individual hospitals, risk-adjusted mortality rates are seriously inaccurate. Publication of hospital mortality rates misinforms the public about hospital quality.

The effects of New York's bypass surgery provider profiling on access to care and patient outcomes in the elderly

OBJECTIVE

The aim of this study was to examine the effects of provider profiling on bypass surgery access and outcomes in elderly patients in New York.

BACKGROUND

Since 1989, New York (NY) has compiled provider-specific bypass surgery mortality reports. While some have proposed that "provider profiling" has led to lower surgical mortality rates, critics have suggested that such programs lower in-state procedural access (increasing out-of-state transfers) without improving patient outcomes.

METHODS

Using national Medicare data, we examined trends in the percentages of NY residents aged 65 years or older receiving out-of-state bypass surgery between 1987 and 1992 (before and after program initiation). We also examined in-state procedure use among elderly myocardial infarction patients during this period. Finally, we compared trends in surgical outcomes in NY Medicare patients with those for the rest of the nation.

RESULTS

Between 1987 and 1992, the percentage of NY residents receiving bypass out-of-state actually declined (from 12.5% to 11.3%, p < 0.01 for trend). An elderly patient's likelihood for bypass following myocardial infarction in NY increased significantly since the program's initiation. Between 1987 and 1992, unadjusted 30-day mortality rates following bypass declined by 33% in NY Medicare patients compared with a 19% decline nationwide (p < 0.001). As a result of this improvement, NY had the lowest risk-adjusted bypass mortality rate of any state in 1992.

CONCLUSIONS

We found no evidence that NY's provider profiling limited procedure access in NY's elderly or increased out-of-state transfers. Despite an increasing preoperative risk profile, procedural outcomes in NY improved significantly faster than the national average.

What patients should ask of consumers' guides to health care quality

Consumers' guides that profile the quality of care of individual health care providers may be influential in shaping health care markets. We propose four simple questions that can be used to evaluate such guides: (a) Does the guide measure distinct and important domains of health care quality? (b) Are the individual measures of quality described simply and precisely? (c) Do the measures take into account relevant differences between patients? (d) Are the ratings of quality presented fairly? Using these four questions, we examine the validity of one prominent guide that annually identifies America's best hospitals and present a set of recommendations for the design of future guides. Although the evaluation of health care quality is undoubtedly complex, the four questions that we pose provide a basis for developing a more rational approach to informing the public about health care quality.

Trust in performance indicators?

The 1980s and 90s have seen the proliferation of all forms of performance indicators as part of attempts to command and control health services. The latest area to receive attention is health outcomes. Published league tables of mortality and other health outcomes have been available in the United States for some time and in Scotland since the early 1990s; they have now been developed for England and Wales. Publication of these data has proceeded despite warnings as to their limited meaningfulness and usefulness. The time has come to ask whether the remedy is worse than the malady: are published health outcomes contributing to quality efforts or subverting more constructive approaches? This paper argues that attempts to force improvements through publishing health outcomes can be counterproductive, and outlines an alternative approach which involves fostering greater trust in professionalism as a basis for quality enhancements.

Evaluation of acute physiology and chronic health evaluation III predictions of hospital mortality in an independent database

OBJECTIVE

To assess the accuracy and validity of Acute Physiology and Chronic Health Evaluation (APACHE) III hospital mortality predictions in an independent sample of U.S. intensive care unit (ICU) admissions.

DESIGN

Nonrandomized, observational, cohort study.

SETTING

Two hundred eighty-five ICUs in 161 U.S. hospitals, including 65 members of the Council of Teaching Hospitals and 64 nonteaching hospitals.

PATIENTS

A consecutive sample of 37,668 ICU admissions during 1993 to 1996; including 25,448 admissions at hospitals with >400 beds and 1,074 admissions at hospitals with <200 beds.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We used demographic, clinical, and physiologic information recorded during ICU day 1 and the APACHE III equation to predict the probability of hospital mortality for each patient. We compared observed and predicted mortality for all admissions and across patient subgroups and assessed predictive accuracy using tests of discrimination and calibration. Aggregate hospital death rate was 12.35% and predicted hospital death rate was 12.27% (p =.541). The model discriminated between survivors and nonsurvivors well (area under receiver operating curve = 0.89). A calibration curve showed that the observed number of hospital deaths was close to the number of deaths predicted by the model, but when tested across deciles of risk, goodness-of-fit (Hosmer-Lemeshow statistic, chi-square = 48.71, 8 degrees of freedom, p< .0001) was not perfect. Observed and predicted hospital mortality rates were not significantly (p < .01) different for 55 (84.6%) of APACHE III's 65 specific ICU admission diagnoses and for 11 (84.6%) of the 13 residual organ system-related categories. The most frequent diagnoses with significant (p < .01) differences between observed and predicted hospital mortality rates included acute myocardial infarction, drug overdose, nonoperative head trauma, and nonoperative multiple trauma.

CONCLUSIONS

APACHE III accurately predicted aggregate hospital mortality in an independent sample of U.S. ICU admissions. Further improvements in calibration can be achieved by more precise disease labeling, improved acquisition and weighting of neurologic abnormalities, adjustments that reflect changes in treatment outcomes over time, and a larger national database.

Does volume really affect outcome? Lessons from the evidence

There is a prevailing consensus that the quality of health services can be improved by concentrating care in the hands of those providers who carry out larger volumes of activity. The substantial research literature indicates a positive volume-quality relationship. However, these conclusions are largely based on observational studies using administrative databases which are poorly adjusted for case mix. Better control for confounding shows that volume-quality effects in several cases may be an artefact. The research is also difficult to interpret because of the limited measurement of outcomes, poor analysis of the relative contributions of the clinician and the hospital levels, and the lack of clarity about the direction of cause and effect. Most research is insufficiently reliable to inform policy on the use of volume for credentialling or for the re-configuration of services.

Variations in standardized hospital mortality rates for six common medical diagnoses: implications for profiling hospital quality

OBJECTIVES

The authors determined whether standardized hospital mortality rates varied for six common medical diagnoses.

METHODS

The retrospective cohort study included 89,851 patients aged 18 years and older discharged from 30 hospitals in a large metropolitan area in 1991 to 1993 with a principal diagnosis of acute myocardial infarction, congestive heart failure, pneumonia, stroke, obstructive lung disease, or gastrointestinal hemorrhage. For each hospital, standardized mortality ratios (observed/predicted mortality) were determined using validated risk-adjustment models that were based on clinical data elements abstracted from patients' hospital records. Hospitals also were categorized into quintiles on the basis of standardized mortality ratios. Correlations between standardized mortality ratios and agreement between quintile rankings were determined for each pair of diagnoses.

RESULTS

Correlations between hospital-standardized mortality ratios for individual diagnoses were generally weak. For the 15 possible pairs of diagnoses, Pearson coefficients ranged from -0.10 to 0.43; only six were 0.30 or greater. Agreement between hospital quintile rankings was also generally low, with weighted kappa values ranging from -0.12 to 0.42. Three of 15 kappa values were less than 0 (ie, agreement lower than chance), and only four exceeded 0.20, the threshold for "fair" agreement. Although simulated analyses found that random variation and relatively low hospital volumes accounted for some of the difference in standardized mortality ratios for diagnoses, a large proportion of the difference remained unexplained.

CONCLUSIONS

Standardized hospital mortality rates varied for six diagnoses that likely are managed by similar practitioners. Although variability may be decreased by restricting analyses to hospitals with large volumes, the findings indicate that for many hospitals, diagnosis-specific mortality rates may be an inconsistent measure of hospital quality, even when data are aggregated for multiple years.

Annual league tables of mortality in neonatal intensive care units: longitudinal study. International Neonatal Network and the Scottish Neonatal Consultants and Nurses Collaborative Study Group

OBJECTIVE

To assess whether crude league tables of mortality and league tables of risk adjusted mortality accurately reflect the performance of hospitals.

DESIGN

Longitudinal study of mortality occurring in hospital.

SETTING

9 neonatal intensive care units in the United Kingdom.

SUBJECTS

2671 very low birth weight or preterm infants admitted to neonatal intensive care units between 1988 and 1994.

MAIN OUTCOME MEASURES

Crude hospital mortality and hospital mortality adjusted using the clinical risk index for babies (CRIB) score.

RESULTS

Hospitals had wide and overlapping confidence intervals when ranked by mortality in annual league tables; this made it impossible to discriminate between hospitals reliably. In most years there was no significant difference between hospitals, only random variation. The apparent performance of individual hospitals fluctuated substantially from year to year.

CONCLUSIONS

Annual league tables are not reliable indicators of performance or best practice; they do not reflect consistent differences between hospitals. Any action prompted by the annual league tables would have been equally likely to have been beneficial, detrimental, or irrelevant. Mortality should be compared between groups of hospitals using specific criteria-such as differences in the volume of patients, staffing policy, training of staff, or aspects of clinical practice-after adjusting for risk. This will produce more reliable estimates with narrower confidence intervals, and more reliable and rapid conclusions.

Issues of cost and quality: barriers to an informed debate

Debates over health reform in the United States are hampered by a poorly informed public and misunderstandings about the concepts of quality, cost containment and their relationship to each other. This paper explores the nature and persistence of barriers to an informed public discussion of reform proposals. Those barriers are: (1) multiple definitions of quality, cost and cost containment, (2) the impact of the media on those definitions, (3) a false assumption that cost containment automatically results in diminished quality, and (4) the perceived impact of managed care and for-profit health firms on that assumption. We suggest a framework for building the understanding and knowledge base necessary to a reform of the nation's health care system.

Total hip replacement: a case history

The history of total hip replacement in the U.S. demonstrates that health care providers can reduce costs while improving quality. Nationwide, the cost of total hip replacements has declined dramatically while quality has improved. This article describes 14 clinical and management innovations ranging from patient education to competitive bidding.

League tables and acute myocardial infarction

BACKGROUND

Comparison of the outcomes of care provided by hospitals is a growing trend. Outcomes need to be distinguished into those attributable to the practice of hospitals and those that arise from differences in the characteristics of patients and the underlying morbidity of the populations for whom hospitals provide care. We explored these issues for deaths in hospital or within 30 days of discharge after acute myocardial infarction in Scotland, UK.

METHODS

We used records from December, 1992, to November, 1993, for 14,359 episodes of acute myocardial infarction, the death records of those who died, and 9391 death records for individuals who died after acute myocardial infarction but who had not been in hospital in the 30 days before death. Hospital discharge records were taken from the Scottish Morbidity Records. The outcomes we investigated were all-cause mortality within 30 days of discharge from hospital, and death from acute myocardial infarction at any time during the study period. We estimated separately effects attributable to patients' characteristics, hospitals, and areas of residence with multilevel modelling.

FINDINGS

We found significant differences between hospitals by age, sex, and medical history. The odds ratios for death ranged from 0.62 (95% CI 0.50-0.80) to 1.28 (1.07-1.59), relative to the average performance for Scotland as a whole. Analysis including area of residence, deaths occurring out of hospital, and more detailed information about patients showed no significant differences between hospitals for patients aged 70 years. By postcode area, there was a strong association between out-of-hospital deaths and deaths in hospital or shortly after discharge.

INTERPRETATION

Hospital outcomes may vary from one subgroup of patients to another and should be assessed independently of patients' areas of residence. Measures of performance that do not provide valid comparisons could diminish public confidence in hospital services.

A mental health program report card: a multidimensional approach to performance monitoring in public sector programs

This report presents a comprehensive, multi-dimensional mental health program performance monitoring system that has recently been implemented in the Department of Veterans Affairs. Principles underlying the development of the system are reviewed and 68 specific monitors are described addressing four major performance domains: access, inpatient care, outpatient care, and economic performance. Simple methods are presented for identifying outliers, for generating summary performance scores across series' of related monitors, and for adjusting results for differences in patient characteristics across locales. Although still technically imperfect, and therefore requiring continuous improvement, monitoring systems such as the one presented can be useful tools guiding and improving service delivery and mental health system performance, and providing a medium of accountability to consumers and other stakeholders.

[First annual report of practitioners of interventional cardiology in private practice in Germany. Results of procedures of left heart catheterization and coronary interventions in the year 1996]

The German Society for Cardiac Angiography and Interventions in Private Practice has started a registry of cardiac procedures since 1996 in order to establish a standard for performance. Although quality management for the cath lab makes sense and is also legally required, there is no generally recommended infrastructure for quality assurance existing in Germany at this time. Therefore, the German Society of Cardiologists in Private Practice (BNK) initiated a project in 1994 to develop a computer program for paperless documentation of diagnostic cardiac catheterizations and coronary interventions (PTCA) using a minimal data set. In 1996, 8 private associated groups participated in this project. The (anonymous) analysis of 10,316 diagnostic cardiac catheterizations and 2597 PTCA yielded the following results: In 95% of the patients, diagnostic cardiac catheterization was performed using the femoral and in 5% the brachial/radial approach. The mean volume of administered contrast medium was 164 +/- 138 ml/patient. The mean LV-EF was greater than 50% in 58.4% of the patients and between 30% and 50% in 10.1%. Coronary artery disease was diagnosed in 69.6% of the patients and valvular/congenital heart disease in 8.5%. In 18.4% of the patients undergoing diagnostic cardiac catheterizations no significant heart disease was identified. Mortality in the cath lab as well as the rate of cerebral insults was 0.05%. In 22.9% and 19% of the patients PTCA and cardiac surgery respectively was recommended. In patients undergoing PTCA, stable angina was present in 74.4% and unstable angina in 13.1%. Of the total number of PTCA procedures, 5.8% were performed in the setting of acute myocardial infarction. The PTCA lesion success rate was 96%, the mean diameter stenosis was 81% pre and 6% post-intervention. The mortality rate at 1 month post-PTCA was 0.4%, and myocardial infarction 1.0%. An acute occlusion occurred in 1.3% of the PTCA patients; 0.6% had to be transferred for emergency bypass surgery. None of the cath labs had on-site surgery. In comparison to other registries, our data show some similarities but also some different trends. Thus, our newly developed software proved to be reliable, fast and easy to use. Participating centers receive immediate feedback regarding their position within the whole group.

How Pennsylvania hospitals have responded to publicly released reports on coronary artery bypass graft surgery

BACKGROUND

A Consumer Guide to Coronary Artery Bypass Graft Surgery, published annually since 1992 by the Pennsylvania Health Care Cost Containment Council, compares the outcomes and charges for the state's hospitals and surgeons providing this surgery. To determine whether performance data caused hospitals to change their policies and practices, hospitals were surveyed in Pennsylvania, where the state releases annual coronary artery bypass graft (CABG) outcomes data and, as a control, in New Jersey, where the state does not release these data.

METHODS

Key informants representing hospitals, health insurance payers, health maintenance organizations, and purchasers were asked to list specific changes made because of comparative performance data released in public reports. Focus groups were conducted and surveys were then developed and administered to samples of hospitals, payers, and purchasers in both states.

RESULTS

The results suggested, for example, that access to performance information encouraged hospitals to implement new approaches to marketing their CABG services. Thirty-eight percent of Pennsylvania CABG hospitals reported using performance information to recruit staff thoracic surgeons and residents, compared with none in New Jersey. For the most frequently initiated changes in patient care, the Pennsylvania hospitals depended on performance information released by a "government agency" to a much greater degree than did the hospitals in New Jersey.

DISCUSSION

The results suggest that public release of performance information has encouraged hospitals in Pennsylvania to make changes in the areas of marketing, governance, and clinical care and that the impact of the release of public data on performance was greater in Pennsylvania hospitals than New Jersey hospitals.

Caveat doctor: how to analyze claims-based report cards

"Report cards" based on claims (billing) data are being widely used to evaluate the quality of care given by providers, even though they often lack sufficient clinical detail to render definitive judgments. Furthermore, their accuracy, especially for outpatient care, is quite variable. Nevertheless, claims data will continue to be used until better clinical information becomes widely available. To determine the suitability of automated claims data for measuring clinical performance, careful attention should be paid to the integrity of the data. Providers profiled by claims-based report cards should ask four questions about the source, robustness, management, and analysis of the data: 1. What are the key characteristics of the data set used to construct the profile? These include the insurer's name, coverage type, time period, geographic area, and number of patients, claims lines, and providers. 2. What clinical conditions and events are being measured and how well? In short, are the patients' conditions and their clinical encounters reasonably well characterized? 3. Is the information about the patients and providers accurate and up to date? 4. Once the insurer receives the medical claim, are data elements deleted or altered in ways that might affect their accuracy and completeness? Ensuring data integrity is not sufficient; the analysis of the data must be scrutinized. Potential pitfalls in analyzing claims data arise in choosing clinically meaningful measures, recognizing important differences in patients and their providers, and making fair comparisons against appropriate benchmarks. Monitoring patient care outcomes is no longer voluntary. By routinely constructing and augmenting profiles using outpatient claims data, provider groups become proactive rather than reactive in evaluating their patients' care.

League tables for performance improvement in health care

OBJECTIVES

Increasingly health care performance data are being disseminated in the form of 'league tables' of health care providers, with the implication that such publication helps purchasers select the better providers, and spurs providers into improvements. This paper examines progress to date.

METHODS

Three stages of the league table process are considered: measurement, analysis and action.

RESULTS

A wide range of measurement schemes are now in place, although the emphasis has been on process variables and mortality as a measure of outcome. Several analytical techniques have been deployed to help users make sense of league tables, and to help determine the causes of variations in reported performance. The weakest aspect of current methods relates to the use to which such analysis is put.

CONCLUSIONS

A haphazard approach to using league table data exists, with few reports on the impact of publication. A variety of directions for future research into the use of performance data are needed.

Predicting in-hospital deaths from coronary artery bypass graft surgery. Do different severity measures give different predictions?

OBJECTIVES

Severity-adjusted death rates for coronary artery bypass graft (CABG) surgery by provider are published throughout the country. Whether five severity measures rated severity differently for identical patients was examined in this study.

METHODS

Two severity measures rate patients using clinical data taken from the first two hospital days (MedisGroups, physiology scores); three use diagnoses and other information coded on standard, computerized hospital discharge abstracts (Disease Staging, Patient Management Categories, all patient refined diagnosis related groups). The database contained 7,764 coronary artery bypass graft patients from 38 hospitals with 3.2% in-hospital deaths. Logistic regression was performed to predict deaths from age, age squared, sex, and severity scores, and c statistics from these regressions were used to indicate model discrimination. Odds ratios of death predicted by different severity measures were compared.

RESULTS

Code-based measures had better c statistics than clinical measures: all patient refined diagnosis related groups, c = 0.83 (95% C.I. 0.81, 0.86) versus MedisGroups, c = 0.73 (95% C.I. 0.70, 0.76). Code-based measures predicted very different odds of dying than clinical measures for more than 30% of patients. Diagnosis codes indicting postoperative, life-threatening conditions may contribute to the superior predictive power of code-based measures.

CONCLUSIONS

Clinical and code-based severity measures predicted different odds of dying for many coronary artery bypass graft patients. Although code-based measures had better statistical performance, this may reflect their reliance on diagnosis codes for life-threatening conditions occurring late in the hospitalization, possibly as complications of care. This compromises their utility for drawing inferences about quality of care based on severity-adjusted coronary artery bypass graft death rates.

Public release of cardiac surgery outcomes data in New York: what do New York state cardiologists think of it?

BACKGROUND AND OBJECTIVE

Since 1990, risk-adjusted outcomes for patients undergoing coronary artery bypass graft surgery in New York state have been released to the public. The purpose of this study was to assess the extent to which referring cardiologists share these data with patients and use these data to make referrals.

METHODS

A survey questionnaire was sent to all cardiologists in New York in the New York State Chapter of the American College of Cardiology.

RESULTS

Four hundred fifty cardiologists responded to the survey. Most (94%) found the report "easy to read." A majority (67%) found the report to be "very accurate" or "somewhat accurate" in capturing differences in the performance of cardiac surgeons, whereas 33% found it to be "not at all accurate." Twenty-two percent reported that they "routinely discuss the reports with their patients," and 38% responded that the information has affected their referrals to surgeons "very much" or "somewhat."

CONCLUSIONS

A majority of cardiologists has not generally changed their well-established referral patterns as a result of the New York coronary artery bypass graft surgery reports. However, there has been a modest impact on referrals resulting from the distribution of these reports. The findings also suggest that increased dialogue between clinicians and policy makers regarding the format and structure of public releases would be a valuable undertaking.

Quality assessment and tracking results of cardiac surgery

Long-standing efforts to assess quality in medical care have been intensified by the striking expansion of managed care plans. Agencies such as the Joint Commission on Accreditation of Health Care Organizations and the National Committee on Quality Assurance have formalized the evaluation of health plan quality using criteria of structure, process, and outcome. A review of attempts to apply these criteria to individual physicians and to disease-specific interventions such as myocardial revascularization demonstrates the great difficulty of reliable quality assessment in this evolving surgical field. Cardiac surgeons must continue their work in deriving valid socioeconomic and clinical conclusions from The Society of Thoracic Surgeons and Veterans Affairs databases. This may prevent the precipitate adoption of newer treatment methods driven by entrepreneurial technology companies and large group purchasers of care. These entities tend to focus on economics rather than patient welfare. New technologies may also delude patients into insisting on treatment featuring short-term convenience and comfort despite less satisfactory long-term results. "Black box" methodology providing practice profiles and physician report cards must have critical validation.

Assessing the outcomes of coronary artery bypass graft surgery: how many risk factors are enough? Steering Committee of the Cardiac Care Network of Ontario

OBJECTIVES

We sought to determine whether more comprehensive risk-adjustment models have a significant impact on hospital risk-adjusted mortality rates after coronary artery bypass graft surgery (CABG) in Ontario, Canada.

BACKGROUND

The Working Group Panel on the Collaborative CABG Database Project has categorized 44 clinical variables into 7 core, 13 level 1 and 24 level 2 variables, to reflect their relative importance in determining short-term mortality after CABG.

METHODS

Using clinical data for all 5,517 patients undergoing isolated CABG in Ontario in 1993, we developed 12 increasingly comprehensive risk-adjustment models using logistic regression analysis of 6 of the Panel's core variables and 6 of the Panel's level 1 variables. We studied how the risk-adjusted mortality rates of the nine cardiac surgery hospitals in Ontario changed as more variables were included in these models.

RESULTS

Incorporating six of the core variables in a risk-adjustment model led to a model with an area under the receiver operating characteristic (ROC) curve of 0.77. The ROC curve area slightly improved to 0.79 with the inclusion of six additional level 1 variables (p = 0.063). Hospital risk-adjusted mortality rates and relative rankings stabilized after adjusting for six core variables. Adding an additional six level 1 variables to a risk-adjustment model had minimal impact on overall results.

CONCLUSIONS

A small number of core variables appear to be sufficient for fairly comparing risk-adjusted mortality rates after CABG across hospitals in Ontario. For efficient interprovider comparisons, risk-adjustment models for CABG could be simplified so that only essential variables are included in these models.

Using routine data to evaluate quality of care in British hospitals

OBJECTIVES

As part of the process of developing an international comparative study of the outcome of hospital care it has been necessary to understand what data on hospital activity are available and how they compare with those in other countries. The authors describe the administrative data collected in British National Health Service Hospitals, with emphasis on how they differ from that available in the United States.

METHODS

A description of the content of administrative data and a selective review of published literature related to their uses.

RESULTS

Administrative data in the United Kingdom resembles that available in the United States, but it also has some important differences. These include a different system of procedure classification, the use of International Classification of Diseases, 10th Revision (rather than International Classification of Diseases, Ninth Revision, Clinical Modification), and the use of a different denominator ("finished consultant episodes," rather than admission spells). There also are important differences in the completeness and precision of the data that are collected.

CONCLUSIONS

British administrative data can be used to describe patterns of care and have the potential to identify possible differences in outcome that can then be subjected to more detailed scrutiny. They are substantially less detailed than US data but are also somewhat less costly to collect.

Are the best coronary artery bypass surgeons identified by physician surveys?

OBJECTIVES

This study assessed the validity of surveys for identifying the best coronary artery bypass surgeons.

METHODS

Data on physicians who performed coronary artery bypass surgery were available from New York, Pennsylvania, and Wisconsin. Data on physicians' reputation were obtained from one national and five city surveys. The measure of surgical performance was the mortality ratio (MR), that is, the ratio of the observed to the predicted patient mortality rate.

RESULTS

Mortality ratios were very similar for the 10,722 patients treated by the 31 surgeons defined as "best" doctors in the surveys (MR = 98) and for the 74,854 patients treated by 243 other surgeons who had more than a minimal number of cases (MR = .96). The mortality ratio was 1.34 for the patients treated by surgeons with the lowest volumes and .87 for the surgeons who performed more than 400 coronary artery bypass surgeries in 3 years.

CONCLUSIONS

These results suggest that the quality of a coronary artery bypass surgeon may be more closely associated with patient volume than with the surgeon's reputation among peers.

Variations in medical care for HIV-related Pneumocystis carinii pneumonia: a comparison of process and outcome at two hospitals

BACKGROUND

Institutional variation in the quality of medical care may be evaluated by examining process measures, such as use of diagnostic procedures or treatment modalities, or outcome measures, such as mortality. We undertook this study to examine variations in both process and outcome of care for patients with HIV-related Pneumocystis carinii pneumonia (PCP) at two geographically diverse, HIV-experienced, public municipal hospitals.

DESIGN

Retrospective review of hospitalized patients diagnosed as having PCP cared for at two municipal hospitals from 1988 to 1990. At hospital A, charts of all patients diagnosed as having PCP were abstracted (n=209); at hospital B, a random sample of 15% were abstracted (=136).

RESULTS

Among all hospitalized patients diagnosed as having PCP, the frequency of making a definitive diagnosis of PCP (as opposed to treating empirically) differed markedly at the two hospitals (85% in hospital A vs 26% in hospital B; p<0.001), as did the use of intensive care (18% vs 3%; p<0.001) and "do-not-resuscitate" orders (39% vs 14%; p<0.001), although the timing of starting anti-Pneumocystis medications (89% vs 88% within the first 2 hospital days) and the use of corticosteroids (21% vs 23%) were similar. Despite differences in the process of care, survival rates were similar at the two institutions (75% vs 76%; p=0.8) and remained similar when logistic regression was used to control for demographic variables and severity of illness (odds ratio for survival, hospital B vs A, 1.2 [95% confidence interval, 0.7, 2.0]). The 95% confidence intervals (0.7, 2.0), however, were consistent with a considerable (and clinically significant) disparity in survival (from 30% lower to a twofold higher odds of survival). Sample size calculations showed that a sample of 10 cases in each hospital would be required to detect the observed difference in definitive diagnosis rates (85% vs 26%), but 722 cases in each hospital would be required to detect a relevant difference in mortality.

CONCLUSIONS

The process of care for hospitalized patients with PCP in these two institutions differed considerably, but the survival rates were not significantly different, even after adjusting for confounding factors. While sample sizes available at the individual institutions were sufficient for evaluation of the process of care, they did not provide the power necessary to evaluate outcomes. Comparisons of outcomes such as mortality between individual hospitals may not have the statistical power to exclude important differences.

Declines in hospital mortality associated with a regional initiative to measure hospital performance

To determine changes in hospital mortality that occurred in association with the dissemination of data by a regional initiative to profile hospital performance, we conducted a retrospective cohort study of patients admitted before and subsequent to dissemination of comparative data in 1992. The analysis included 101,060 consecutive eligible discharges from 30 hospitals in Northeast Ohio with eight diagnoses: acute myocardial infarction, congestive heart failure (CHF), obstructive airway disease, gastrointestinal hemorrhage, pneumonia, stroke, coronary artery bypass surgery, and lower bowel resection. Baseline (1991, N = 35,629) mortality rates were compared to rates during three subsequent periods (July-December 1992, N = 20,392; January-June 1993, N = 23,070; and July-December 1993, N = 21,969). Mortality rates were risk-adjusted using validated multivariable models based on data abstracted from patient's medical records. For all conditions, risk-adjusted mortality declined from a baseline rate of 7.5% to rates of 6.8%, 6.8%, and 6.5%, respectively, during the three subsequent periods. Using weighted linear regression analysis to estimate trends across periods, declines in mortality rates were significant for CHF (0.50% per period; P = 0.002) and pneumonia (0.38% per period; P = 0.03). We conclude that hospital mortality declined in association with the dissemination of comparative data. Although changes in hospital care were not directly examined, the results suggest that initiatives to examine provider performance may have a beneficial impact on quality of care.

Public release of cardiac surgery outcomes data in New York: what do New York state cardiologists think of it?

BACKGROUND AND OBJECTIVE

Since 1990, risk-adjusted outcomes for patients undergoing coronary artery bypass graft surgery in New York state have been released to the public. The purpose of this study was to assess the extent to which referring cardiologists share these data with patients and use these data to make referrals.

METHODS

A survey questionnaire was sent to all cardiologists in New York in the New York State Chapter of the American College of Cardiology.

RESULTS

Four hundred fifty cardiologists responded to the survey. Most (94%) found the report "easy to read." A majority (67%) found the report to be "very accurate" or "somewhat accurate" in capturing differences in the performance of cardiac surgeons, whereas 33% found it to be "not at all accurate." Twenty-two percent reported that they "routinely discuss the reports with their patients," and 38% responded that the information has affected their referrals to surgeons "very much" or "somewhat."

CONCLUSIONS

A majority of cardiologists has not generally changed their well-established referral patterns as a result of the New York coronary artery bypass graft surgery reports. However, there has been a modest impact on referrals resulting from the distribution of these reports. The findings also suggest that increased dialogue between clinicians and policy makers regarding the format and structure of public releases would be a valuable undertaking.

The MHSIP mental health report card. A consumer-oriented approach to monitoring the quality of mental health plans. Mental Health Statistics Improvement Program

Health care report cards have been endorsed as a mechanism for efficiently comparing key quantifiable aspects of performance across a range of health systems or plans. There are challenges in determining what to measure; how to gather and analyze data; and how to report, interpret, and use findings. Mental health has received little attention, and a consumer perspective is typically not included. The proposed MHSIP mental health report card (MMHRC) addresses these concerns. General issues for report cards are discussed, and the MMHRC is described in terms of content, data sources and quality, and analysis and reporting.

Coronary artery bypass risk prediction using neural networks

BACKGROUND

Neural networks are nonparametric, robust, pattern recognition techniques that can be used to model complex relationships.

METHODS

The applicability of multilayer perceptron neural networks (MLP) to coronary artery bypass grafting risk prediction was assessed using The Society of Thoracic Surgeons database of 80,606 patients who underwent coronary artery bypass grafting in 1993. The results of traditional logistic regression and Bayesian analysis were compared with single-layer (no hidden layer), two-layer (one hidden layer), and three-layer (two hidden layer) MLP neural networks. These networks were trained using stochastic gradient descent with early stopping. All prediction models used the same variables and were evaluated by training on 40,480 patients and cross-validation testing on a separate group of 40,126 patients. Techniques were also developed to calculate effective odds ratios for MLP networks and to generate confidence intervals for MLP risk predictions using an auxiliary "confidence MLP."

RESULTS

Receiver operating characteristic curve areas for predicting mortality were approximately 76% for all classifiers, including neural networks. Calibration (accuracy of posterior probability prediction) was slightly better with a two-member committee classifier that averaged the outputs of a MLP network and a logistic regression model. Unlike the individual methods, the committee classifier did not overestimate or underestimate risk for high-risk patients.

CONCLUSIONS

A committee classifier combining the best neural network and logistic regression provided the best model calibration, but the receiver operating characteristic curve area was only 76% irrespective of which predictive model was used.

Effect of surgical experience on results of esophagectomy for esophageal carcinoma

BACKGROUND

Esophagectomy for esophageal cancer is associated with substantial operative morbidity and mortality. The effect of surgical experience on results of esophagectomy has received little attention in the medical literature.

METHODS

A retrospective review of esophagectomies for cancer was done.

RESULTS

Seventy-four patients underwent esophagectomy by 20 different surgeons. Three surgeons performed 6 or more esophagectomies per year ("frequent" surgeons), whereas the other 17 surgeons performed 5 or fewer esophagectomies per year ("occasional" surgeons). Forty-two patients were operated on by frequent surgeons. There were 3 (7%) anastomotic leaks and no deaths. In 32 patients operated on by occasional surgeons, there were 7 (22%) anastomotic leaks and 7 (22%) operative deaths. The anastomotic leak rates were not significantly different (P < .07), but frequent surgeons had a significantly lower operative mortality (P < .0014).

CONCLUSIONS

Esophagectomy for esophageal cancer should be performed by experienced esophageal surgeons with sufficient yearly volume of procedures to maintain competence.

Assessing open heart surgery mortality in Catalonia (Spain) through a predictive risk model

OBJECTIVE

To develop a risk stratification model to assess open heart surgery mortality in Catalonia (Spain) in order to use risk-adjusted hospital mortality rates as an approach to analyze quality of care.

METHODS

Data were prospectively collected through a specific data-sheet during 6 1/2 months in consecutive adult patients subjected to open heart surgery. The dependent variable was surgical mortality, and independent variables included were presurgical (sociodemographic data, clinical antecedents, morphological and functional studies) and surgical. The model was built on a subsample (70% of study population) through univariate and logistic regression analysis and validated in the rest of the sample.

RESULTS

The total sample was of 1309 procedures in seven hospitals; 47% of them were valve procedures. The overall crude mortality rate was 10.9% and varied among centers (range, 2.8-14.8%). Risk factors included in the model received a weight based on the logistic regression coefficient and a score was generated for each patient. The factors with the highest weight were patient older than 80 and second reoperation. Score was stratified in five categories of increasing risk. There was a good agreement between observed and predicted mortality rates in the validation group. Overall patient distribution was as follows: 52% low risk level, 16% fair, 13% high, 12% very high, and 6% extremely high risk level. Mortality rate increased from 4.2% in the low risk to 54.4% in the highest risk group. Case mix adjustment was performed through the risk score level. There were statistically significant differences in the risk profiles of patients admitted among centers. After adjustment by risk profiles, there were no differences in mortality by hospital.

CONCLUSION

A risk stratification model through a multicentric, prospective and exhaustive collection of data in all types of open heart procedures was developed. In spite of wide differences on crude rates and in the risk profiles of patients admitted, we did not find statistically significant differences in adjusted mortality rates among centers. Timely and accurate information about surgical outcomes can lead to improvements in clinical practice and quality of care.

The public release of hospital and physician mortality data in Pennsylvania. A case study

OBJECTIVES

Using the public reports of the Pennsylvania Health Care Cost Containment Council on coronary artery bypass graft surgery for 1990 to 1992 as a case study, the authors assess the sensitivity of results to the choice of data and statistical methodology.

METHODS

Using the Council's public-release data, surgical mortality and utilization were reanalyzed by standard linear models, empirical Bayes methods, Monte Carlo simulations, and hierarchical statistical models.

RESULTS

Statistical power calculations demonstrate that the annual volume of bypass surgery for many hospitals and for most surgeons is too small for meaningful mortality comparisons. The number of hospitals and physicians designated as mortality "outliers" in the Council's reports results in part from a failure to adjust critical P values for multiple comparisons. Hierarchical statistical models implemented by mixed effects logistic regression, by contrast, can detect true differences in performance without producing false outliers. Mortality analyses are sensitive to the choice of comorbidities used for severity adjustment of a mortality model. Small-area analyses indicate large differences in the rates of bypass surgery across Pennsylvania, with lower population-based rates of surgery associated with higher population-based inpatient mortality.

CONCLUSIONS

Analyses of mortality by operative procedure, rather than by patient diagnosis, should consider the potential for selection bias caused by the decision to elect surgery. The clinical and statistical issues of operative mortality are sufficiently complex to merit review by independent experts before public release of hospital and physician performance measures.

Differences in procedure use, in-hospital mortality, and illness severity by gender for acute myocardial infarction patients: are answers affected by data source and severity measure?

OBJECTIVES

According to some studies, women with heart disease receive fewer procedures and have higher in-hospital death rates than men. These studies vary by data source (hospital discharge abstract versus detailed clinical information) and severity measurement methods. The authors examined whether evaluations of gender differences for acute myocardial infarction patients vary by data source and severity measure.

METHODS

The authors considered 10 severity measures: four using clinical medical record data and six using discharge abstracts (diagnosis and procedure codes). The authors studied all 14,083 patients admitted in 1991 for acute myocardial infarction to 100 hospitals nationwide, examining in-hospital death and use of coronary angiography, coronary artery bypass graft surgery (CABG), and percutaneous transluminal coronary angioplasty (PTCA). Logistic regression was used to calculate odds ratios for death and procedure use for women compared with men, controlling for age and each of the severity scores.

RESULTS

After adjusting only for age, women were significantly more likely than men to die and less likely to receive CABG and coronary angiography. Severity measures provided different assessments of whether women were sicker than men; for all cases, clinical data-based MedisGroups rated women's severity compared with men's, whereas four code-based severity measures viewed women as sicker. After adjusting for severity and age, women were significantly more likely than men to die in-hospital and less likely to receive coronary angiography and CABG; women and men had relatively equal adjusted odds ratios of receiving PTCA. Odds ratios reflecting gender differences in procedure use and death rates were similar across severity measures.

CONCLUSIONS

Comparisons of severity-adjusted in-hospital death rates and invasive procedure use between men and women yielded similar findings regardless of data source and severity measure.

Identification of neonatal deaths in a large managed care organisation

The neonatal (< 28 days) mortality rate (NMR) is one of the most commonly employed maternal and child health epidemiological measures. It is also being employed in quality measures ("report cards') used to assess the performance of health care organisations. The objectives were to (1) develop methods for the rapid quantification of the neonatal mortality rate in a multi-hospital system, the Kaiser Permanente Medical Care Program's Northern California Region (KPMCP NCR), (2) develop methods for generating facility-specific rates and case lists, and (3) ascertain the capture rates of the information sources available to us. Potential neonatal deaths were identified in the KPMCP NCR for the 1990 and 1991 calendar years from 3 sources: (1) clerical searches of local facility records, (2) electronic searches of the KPMCP NCR hospitalisation database, and (3) linking KPMCP electronic birth records to death certificate tapes. The medical records of all infants identified through these methods were reviewed. The neonatal mortality rate was calculated in three ways: (1) including all livebirths, (2) excluding births weighing < 500 g, and (3) adjusting for prematurity by increasing the follow-up period in preterm babies (these babies were included as neonatal deaths if they died up to 40 weeks corrected age + 27.9 days). A total of 352 records out of 64 469 birth records in the KPMCP NCR were reviewed. If one includes babies < 500 g, the neonatal mortality rate was 3.72/1000 livebirths; if these babies are excluded, the rate was 3.05/1000. Adjusting for prematurity increased these rates to 3.91/1000 and 3.24/1000, respectively. Accurate quantification of the neonatal mortality rate in a multi-hospital system requires the use of multiple information sources. Use of a single source can lead to varying rates of over- or under-estimation. It is possible to employ our methodology for both research and operational purposes.