Report cards on cardiac surgeons. Assessing New York State's approach

Cardiac Surgery Risk Models: A Position Article

Differences in medical outcomes may result from disease severity, treatment effectiveness, or chance. Because most outcome studies are observational rather than randomized, risk adjustment is necessary to account for case mix. This has usually been accomplished through the use of standard logistic regression models, although Bayesian models, hierarchical linear models, and machine-learning techniques such as neural networks have also been used. Many factors are essential to insuring the accuracy and usefulness of such models, including selection of an appropriate clinical database, inclusion of critical core variables, precise definitions for predictor variables and endpoints, proper model development, validation, and audit. Risk models may be used to assess the impact of specific predictors on outcome, to aid in patient counseling and treatment selection, to profile provider quality, and to serve as the basis of continuous quality improvement activities.

Variations In The Quality Of Care For Very-Low-Birthweight Infants: Implications For Policy

Much of the decline in childhood mortality over the past two decades is attributable to improvements in neonatal intensive care for very-low-birthweight infants. Yet large and persistent disparities persist in the quality of neonatal intensive care across hospitals. Improving care for infants now served by hospitals with poor outcomes can greatly reduce infant mortality, particularly among minority infants who are more likely to be very low birthweight and cared for by hospitals with poor outcomes. Referral of high-risk births to hospitals with the best outcomes is another promising strategy.

Quality care and practice variation: the roles of practice guidelines and public profiles

Goals of the quality-of-care initiative are to improve the structure, process, and outcome of health care. The effectiveness of methods to improve quality have been largely unverified. Most methods are costly to implement and time-consuming to perform; some threaten professional autonomy. The characteristic feature of modern medicine that fuels the debate over quality is the variation in the delivery of health care. This review examines the "variation phenomenon" in medicine and the roles that practice guidelines and physician profiling have in improving health care, in general, and for adult cataract, in particular.

Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma

The history of monitoring the outcomes of health care by external agencies can be traced to ancient times. However, the danger, now as then, is that in the search for improvement, comparative measures of mortality and morbidity are often overinterpreted, resulting in judgments about the underlying quality of care. Such judgments can translate into performance management strategies in the form of capricious sanctions (such as star ratings) and unjustified rewards (such as special freedoms or financial allocations). The resulting risk of stigmatising an entire institution injects huge tensions into health-care organisations and can divert attention from genuine improvement towards superficial improvement or even gaming behaviour (ie, manipulating the system). These dangers apply particularly to measures of outcome and throughput. We argue that comparative outcome data (league tables) should not be used by external agents to make judgments about quality of hospital care. Although they might provide a reasonable measure of quality in some high-risk surgical situations, they have little validity in acute medical settings. Their use to support a system of reward and punishment is unfair and, unsurprisingly, often resisted by clinicians and managers. We argue further that although outcome data are useful for research and monitoring trends within an organisation, those who wish to improve care for patients and not penalise doctors and managers, should concentrate on direct measurement of adherence to clinical and managerial standards.

How to make a silk purse from a sow's ear--a comprehensive review of strategies to optimise data for corrupt managers and incompetent clinicians

The introduction of performance league tables for UK surgeons and hospitals has forced them to learn how to present data in the best possible light. Though there is an urgent need for guidance, official guidelines on how to optimise performance data are lacking

[Analysis of in-hospital mortality from coronary artery bypass grafting surgery]

INTRODUCTION AND OBJECTIVES

Outcomes research and monitoring are of key importance in efforts to improve health care effectiveness and quality. The aim of this study was to describe in-hospital mortality from coronary artery bypass grafting in Spain. Data in an administrative database were used to estimate the statistical performance of two risk-adjustment methods, the Charlson and Ghali indexes.

PATIENTS AND METHOD

From the Spanish Hospital Minimum Basic Data Set corresponding to 1997 and 1998 all records which included a code for coronary artery bypass grafting were selected. With in-hospital mortality as the outcome variable, two risk-adjusted logistic multiple regression models were constructed.

RESULTS

The database included 13,203 cases, of which 80% were men; mean age was 64.5 years. In-hospital mortality was 7.3%. The figure was significantly higher for women and increased with age. A score of one on the Charlson and Ghali indexes was associated, respectively, with a 23 and 20% increase in the risk of mortality. Probability calculated with the Hosmer-Lemeshow goodness of fit test was 0.765 and 0.965, and the C index was 0.66 and 0.67. Values of Nagelkerke's R2 were 0.051 y 0.058.

CONCLUSIONS

In-hospital mortality from coronary artery bypass grafting is much higher in Spain than in other countries. The Minimum Basic Data Set, a low-cost information system that is easy to access, yields interesting and useful information to measure health care quality.

Quality of health care for women: a demonstration of the quality assessment tools system

BACKGROUND

Consumers, purchasers, and regulators are seeking information on quality for a variety of purposes. To address these demands, methods are required that are flexible in meeting the information needs of different audiences.

OBJECTIVES

To test a new clinically detailed, comprehensive approach to quality measurement called Quality Assessment (QA) Tools.

DESIGN

Quality measures were developed for women ages 18 to 50 years for preventive care and 17 clinical areas that included chronic and acute health problems. A stratified random sample of women enrolled in 1 of 2 health plans in 1996 to 1997 was drawn and data abstracted from the medical records of all their providers for a 2-year period.

FINDINGS

We evaluated quality for 758 women in 2 managed care plans. Quality of care varied substantially depending on the dimension being examined. For example, acute care was significantly better than chronic or preventive care. Quality was highest for follow-up care and lowest for treatment in both plans. Quality by modality ranged from approximately 90% for referral or admission to 16% for education and counseling. We found significant differences between the plans in the quality of care for 7 of the 17 conditions studied.

CONCLUSION

The QA Tools system offers an alternative approach to evaluating health system performance. Potential advantages include the richness of the information produced by the system, the ability to create summary scores for consumers and purchasers, and the system-level performance information for use in quality improvement activities.

Using routine comparative data to assess the quality of health care: understanding and avoiding common pitfalls

Measuring the quality of health care has become a major concern for funders and providers of health services in recent decades. One of the ways in which quality of care is currently assessed is by taking routinely collected data and analysing them quantitatively. The use of routine data has many advantages but there are also some important pitfalls. Collating numerical data in this way means that comparisons can be made--whether over time, with benchmarks, or with other healthcare providers (at individual or institutional levels of aggregation). Inevitably, such comparisons reveal variations. The natural inclination is then to assume that such variations imply rankings: that the measures reflect quality and that variations in the measures reflect variations in quality. This paper identifies reasons why these assumptions need to be applied with care, and illustrates the pitfalls with examples from recent empirical work. It is intended to guide not only those who wish to interpret comparative quality data, but also those who wish to develop systems for such analyses themselves.

Quality report cards and nursing home quality

PURPOSE

This study examined the potential role that publicly disseminated quality report cards can play in improving quality of care in nursing homes.

DESIGN AND METHODS

We review the literature and the experience gained over the last two decades with report cards for hospitals, physicians, and health plans, and consider the issues that are of particular importance in the context of nursing home care.

RESULTS

Experience with report cards in other areas of the health care system suggests that nursing home quality reports may have a role to play in informing consumers' choices and providing incentives for quality improvement. Their impact may, however, not be large. Methodological issues that may limit the accuracy of quality indicators and issues related to the design and comprehension of the information by consumers are discussed.

IMPLICATIONS

Quality report cards should be viewed as one of several options to ensure higher quality nursing home care.

Colonoscopy: practice variation among 69 hospital-based endoscopists

BACKGROUND

The medical profession, payers, and patients are interested increasingly in the quality of endoscopic procedures, including colonoscopy. The American Society for Gastrointestinal Endoscopy has recommended "report cards" by which endoscopists may keep track of certain key elements of their practice including indications, findings, duration, technical end points, complications, and patient satisfaction.

METHODS

The GI-Trac endoscopy reporting database includes many of the data points recommended by ASGE for report cards. Seven hospital centers in North America have been collecting data prospectively for varying periods since 1994. These data were aggregated and analyzed by individual endoscopist. A total of 69 endoscopists performed 17,868 colonoscopies.

RESULTS

Twelve percent of the endoscopists reported that more than 20% of procedures they performed were completely normal. The average time taken by 27% of endoscopists was more than 40 minutes (without trainees involved), and only 55% achieved a cecal intubation rate of over 90%; for 9% the rate was less than 80%. Complication rates were too low for individual comparisons.

CONCLUSION

These data provide an idea of colonoscopy performance by individual endoscopists in mainly academic centers. Incorporating all recommended data elements in future reporting databases will contribute to meaningful bench marking and to quality improvement efforts.

The volume-outcome relationship: from Luft to Leapfrog

Numerous reports have documented a volume-outcome relationship for complex medical and surgical care, although many such studies are compromised by the use of discharge abstract data, inadequate risk adjustment, and problematic statistical methodology. Because of the volume-outcome association, and because valid outcome measurements are unavailable for many procedures, volume-based referral strategies have been advocated as an alternative approach to health-care quality improvement. This is most appropriate for procedures with the greatest outcome variability between low-volume and high-volume providers, such as esophagectomy and pancreatectomy, and for particularly high-risk subgroups of patients. Whenever possible, risk-adjusted outcome data should supplement or supplant volume standards, and continuous quality improvement programs should seek to emulate the processes of high-volume, high-quality providers. The Leapfrog Group has established a minimum volume requirement of 500 procedures for coronary artery bypass grafting. In view of the questionable basis for this recommendation, we suggest that it be reevaluated.

Using performance measurement to drive improvement: a road map for change

BACKGROUND

Performance measures and reporting have not been adopted throughout the US health care system despite their central role in encouraging increased participation by consumers in decision-making. Understanding whether the failure of measurement and reporting to diffuse throughout the health system can be overcome is critical for determining future policy in this area.

OBJECTIVES

To create a conceptual framework for analyzing the current rate of adoption and evaluating alternatives for accelerating adoption, and to recommend a set of concrete steps that can be taken to increase the use of performance measurement and reporting.

RESEARCH DESIGN

Review of three theoretic models (Rogers, Prochaska/DiClemente, Gladwell), examination of the literature on previous experiences with quality measurement and reporting, and interviews with select stakeholders.

FINDINGS

The three theoretic models provide a valuable framework for understanding why the use of performance measures is stalled ("the circle of unaccountability") and for generating ideas about concrete steps that could be taken to accelerate adoption. Six steps are recommended: (1) raise public awareness, (2) redesign measures and reports, (3) make the delivery of information timely, (4) require public reporting, (5) develop and implement systems to reward quality, and (6) actively court leaders.

CONCLUSIONS

The recommended six steps are interconnected; action on all will be required to drive significant acceleration in rates of adoption of performance measurement and reporting. Leadership and coordination are necessary to ensure these steps are taken and that they work in concert with one another.

Report cards: relevance for nursing and patient care safety

The emergence of health care report cards in the North American environment is outlined. While it is evident that substantial activity has emerged, the majority of these initiatives excluded nursing, or use a broad indicator for nursing that may not provide meaningful representations of the quality of nursing care provided in the system and the relevance of this care to patient care safety. Given that nurses are the primary care provider in health care settings, this represents a significant gap in health care report cards. The pioneering work of the American Nurses Association (ANA) Nursing Report Card in the development and validation of report card indicators for nursing is discussed. Challenges related to data availability and data quality are identified. Potential opportunities for linking nursing practice outcomes to patient care quality and patient safety through a report card process are outlined.

Whom should we profile? Examining diabetes care practice variation among primary care providers, provider groups, and health care facilities

OBJECTIVE

To evaluate the amount of variation in diabetes practice patterns at the primary care provider (PCP), provider group, and facility level, and to examine the reliability of diabetes care profiles constructed using electronic databases.

DATA SOURCES/STUDY SETTING

Clinical and administrative data obtained from the electronic information systems at all facilities in a Department of Veterans Affairs' (VA) integrated service network for a study period of October 1997 through September 1998.

STUDY DESIGN

This is a cohort study. The key variables of interest are different types of diabetes quality indicators, including measures of technical process, intermediate outcomes, and resource use.

DATA COLLECTION/EXTRACTION METHODS

A coordinated registry of patients with diabetes was constructed by integrating laboratory, pharmacy, utilization, and primary care provider data extracted from the local clinical information system used at all VA medical centers. The study sample consisted of 12,110 patients with diabetes, 258 PCPs, 42 provider groups, and 13 facilities.

PRINCIPAL FINDINGS

There were large differences in the amount of practice variation across levels of care and for different types of diabetes care indicators. The greatest amount of variance tended to be attributable to the facility level. For process measures, such as whether a hemoglobin A1c was measured, the facility and PCP effects were generally comparable. However, for three resource use measures the facility effect was at least six times the size of the PCP effect, and for inter-mediate outcome indicators, such as hyperlipidemia, facility effects ranged from two to sixty times the size of the PCP level effect. A somewhat larger PCP effect was found (5 percent of the variation) when we examined a "linked" process-outcome measure linking hyperlipidemia and treatment with statins). When the PCP effect is small (i.e., 2 percent), a panel of two hundred diabetes patients is needed to construct profiles with 80 percent reliability.

CONCLUSIONS

little of the variation in many currently measured diabetes care practices is attributable to PCPs and, unless panel sizes are large, PCP profiling will be inaccurate. If profiling is to improve quality, it may be best to focus on examining facility-level performance variations and on developing indicators that promote specific, high-priority clinical actions.

Is it safe to train residents to perform cardiac surgery?

BACKGROUND

The impact of surgical training on patient outcomes in cardiac surgery is unknown.

METHODS

All cases performed by residents from 1998 to 2001 were compared to staff surgeon cases using prospectively collected data. Operative mortality and a composite morbidity of: reoperation for bleeding perioperative myocardial infarction, infection, stroke, or ventilation more than 24 hours were compared using multivariate analysis.

RESULTS

Four residents performed 584 cases. The cases were as follows: coronary artery bypass grafting (CABG), 366 cases; aortic valve replacement (AVR) with or without CABG (AVR +/- CABG), 86 cases; mitral valve replacement, 31 cases; mitral valve repair, 25 cases; thoracic aneurysm/dissection, 22 cases; aortic root, 20 cases; transplantations, 14 cases; and adult congenital defect repairs, 20 cases. There were 2,638 CABGs and 363 AVR +/- CABG performed by the staff during the same period. Crude operative mortality in CABG patients was 2.5% (resident) and 2.9% (staff) (p = 0.62). In multivariate analysis, resident was not associated with operative mortality odds ratio (OR) of 0.59 (p = 0.19). Resident cases had a higher incidence of the composite morbidity outcome for CABG cases (19.4% vs 13.6% for staff; p = 0.003). However, in multivariate analysis, resident was not associated with increased morbidity (OR = 1.23, p = 0.16). The AVR +/- CABG crude mortality was 3.6% (resident) and 2.8% (staff) (p = 0.69). Because of the small number of cases (n = 447), operative mortality was combined with the composite morbidity outcome for the AVR +/- CABG model. In all, 16.7% of resident cases and 19.8% of staff cases had the composite outcome or died (p = 0.51). In multivariate analysis resident was not associated with this outcome (OR = 0.74, p = 0.35).

CONCLUSIONS

In this analysis of our experience with residency training, the operative morbidity and mortality in CABG and AVR patients was similar for residents and staff. Training residents to perform cardiac surgery appears to be safe.

Measuring outcomes with tools of proven feasibility and utility: the example of a patient-focused asthma measure

Best quality care is clearly desirable and instituting quality assurance should be logical and simple. However, the practicality of setting standards for a product or service, and designing systems to measure against the standards, is more difficult. In the health service it is only likely to be feasible if data can be generated from efficient and reliable information systems. The ideal measure of quality is an outcome measure that evaluates whether or not the quality of care has achieved the desired standard of outcome. Direct measures of outcome are not easy to construct and the information systems required to provide data are not widely available. The National Centre for Health Outcomes Development (NCHOD) has produced a series of indicators in 10 areas of health care, where an indicator is a pointer to, rather than a direct measure of, a desired outcome. Feasibility studies measuring their sensitivity and reliability have drawn attention to their possible utility within different health care settings. This paper reports on an investigation into a patient-focused outcome indicator for asthma. There is broad agreement about the need to measure the outcome of disease. However, when outcome indicators are defined there are major obstacles to their successful uptake. A key challenge for outcomes measurement is to ensure that the cost of collecting the data and ensuring completeness, accuracy and standardization are justified by the benefits derived. Health outcome indicators should not be treated as a panacea, but as a part of the clinical and health care tool kit.

Cardiac surgery report cards: comprehensive review and statistical critique

Public report cards and confidential, collaborative peer education represent distinctly different approaches to cardiac surgery quality assessment and improvement. This review discusses the controversies regarding their methodology and relative effectiveness. Report cards have been the more commonly used approach, typically as a result of state legislation. They are based on the presumption that publication of outcomes effectively motivates providers, and that market forces will reward higher quality. Numerous studies have challenged the validity of these hypotheses. Furthermore, although states with report cards have reported significant decreases in risk-adjusted mortality, it is unclear whether this improvement resulted from public disclosure or, rather, from the development of internal quality programs by hospitals. An additional confounding factor is the nationwide decline in heart surgery mortality, including states without quality monitoring. Finally, report cards may engender negative behaviors such as high-risk case avoidance and "gaming" of the reporting system, especially if individual surgeon results are published. The alternative approach, continuous quality improvement, may provide an opportunity to enhance performance and reduce interprovider variability while avoiding the unintended negative consequences of report cards. This collaborative method, which uses exchange visits between programs and determination of best practice, has been highly effective in northern New England and in the Veterans Affairs Administration. However, despite their potential advantages, quality programs based solely on confidential continuous quality improvement do not address the issue of public accountability. For this reason, some states may continue to mandate report cards. In such instances, it is imperative that appropriate statistical techniques and report formats are used, and that professional organizations simultaneously implement continuous quality improvement programs. The statistical methodology underlying current report cards is flawed, and does not justify the degree of accuracy presented to the public. All existing risk-adjustment methods have substantial inherent imprecision, and this is compounded when the results of such patient-level models are aggregated and used inappropriately to assess provider performance. Specific problems include sample size differences, clustering of observations, multiple comparisons, and failure to account for the random component of interprovider variability. We advocate the use of hierarchical or multilevel statistical models to address these concerns, as well as report formats that emphasize the statistical uncertainty of the results.

Performance indicators from routine hospital data: death following aortic surgery as a potential measure of quality of care

BACKGROUND

There is increasing pressure to monitor surgical performance. In the UK, the Department of Health has produced clinical indicators based on routine data to monitor performance. This study analysed whether such data could measure performance in aortic surgery.

METHODS

Routine hospital data on postoperative mortality were collected for 1995-1997 in the Trent region. Procedural and diagnostic codes, modes of admission, districts of residence, treatment and specialty data were compared with audit data and the Operating Theatre Information System.

RESULTS

Inaccuracies in the Health Resource Group (HRG) codes meant that 21.4 per cent of elective aortic cases (HRG Q02) were probably emergencies and 26 per cent of probable ruptured aneurysms were not coded as a vascular emergency. Case mix and patient selection introduced a bias, apparent between tertiary and district general hospitals. For patients aged over 80 years, two district hospitals undertook no elective aortic surgery; the rate for emergency aortic surgery varied between 16 and 25 per cent in the district hospitals, and was 77 per cent in the tertiary centre.

CONCLUSION

Crude mortality rates used as an indicator of performance are subject to bias and distortion owing to the collection of incorrect information, variation in patient selection between hospitals and case-mix differences. There was a considerable variation in selection and outcomes of patients undergoing aortic surgery in this study.

Publicly disclosed information about the quality of health care: response of the US public

Public disclosure of information about the quality of health plans, hospitals, and doctors continues to be controversial. The US experience of the past decade suggests that sophisticated quality measures and reporting systems that disclose information on quality have improved the process and outcomes of care in limited ways in some settings, but these efforts have not led to the "consumer choice" market envisaged. Important reasons for this failure include limited salience of objective measures to consumers, the complexity of the task of interpretation, and insufficient use of quality results by organised purchasers and insurers to inform contracting and pricing decisions. Nevertheless, public disclosure may motivate quality managers and providers to undertake changes that improve the delivery of care. Efforts to measure and report information about quality should remain public, but may be most effective if they are targeted to the needs of institutional and individual providers of care.

Risk-adjusted surgical outcomes

Measures of risk-adjusted outcome are particularly suited for the assessment of the quality of surgical care. The reliability of measures of quality that use surgical outcomes is enhanced by prospective data acquisition and should be adjusted for the preoperative severity of illness. Such measures should be based only on reliable and validated data, and they should apply state-of-the-art analytical methods. The risk-adjusted postoperative mortality rate is useful as a quality measure only in specialties and operations expected to have a high rate of postoperative deaths. Risk-adjusted complications are more common but are limited as a comparative measure of quality by a lack of uniform definitions and data collection mechanisms. In specialties in which the expected postoperative mortality is low, risk-adjusted functional outcomes are promising measures for the assessment of the quality of surgical care. Measures of cost and patient satisfaction should also be incorporated in systems designed to measure the quality and cost-effectiveness of surgical care.

Quality improvement efforts in oncology: are we ready to begin?

Large variations in the quality of cancer care are a matter of concern in the United States. Despite spending over 15% of our GNP on health care, more than any other country in the world, some cancer patients face significant risks of dying from their treatment precisely because of their choice of physician. The Institute of Medicine has reported that variations in the quality of cancer are large, and that low-experienced providers are more likely to provide a lower quality of medical care. Increased pressures to contain costs have led to concern that the quality and outcomes of cancer care may only worsen. One reaction to this situation is a greater reliance on "report cards." In an effort to address both quality and cost issues, providers are looking outside the health care sector for guidance for more acceptable alternatives to report cards, which are often viewed as punitive. The approach that they most often have selected recently is termed continuous quality improvement (CQI) or total quality management (TQM). In this article, we describe the potential benefits and drawbacks of CQI efforts in oncology, review experiences with four different CQI cancer programs, and make recommendations about future CQI efforts.

Result-based compensation in health care: a good, but limited, idea

David Hyman and Charles Silver are quite right. Opinion 6.01 in the American Medical Association's (AMA) Code of Medical Ethics is difficult to defend. Ties between compensation and outcomes need not mislead patients into thinking that results are guaranteed; they are widely used in other fields with considerable success, even if they have some disadvantages; they can potentially bring patients more actively into decision-making about whether and from whom to purchase which medical care; and, if carefully tuned, they can promote quality by aligning providers’ welfare more closely with patients’.

The purpose of this commentary is thus not to disagree with the fundamental thesis that result-based compensation arrangements (RBCAs) can be appropriate and useful in the health-care setting. Rather, the objective is to “cut the next swath,” so to speak. While Hyman and Silver are right that RBCAs have potential to do good, they may be overly optimistic about benefits, while underestimating the potential limitations and hazards.

Commentary: public disclosure in the health field: is there a relevant option?

Steadily increasing numbers of private, not-for-profit, and public agencies are releasing quality of care and financial information to enhance the market power of those purchasing America's health services. These sources range widely from the Health Plan Employer Data and Information Set (HEDIS) to the federal Securities and Exchange Commission (SEC). A growing interest in publicly disclosing performance information that concerns providers, as recently recommended by the Institute of Medicine (IOM) report on patient safety, until now has only achieved a modest impact. A major exception is in those metropolitan areas where acute-care facilities experience intense competition for patient admissions. Although the comparative quality and cost information currently available is for the most part in the public interest, it is concluded that, in the foreseeable future, these report cards as now structured will not result in any significant enhancement in access, improvement in quality, or reduction in cost. Options to encourage more public disclosure explored in this paper are Congress potentially passing legislation (a) to establish a Center for Patient Safety within the US Department of Health and Human Services as recommended by the recent Institute of IOM study, or (b) empowering the SEC beyond its current mandate to collect and disseminate all pertinent quality of care and cost information on every provider in the United States. The latter alternative would include user-friendly, comparative analyses to be provided on the Internet and elsewhere and would make readily available information from HEDIS, the Joint Commission on the Accreditation of Healthcare Organization (JCAHO), coronary artery bypass graft (CABG) surgery studies, cost comparisons based on Medicare cost reports and SEC filings, and other similar sources.

Challenges in comparing risk-adjusted bypass surgery mortality results: results from the Cooperative Cardiovascular Project

OBJECTIVES

We sought to evaluate the predictive accuracy of four bypass surgery mortality clinical risk models and to examine the extent to which hospitals' risk-adjusted surgical outcomes vary depending on which risk-adjustment method is applied.

BACKGROUND

Cardiovascular "report cards" often compare risk-adjusted surgical outcomes; however, it is unclear to what extent the risk-adjustment process itself may affect these metrics.

METHODS

As part of the Cooperative Cardiovascular Project's Pilot Revascularization Study, we compared the predictive accuracy of four bypass clinical risk models among 3,654 Medicare patients undergoing surgery at 28 hospitals in Alabama and Iowa. We also compared the agreement in hospital-level risk-adjusted bypass outcome performance ratings depending on which of the four risk models was applied.

RESULTS

Although the four risk models had similar discriminatory abilities (C-index, 0.71 to 0.74), certain models tended to overpredict mortality in higher-risk patients. There was high correlation between a hospital's risk-adjusted mortality rates regardless of which of the four models was used (correlation between risk-adjusted rating, 0.93 to 0.97). In contrast, there was limited agreement in which hospitals were identified as "performance outliers" depending on which risk-adjustment model was used and how outlier status was defined.

CONCLUSIONS

A hospital's risk-adjusted bypass surgery mortality rating, relative to its peers, was consistent regardless of the risk-adjustment model applied, supporting their use as a means of provider performance feedback. Designation of performance outliers, however, can vary significantly depending on the benchmark and methods used for this determination.

Washington State's model of physician leadership in cardiac outcomes reporting

BACKGROUND

In 1993, the cardiac surgery community in Washington State opposed an effort by the state Health Care Authority (HCA) to identify "centers of excellence" for selective contracting of coronary artery bypass grafting (CABG) procedures, and proposed an alternate model that would create a statewide cardiac outcomes registry under physician governance to be used by all institutions for internal quality improvement activities.

METHODS

A prospective pilot data collection effort, which examined preoperative and postoperative patient-reported health status, served as the basis for evaluating the capacity of a physician-led organization to develop a collaborative atmosphere and facilitate universal hospital participation.

RESULTS

A surgical steering group met on a regular basis and reached consensus on governance issues, protocols for standardized data collection, and policies regarding data dissemination. All 14 centers that performed bypass surgery in the state participated. Patients who were surveyed reported statistically significant improvements in physical, emotional, and anginal-specific health status after bypass surgery. Baseline patient characteristics and longitudinal outcomes were compared across institutions.

CONCLUSIONS

Based on the feasibility of this collaborative outcomes reporting program, the HCA revised its policy regarding selective contracting and has helped to support an ongoing physician-led and -governed cardiac outcomes reporting system that is particularly notable for the subsequent integration of both CABG surgery and catheterization-based procedures into one standardized registry.

Does clinical evidence support ICD-9-CM diagnosis coding of complications?

BACKGROUND

Hospital discharge diagnoses, coded by use of the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM), increasingly determine reimbursement and support quality monitoring. Prior studies of coding validity have investigated whether coding guidelines were met, not whether the clinical condition was actually present.

OBJECTIVE

To determine whether clinical evidence in medical records confirms selected ICD-9-CM discharge diagnoses coded by hospitals.

RESEARCH DESIGN AND SUBJECTS

Retrospective record review of 485 randomly sampled 1994 hospitalizations of elderly Medicare beneficiaries in Califomia and Connecticut.

MAIN OUTCOME MEASURE

Proportion of patients with specified ICD-9-CM codes representing potential complications who had clinical evidence confirming the coded condition.

RESULTS

Clinical evidence supported most postoperative acute myocardial infarction diagnoses, but fewer than 60% of other diagnoses had confirmatory clinical evidence by explicit clinical criteria; 30% of medical and 19% of surgical patients lacked objective confirmatory evidence in the medical record. Across 11 surgical and 2 medical complications, objective clinical criteria or physicians' notes supported the coded diagnosis in >90% of patients for 2 complications, 80% to 90% of patients for 4 complications, 70% to <80% of patients for 5 complications, and <70% for 2 complications. For some complications (postoperative pneumonia, aspiration pneumonia, and hemorrhage or hematoma), a large fraction of patients had only a physician's note reporting the complication.

CONCLUSIONS

Our findings raise questions about whether the clinical conditions represented by ICD-9-CM codes used by the Complications Screening Program were in fact always present. These findings highlight concerns about the clinical validity of using ICD-9-CM codes for quality monitoring.

Results of report cards for patients with congestive heart failure depend on the method used to adjust for severity

BACKGROUND

The validity of outcome report cards may depend on the ways in which they are adjusted for risk.

OBJECTIVES

To compare the predictive ability of generic and disease-specific survival prediction models appropriate for use in patients with heart failure, to simulate outcome report cards by comparing survival across hospitals and adjusting for severity of illness using these models, and to assess the ways in which the results of these comparisons depend on the adjustment method.

DESIGN

Analysis of data from a prospective cohort study.

SETTING

A university hospital, a Veterans Affairs (VA) medical center, and a community hospital.

PATIENTS

Sequential patients presenting in the emergency department with acute congestive heart failure.

MEASUREMENTS

Unadjusted 30-day and 1-year mortality across hospitals and 30-day and 1-year mortality adjusted by using disease-specific survival prediction models (two sickness-at-admission models, the Cleveland Health Quality Choice model, the Congestive Heart Failure Mortality Time-Independent Predictive Instrument) and generic models (Acute Physiology and Chronic Health Evaluation [APACHE] II, APACHE III, the mortality prediction model, and the Chadson comorbidity index).

RESULTS

The community hospital's unadjusted 30-day survival rate (85.0%) and the VA medical center's unadjusted 1-year survival rate (60.9%) were significantly lower than corresponding rates at the university hospital (92.7% and 67.5%, respectively). No severity model had excellent ability to discriminate patients by survival rates (all areas under the receiver-operating characteristic curve < 0.73). Whether the VA medical center, the community hospital, both, or neither had worse survival rates on simulated report cards than the university hospital depended on the prediction model used for adjustment.

CONCLUSIONS

Results of simulated outcome report cards for survival in patients with congestive heart failure depend on the method used to adjust for severity.

Enhancing performance in health care: a theoretical perspective on agency and the role of information

This paper examines the role of information in securing control of health care systems. The discussion focuses on the impact of the proposed 'Performance Framework', which entails a significant increase in the importance attached to formal performance indicators in the management of the UK National Health Service. The paper starts with a discussion of the role of performance data in securing organizational control within health care systems and summarizes recent research into the behavioural consequences of seeking to control health care agents using such information. A theoretical principal/agent model is then used to illustrate the incentives that exist for dysfunctional behaviour within health care when only imperfect information systems are available. The theoretical results are then examined in the context of a qualitative empirical study, which elicited the perceptions of managers and health care professionals connected with eight NHS hospitals. The study confirmed the existence and importance of serious dysfunctional consequences arising from the use of information as a means of control, and concludes that the Performance Framework will be successful only if it is used in careful conjunction with other means of control.

The STS National Database: current changes and challenges for the new millennium. Committee to Establish a National Database in Cardiothoracic Surgery, The Society of Thoracic Surgeons

BACKGROUND

The Society of Thoracic Surgeons (STS) established the National Database (NDB) for Cardiac Surgery in 1989. Since then it has grown to be the largest database of its kind in medicine. The NDB has been one of the pioneers in the analysis and reporting of risk-adjusted outcomes in cardiothoracic surgery.

METHODS AND RESULTS

This report explains the numerous changes in the NDB and its structure that have occurred over the past 2 years. It highlights the benefits of these changes, both to the individual member participants and to the STS overall. Additionally, the vision changes to the NDB and reporting structure are identified. The individuals who have participated in this effort since 1989 are acknowledged, and the STS owes an enormous debt of gratitude to each of them.

CONCLUSIONS

Because of their collective efforts, the goal to establish the STS NDB as a "gold standard" worldwide for process and outcomes analysis related to cardiothoracic surgery is becoming a reality.

Exploring the degree of concordance of coded and textual data in answering clinical queries from a clinical data repository

OBJECTIVE

To query a clinical data repository (CDR) for answers to clinical questions to determine whether different types of fields (coded and free text) would yield confirmatory, complementary, or conflicting information and to discuss the issues involved in producing the discrepancies between the fields.

METHODS

The appropriate data fields in a subset of a CDR (5,135 patient records) were searched for the answers to three questions related to surgical procedures. Each search included at least one coded data field and at least one free-text field. The identified free-text records were then searched manually to ensure correct interpretation. The fields were then compared to determine whether they agreed with each other, were supportive of each other, contained no entry (absence of data), or were contradictory.

RESULTS

The degree of concordance varied greatly according to the field and the question asked. Some fields were not granular enough to answer the question. The free-text fields often gave an answer that was not definitive. Absence of data was most logically interpreted in some cases as lack of completion of data and in others as a negative answer. Even with a question as specific as which side a hernia was on, contradictory data were found in 5 to 8 percent of the records.

CONCLUSIONS

Using the data in the CDR to answer clinical questions can yield significantly disparate results depending on the question and which data fields are searched. A database cannot just be queried in automated fashion and the results reported. Both coded and textual fields must be searched to obtain the fullest assessment. This can be expected to result in information that may be confirmatory, complementary, or conflicting. To yield the most accurate information possible, final answers to questions require human judgment and may require the gathering of additional information.

Measuring surgical outcome

A surgical outcome is any event or measure of a patient's health that may occur or change, for better or worse, as a result of an operation. The introduction of clinical governance has led to debate about how to monitor the quality of surgical services in the NHS. One method that has been proposed is the publication of data on surgical outcomes for named hospitals and surgeons (Dobson, 1998).

Influence of patient education on profiles of physician practices

BACKGROUND

Few data are available about the effect of patient socioeconomic status on profiles of physician practices.

OBJECTIVE

To determine the ways in which adjustment for patients' level of education (as a measure of socioeconomic status) changes profiles of physician practices.

DESIGN

Cross-sectional survey of patients in physician practices.

SETTING

Managed care organization in western New York State.

PARTICIPANTS

A random sample of 100 primary care physicians and 50 consecutive patients seen by each physician.

MEASUREMENTS

Ranks of physicians for patient physical and mental health (Short Form 12-Item Health Survey) and satisfaction (Patient Satisfaction Questionnaire), adjusted for patient age, sex, morbidity, and education.

RESULTS

Physicians whose patients had a lower mean level of education had significantly better ranks for patient physical and mental health status after adjustment for patients' level of education level than they did before adjustment (P < 0.001); this result was not seen for patient satisfaction. After adjustment for patients' level of education, each 1-year decrease in mean educational level was associated with a rank that improved by 8.1 (95% CI, 6.6 to 9.6) for patient physical health status and by 4.9 (CI, 3.9 to 5.9) for patient mental health status. Adjustment for education had similar effects for practices with more educated patients and those with less educated patients.

CONCLUSIONS

Profiles of physician practices that base ratings of physician performance on patients' physical and mental health status are substantially affected by patients' level of education. However, these results do not suggest that physicians who care for less educated patients provide worse care. Physician profiling should account for differences in patients' level of education.

Public reporting of surgical mortality: a survey of New York State cardiothoracic surgeons

BACKGROUND

Public disclosure of individual surgeons mortality following coronary artery bypass (CAB) is part of the New York State Department of Health Cardiac Surgery Reporting System (CSRS). The effects on the practice of cardiac surgery, as perceived by surgeons, remain unknown.

METHODS

All 150 New York State cardiac surgeons were sent an anonymous mail survey in 1997. Data was analyzed to determine the dominant opinion regarding the CSRS.

RESULTS

One hundred and four surgeons (69.3%) responded. The majority (70%) did not experience a change in practice. Data reporting was performed by the surgeon or an employee (58%). Many picked the incorrect definition of chronic obstructive pulmonary disease (COPD) (45%) or statistical method (60%). The aspect of CSRS most in need of improvement was gaming with risk factors (40%). Most surgeons (62%) refused to operate on at least one high-risk CAB patient over the prior year, primarily because of public reporting. Refusal was more common in surgeons in practice less than 10 years, those with less than 100 cases per year, and those with a mixed cardiothoracic practice (p < 0.05, Pearson's chi2 test). A significantly higher percentage of high-risk CAB patients were treated non-operatively, when compared with ascending aortic dissection patients (not disclosed) (p < 0.001, Wilcoxon signed ranks test).

CONCLUSIONS

The public disclosure of surgical results may be based on imperfect data and appears to have resulted in denial of surgical treatment to high-risk patients.

Intraoperative hemodynamic predictors of mortality, stroke, and myocardial infarction after coronary artery bypass surgery

Evidence that intraoperative hemodynamic abnormalities influence outcome is limited. The purpose of this study was to determine whether intraoperative hemodynamic abnormalities were associated with mortality, stroke, or perioperative myocardial infarction (PMI) in a large cohort of patients undergoing coronary artery bypass grafting. Risk factors and outcomes were queried from a state-mandated cardiac surgery reporting system at two hospitals in New York, NY. Intraoperative hemodynamic abnormalities were derived from computerized anesthesia records by assessing the duration of exposure to moderate or severe extremes of hemodynamic variables. Multivariate logistic regression identified independent predictors of perioperative mortality, stroke, and PMI. Among 2149 patients, there were 50 mortalities, 51 strokes, and 85 PMIs. In the precardiopulmonary bypass (pre-CPB) period, pulmonary hypertension was a predictor of mortality (odds ratio [OR] 2.1, P = 0.029), and bradycardia and tachycardia were predictors of PMI (OR 2.9, P = 0.007 and OR 2.0, P = 0.028, respectively). During CPB, hypotension was a predictor of mortality (OR 1.3, P = 0.025). Post-CPB, tachycardia was a predictor of mortality (OR 3.1, P = 0.001), diastolic arterial hypertension was a predictor of stroke (OR 5.4, P = 0.012), and pulmonary hypertension was a predictor of PMI (OR 7.0, P < 0.001). Increased pulmonary arterial diastolic pressure post-CPB was a predictor of mortality (OR 1.2, P = 0.004), stroke (OR 3.9, P = 0.002), and PMI (OR 2.2, P = 0.001). Rapid intraoperative variations in blood pressure and heart rate were not independent predictors of these outcomes. These findings demonstrate the prognostic significance of intraoperative hemodynamic abnormalities, including data from pulmonary artery catheterization, to adverse postoperative outcomes. It is not known whether interventions to control these variables would improve outcome.

IMPLICATIONS

Intraoperative hemodynamic abnormalities, including pulmonary hypertension, hypotension during cardiopulmonary bypass, and postcardiopulmonary bypass pulmonary diastolic hypertension, were independently associated with mortality, stroke, and perioperative myocardial infarction over and above the effects of other preoperative risk factors.

Validation of risk adjustment models for in-hospital percutaneous transluminal coronary angioplasty mortality on an independent data set

OBJECTIVES

We sought to validate recently proposed risk adjustment models for in-hospital percutaneous transluminal coronary angioplasty (PTCA) mortality on an independent data set of high risk patients undergoing PTCA.

BACKGROUND

Risk adjustment models for PTCA mortality have recently been reported, but external validation on independent data sets and on high risk patient groups is lacking.

METHODS

Between July 1, 1994 and June 1, 1996, 1,476 consecutive procedures were performed on a high risk patient group characterized by a high incidence of cardiogenic shock (3.3%) and acute myocardial infarction (14.3%). Predictors of in-hospital mortality were identified using multivariate logistic regression analysis. Two external models of in-hospital mortality, one developed by the Northern New England Cardiovascular Disease Study Group (model NNE) and the other by the Cleveland Clinic (model CC), were compared using receiver operating characteristic (ROC) curve analysis.

RESULTS

In this patient group, an overall in-hospital mortality rate of 3.4% was observed. Multivariate regression analysis identified risk factors for death in the hospital that were similar to the risk factors identified by the two external models. When fitted to the data set, both external models had an area under the ROC curve >0.85, indicating overall excellent model discrimination, and both models were accurate in predicting mortality in different patient subgroups. There was a trend toward a greater ability to predict mortality for model NNE as compared with model CC, but the difference was not significant.

CONCLUSIONS

Predictive models for PTCA mortality yield comparable results when applied to patient groups other than the one on which the original model was developed. The accuracy of the two models tested in adjusting for the relatively high mortality rate observed in this patient group supports their application in quality assessment or quality improvement efforts.