Report cards on cardiac surgeons. Assessing New York State's approach

Do bad report cards have consequences? Impacts of publicly reported provider quality information on the CABG market in Pennsylvania

Since 1992, the Pennsylvania Health Care Cost Containment Council (PHC4) has published cardiac care report cards for coronary artery bypass graft (CABG) surgery providers. We examine the impact of CABG report cards on a provider's aggregate volume and volume by patient severity and then employ a mixed logit model to investigate the matching between patients and providers. We find a reduction in volume of poor performing and unrated surgeons' volume but no effect on more highly rated surgeons or hospitals of any rating. We also find that the probability that patients, regardless of severity of illness, receive CABG surgery from low-performing surgeons is significantly lower.

'Bench to behavior': translating comparative effectiveness research into improved clinical practice

The new national emphasis on comparative effectiveness research is likely to generate an unprecedented volume of new findings. It is essential to anticipate the obstacles that front-line health care professionals will face in translating these results into better clinical decision making. We review the current barriers to the dissemination of evidence-based clinical recommendations, including problems with continuing medical education, provider incentives, and quality assurance. We then propose solutions, including more effective educational outreach programs, requirements for practitioners to master important findings, and alignment of incentives to encourage evidence-based practice. Such strategies can lead to policies that could encourage the uptake of new comparative effectiveness data and encourage their translation into better clinical practice.

Changes in patient sorting to nursing homes under public reporting: improved patient matching or provider gaming?

OBJECTIVE

To test whether public reporting in the setting of postacute care in nursing homes results in changes in patient sorting.

DATA SOURCES/STUDY SETTING

All postacute care admissions from 2001 to 2003 in the nursing home Minimum Data Set.

STUDY DESIGN

We test changes in patient sorting (or the changes in the illness severity of patients going to high- versus low-scoring facilities) when public reporting was initiated in nursing homes in 2002. We test for changes in sorting with respect to pain, delirium, and walking and then examine the potential roles of cream skimming and downcoding in changes in patient sorting. We use a difference-in-differences framework, taking advantage of the variation in the launch of public reporting in pilot and nonpilot states, to control for underlying trends in patient sorting.

PRINCIPAL FINDINGS

There was a significant change in patient sorting with respect to pain after public reporting was initiated, with high-risk patients being more likely to go to high-scoring facilities and low-risk patients more likely to go to low-scoring facilities. There was also an overall decrease in patient risk of pain with the launch of public reporting, which may be consistent with changes in documentation of pain levels (or downcoding). There was no significant change in sorting for delirium or walking.

CONCLUSIONS

Public reporting of nursing home quality improves matching of high-risk patients to high-quality facilities. However, efforts should be made to reduce the incentives for downcoding by nursing facilities.

The Hospital Compare mortality model and the volume-outcome relationship

OBJECTIVE

We ask whether Medicare's Hospital Compare random effects model correctly assesses acute myocardial infarction (AMI) hospital mortality rates when there is a volume-outcome relationship.

DATA SOURCES/STUDY SETTING

Medicare claims on 208,157 AMI patients admitted in 3,629 acute care hospitals throughout the United States.

STUDY DESIGN

We compared average-adjusted mortality using logistic regression with average adjusted mortality based on the Hospital Compare random effects model. We then fit random effects models with the same patient variables as in Medicare's Hospital Compare mortality model but also included terms for hospital Medicare AMI volume and another model that additionally included other hospital characteristics.

PRINCIPAL FINDINGS

Hospital Compare's average adjusted mortality significantly underestimates average observed death rates in small volume hospitals. Placing hospital volume in the Hospital Compare model significantly improved predictions.

CONCLUSIONS

The Hospital Compare random effects model underestimates the typically poorer performance of low-volume hospitals. Placing hospital volume in the Hospital Compare model, and possibly other important hospital characteristics, appears indicated when using a random effects model to predict outcomes. Care must be taken to insure the proper method of reporting such models, especially if hospital characteristics are included in the random effects model.

The National Bowel Cancer Audit: the risks and benefits of moving to open reporting of clinical outcomes

BACKGROUND

The government's proposals to openly report clinical outcomes poses challenges to the National Bowel Cancer Audit now funded by the UK department of health.

AIM

To identify the benefits and risks of open reporting and to propose ways the risks might be minimized.

METHODS

A review of the literature on clinical audit and the consequences of open reporting.

RESULTS

There are significant potential benefits of a national audit of bowel cancer including protecting patients from sub-standard care, providing clinicians with externally validated evidence of their performance, outcome data for clinical governance and evidence that increases in government expenditure are achieving improvements in survival from bowel cancer. These benefits will only be achieved if the audit captures most of the cases of bowel cancer in the UK, the data collected is complete and accurate, the results are risk adjusted and these are presented to the public in a way that is fair, clear and understandable. Involvement of clinicians who have confidence in the results of the audit and who actively compare their own results against a national standard is essential. It is suggested that a staged move to open reporting should minimise the risk of falsely identifying an outlying unit.

CONCLUSION

The fundamental aim of the National Bowel Cancer Audit is the pursuit of excellence by identification and adoption of best practice. This could achieve a continuous improvement in the care of all patients with bowel cancer in the UK. The ACPGBI suggests a safer way of transition to open reporting to avoid at least some of its pitfalls.

Risk-prediction models for mortality after coronary artery bypass surgery: application to individual patients

INTRODUCTION

We derived a risk-assessment model for predicting mortality after coronary artery bypass surgery from patient data from the 1990s and examined the model's accuracy in predicting early mortality in more contemporary patients. We also examined the accuracy of a completely new model and an older model recalibrated with newer data.

MATERIALS AND METHODS

Three mortality-prediction models were derived: an "old" model from 8959 patients treated during 1993-1999, a "new" model from 5281 patients treated during 2000-2004, and a version of the old model "recalibrated" with the 2000-2004 data. Each model's discriminatory ability was assessed by computing area under receiver-operator characteristic (ROC) curves, and precision was estimated by comparing observed and predicted mortality rates. To test the temporal applicability of the old model, we applied it to the 2000-2004 data and to data from 1879 patients operated on during 2005-2007. To compare the recalibration and remodeling strategies, the new and recalibrated models were applied to the 2005-2007 data.

RESULTS

The old model showed good discrimination (ROC, 0.80) and concordance between observed and predicted mortality for the 2000-2004 patients but overpredicted mortality for the 2005-2007 patients. The new and recalibrated models had good discriminatory ability (ROC, 0.81 and 0.79) and showed similarly good concordance between observed and predicted mortality when applied to the 2005-2007 data.

CONCLUSIONS

Predictive models for mortality after cardiac surgery lose precision within a few years after development. Recalibrating old models and creating new models (i.e., remodeling) are equally good strategies for predicting outcomes in contemporary patient cohorts.

Comparing and ranking hospitals based on outcome: results from The Netherlands Stroke Survey

BACKGROUND

Measuring quality of care and ranking hospitals with outcome measures poses two major methodological challenges: case-mix adjustment and variation that exists by chance.

AIM

To compare methods for comparing and ranking hospitals that considers these.

METHODS

The Netherlands Stroke Survey was conducted in 10 hospitals in the Netherlands, between October 2002 and May 2003, with prospective and consecutive enrollment of patients with acute brain ischaemia. Poor outcome was defined as death or disability after 1 year (modified Rankin scale of > or =3). We calculated fixed and random hospital effects on poor outcome, unadjusted and adjusted for patient characteristics. We compared the hospitals using the expected rank, a novel statistical measure incorporating the magnitude and the uncertainty of differences in outcome.

RESULTS

At 1 year after stroke, 268 of the total 505 patients (53%) had a poor outcome. There were substantial differences in outcome between hospitals in unadjusted analysis (chi(2) = 48, 9 df, P < 0.0001). Adjustment for 12 confounders led to halving of the chi(2) (chi(2) = 24). The same pattern was observed in random effects analysis. Estimated performance of individual hospitals changed considerably between unadjusted and adjusted analysis. Further changes were seen with random effect estimation, especially for smaller hospitals. Ordering by expected rank led to shrinkage of the original ranks of 1-10 towards the median rank of 5.5 and to a different order of the hospitals, compared to ranking based on fixed effects.

CONCLUSION

In comparing and ranking hospitals, case-mix-adjusted random effect estimates and the expected ranks are more robust alternatives to traditional fixed effect estimates and simple rankings.

Impact of public reporting on quality of postacute care

OBJECTIVE

Evidence supporting the use of public reporting of quality information to improve health care quality is mixed. While public reporting may improve reported quality, its effect on quality of care more broadly is uncertain. This study tests whether public reporting in the setting of nursing homes resulted in improvement of reported and broader but unreported quality of postacute care.

DATA SOURCES/STUDY SETTING

1999-2005 nursing home Minimum Data Set and inpatient Medicare claims.

STUDY DESIGN

We examined changes in postacute care quality in U.S. nursing homes in response to the initiation of public reporting on the Centers for Medicare and Medicaid Services website, Nursing Home Compare. We used small nursing homes that were not subject to public reporting as a contemporaneous control and also controlled for patient selection into nursing homes. Postacute care quality was measured using three publicly reported clinical quality measures and 30-day potentially preventable rehospitalization rates, an unreported measure of quality.

PRINCIPAL FINDINGS

Reported quality of postacute care improved after the initiation of public reporting for two of the three reported quality measures used in Nursing Home Compare. However, rates of potentially preventable rehospitalization did not significantly improve and, in some cases, worsened.

CONCLUSIONS

Public reporting of nursing home quality was associated with an improvement in most postacute care performance measures but not in the broader measure of rehospitalization.

Incorporating natural variation into IVF clinic league tables: The Expected Rank

Background

Rankings based on outcome are often used to present health care provider performance. These rankings do however not reflect that part of the variation in outcome between providers is caused by natural variation, and not by any differences in quality of care. The aim of this study is to compare standard methods for ranking with a novel method that takes into account natural variation.

Methods

We used data on the number of treatment cycles and the number of pregnancies of 13 Dutch IVF clinics from 2004. We calculated the Expected Rank (ER), an estimate of the true rank of a provider, accounting for natural variation. We rescaled the ER to obtain the Percentile based on ER (PCER), that can be interpreted as the probability that a clinic is worse than a randomly selected other clinic. We also calculated a measure for rankability ρ, which is the part of variation between providers that is due to true differences (as opposed to natural variation).

Results

The expected ranks ranged from 1.4 to 11.9 instead of the original ranks 1–13. The ER showed that some clinics performed very similar, which would be disregarded when using standard ranks. The PCER ranged from 7% to 88%. Rankability was substantial (ρ = 0.9)

Conclusion

The Expected Rank provides a way to combine the attractiveness of a ranking, a single number and easy interpretation, with reliable analyses that does justice to the providers, and also allows individual comparisons.

Variability of interpretive accuracy among diagnostic mammography facilities

BACKGROUND

Interpretive performance of screening mammography varies substantially by facility, but performance of diagnostic interpretation has not been studied.

METHODS

Facilities performing diagnostic mammography within three registries of the Breast Cancer Surveillance Consortium were surveyed about their structure, organization, and interpretive processes. Performance measurements (false-positive rate, sensitivity, and likelihood of cancer among women referred for biopsy [positive predictive value of biopsy recommendation {PPV2}]) from January 1, 1998, through December 31, 2005, were prospectively measured. Logistic regression and receiver operating characteristic (ROC) curve analyses, adjusted for patient and radiologist characteristics, were used to assess the association between facility characteristics and interpretive performance. All statistical tests were two-sided.

RESULTS

Forty-five of the 53 facilities completed a facility survey (85% response rate), and 32 of the 45 facilities performed diagnostic mammography. The analyses included 28 100 diagnostic mammograms performed as an evaluation of a breast problem, and data were available for 118 radiologists who interpreted diagnostic mammograms at the facilities. Performance measurements demonstrated statistically significant interpretive variability among facilities (sensitivity, P = .006; false-positive rate, P < .001; and PPV2, P < .001) in unadjusted analyses. However, after adjustment for patient and radiologist characteristics, only false-positive rate variation remained statistically significant and facility traits associated with performance measures changed (false-positive rate = 6.5%, 95% confidence interval [CI] = 5.5% to 7.4%; sensitivity = 73.5%, 95% CI = 67.1% to 79.9%; and PPV2 = 33.8%, 95% CI = 29.1% to 38.5%). Facilities reporting that concern about malpractice had moderately or greatly increased diagnostic examination recommendations at the facility had a higher false-positive rate (odds ratio [OR] = 1.48, 95% CI = 1.09 to 2.01) and a non-statistically significantly higher sensitivity (OR = 1.74, 95% CI = 0.94 to 3.23). Facilities offering specialized interventional services had a non-statistically significantly higher false-positive rate (OR = 1.97, 95% CI = 0.94 to 4.1). No characteristics were associated with overall accuracy by ROC curve analyses.

CONCLUSIONS

Variation in diagnostic mammography interpretation exists across facilities. Failure to adjust for patient characteristics when comparing facility performance could lead to erroneous conclusions. Malpractice concerns are associated with interpretive performance.

Publicly reported provider outcomes: the concerns of cardiac surgeons in a single-payer system

BACKGROUND

Provider outcomes reports are an important part of quality improvement efforts. The positive and negative impact of such reports on the delivery of care has not been extensively explored.

METHODS

A survey of Ontario cardiac surgeons was performed in September 2003 to understand their concerns regarding performance reports. The questionnaire addressed the use of evidence-based practices, the impact of public-provider profiling on clinical practice and the improvement of current report cards. The survey was conducted with the distribution of a fiscal 2000/2001 cardiac surgery report card.

RESULTS

There was a 95% (52 of 55 cardiac surgeons) survey response rate, of which 80% were high-volume surgeons with a case volume of more than 200 cases per year. Seventy-four per cent of surgeons had more than five years of experience. The majority of surgeons believed that performance reports influenced cardiologist referrals (84%) and patient choices (80%). A minority (48%) of surgeons believed that the reporting of inhospital mortality was very or extremely useful, but a majority (83%) believed mortality rates indicated the relative performance of a cardiac surgeon. The majority of surgeons believed that routine upcoding of data (84%) and inadequate risk adjustment (75%) were weaknesses of present performance reports. Surgeons were divided regarding whether the institutional performance should continue to be publicly reported (51% agreed with public reporting).

CONCLUSIONS

In a single-payer system, performance reports breed provider concerns similar to those seen in market-driven systems including high-risk patient avoidance and upcoding of data. Regardless, providers recognize that institutional performance reports, irrespective of public or confidential reporting, are important in continuous quality improvement.

Regulation, measurements and incentives. The experience in the US and UK: does context matter?

Since the 1990s the US and UK healthcare systems have increasingly sought to measure the quality of services in order to stimulate improvement and publish the results. Despite the very different healthcare systems, there are some striking similarities in the results of these schemes. In both countries information that creates the threat of reputational damage and the possibility of gaining kudos is particularly effective in stimulating a response. However, these responses may be perverse: gaming, falsifying of data and measurement fixation have all been uncovered. In the UK context, information is closely aligned to the regulatory and performance management framework, and there should be a role for the new Care Quality Commission in the emerging 'information landscape' of the NHS in England.

Hitting and missing targets by ambulance services for emergency calls: effects of different systems of performance measurement within the UK

Following devolution, differences developed between UK countries in systems of measuring performance against a common target that ambulance services ought to respond to 75% of calls for what may be immediately life threatening emergencies (category A calls) within 8 minutes. Only in England was this target integral to a ranking system of 'star rating', which inflicted reputational damage on services that failed to hit targets, and only in England has this target been met. In other countries, the target has been missed by such large margins that services would have been publicly reported as failing, if they had been covered by the English system of star ratings. The paper argues that this case-study adds to evidence from comparisons of different systems of hospital performance measurement that, to have an effect, these systems need to be designed to inflict reputational damage on those that have performed poorly; and it explores implications of this hypothesis. The paper also asks questions about the adequacy of systems of performance measurement of ambulance services in UK countries.

Impact of public reporting on unreported quality of care

OBJECTIVE

The impact of quality improvement incentives on nontargeted care is unknown and some have expressed concern that such incentives may be harmful to nontargeted areas of care. Our objective is to examine the effect of publicly reporting quality information on unreported quality of care.

DATA SOURCES/STUDY SETTING

The nursing home Minimum Data Set from 1999 to 2005 on all postacute care admissions.

STUDY DESIGN

We studied 13,683 skilled nursing facilities and examined how unreported aspects of clinical care changed in response to changes in reported care after public reporting was initiated by the Centers for Medicare and Medicaid Services on their website, Nursing Home Compare, in 2002.

PRINCIPAL FINDINGS

We find that overall both unreported and reported care improved following the launch of public reporting. Improvements in unreported care were particularly large among facilities with high scores or that significantly improved on reported measures, whereas low-scoring facilities experienced no change or worsening of their unreported quality of care.

CONCLUSIONS

Public reporting in the setting of postacute care had mixed effects on areas without public reporting, improving in high-ranking facilities, but worsening in low-ranking facilities. While the benefits of public reporting may extend beyond areas that are being directly measured, these initiatives may also widen the gap between high- and low-quality facilities.

Public Health Reporting: The United States Perspective

The release of 2 landmark reports by the Institute of Medicine titled, “To Err Is Human: Building a Safer Health System” and “Crossing the Quality Chasm” were instrumental in the identification of safety and quality issues. Since their release, federal and state programs of public reporting of performance measures have attempted to close the quality gap of care that is inappropriate, not timely, or lacking an evidence base. Cardiac surgery has long been the focus of public scrutiny, and now, as we move from an era of managed care to public reporting, reimbursement for cardiac surgery procedures will be tied to performance. However, the question is whether public reporting and pay for performance will ultimately improve the quality of patient care, safety, and provide the consumer with enough information to make surgeon and institutional choices. Will the cost and focus of achieving perfection with performance standards overshadow any real improvement in clinical outcomes?

The association of Nursing Home Compare quality measures with market competition and occupancy rates

Since 2002, the Centers for Medicare and Medicaid Services have reported quality measures on the Nursing Home Compare Web site. It has been assumed that nursing homes are able to make improvements on these measures. In this study researchers examined nursing homes to see whether they have improved their quality scores, after accounting for regression to the mean. Researchers also examined whether gains varied according to market competition or market occupancy rates. They identified some regression to the mean for the quality measure scores over time; nevertheless, they also determined that some nursing homes had indeed made small improvements in their quality measure scores. As would be predicted based on the market-driven mechanism underlying quality improvements using report cards, the greatest improvements occurred in the most competitive markets and in those with the Lowest average occupancy rates. As policies to promote more competition in Long-term care proceed, further reducing occupancy rates, further, albeit small, quality gains will likely be made in the future.

Impact of the New York State Cardiac Surgery and Percutaneous Coronary Intervention Reporting System on the management of patients with acute myocardial infarction complicated by cardiogenic shock

BACKGROUND

Studies suggest that the New York State Cardiac Surgery and Percutaneous Coronary Intervention Reporting System, which makes public the operator-specific mortality for patients undergoing coronary artery bypass graft surgery (CABG) or percutaneous coronary intervention (PCI), may deter operators from providing revascularization to high-risk cardiac patients in New York compared to other states.

METHODS

We performed a retrospective analysis of 545 US patients with acute myocardial infarction and cardiogenic shock due to predominant left ventricular failure enrolled in the SHOCK Registry. Adjusting for case mix using a propensity score method, we compared the use of coronary angiography, PCI, CABG, and outcomes between 220 patients in New York and 325 in other states.

RESULTS

New York patients were older with similar or less severe baseline characteristics. After propensity score adjustment, New York patients were less likely than non-New York patients to undergo coronary angiography (odds ratio 0.46, 95% CI 0.31-0.68, P < .001) and PCI (odds ratio 0.51, 95% CI 0.33-0.77, P = .002). Coronary artery bypass graft rates were similarly low (14.1% vs 15.1%, P = not significant), but New York patients waited significantly longer after shock onset for surgery (101.2 vs 10.3 hours, P < .001) with only 32.3% of New York patients vs 75.5% of non-New York patients (P < .001) taken for CABG within 3 days of shock onset.

CONCLUSIONS

In our propensity-adjusted retrospective analysis, New York patients with acute myocardial infarction and cardiogenic shock were less likely to undergo coronary angiography and PCI and waited significantly longer to receive CABG than their non-New York counterparts. These findings suggest that state-required reporting to the New York State Cardiac Surgery and Percutaneous Coronary Intervention Reporting System may result in the reluctance to revascularize the highest-risk cardiac patients.

Clinical and nonclinical indicators of performance in thoracic surgery

One of the most important steps in the entire process of monitoring and improving quality of care is to identify the proper quality measures. This may be challenging from the outset since no single indicator can fully comprehend the entire concept of quality of care, which is multidimensional by nature. Ideally, multiple indicators should be used at the same time to obtain a more precise assessment of the quality of care. The quality of care can be measured by observing its structure, its processes, and its outcomes. Each indicator may reflect different aspects of quality and may be of particular interest to different audiences (providers, consumers, regulators, purchasers). The selection of one or the other may depend on the objectives of the analysis and the target audience. Although outcomes represent the ultimate product of health care, if the focus is on identifying and remedying apparent variations in performance, it is often preferable to measure not only outcomes but also the desirable processes of care. From a performance management perspective, the key issue is that a desirable process should be unambiguously associated with improved patient health outcomes. Monitoring the process can then be a substitute for measuring the outcome. Unlike outcome indicators, process measures have the potential to identify for clinicians exactly which processes they followed or did not follow that had the potential to affect patient outcomes. Process indicators provide information that is actionable. Finally, thoracic surgeons should take the lead in the managerial approach to the evaluation of performance, preventing administrative personnel unfamiliar with our multifaceted clinical world from judging our practice through imprecise instruments. We, as physicians, must absolutely improve our skill and confidence in risk analysis, outcome-evaluation methods, and process-based assessment of our practice.

Risk-adjusted sequential probability ratio test control chart methods for monitoring operator and institutional mortality rates in interventional cardiology

BACKGROUND

The objective of this study was to evaluate risk-adjusted sequential probability ratio test control charts for the detection of significant discrepancies between institution or individual interventional cardiologist postprocedural mortality rates and national or local event rate expectations.

METHODS

Eight thousand nine hundred forty-two percutaneous coronary interventional procedures were performed by 27 operators between January 1, 2002, and November 30, 2006. The institution-based evaluation included all procedures, and the individual-based evaluations included 8750 procedures performed by 18 operators who had each done at least 100 PCI procedures. Risk-adjusted sequential probability ratio test control charts were developed to assess whether the odds ratios (ORs) for death were >2.0 for alpha = beta = 0.10. The American College of Cardiology 1.1 prediction model was used to risk-adjust both the institution and individuals, and an additional local model was used for individuals.

RESULTS

After national risk adjustment, the local institution did not show mortality of more than a 1.5 OR. Two operators had a >2.0 mortality OR after national risk adjustment, and one of those remained elevated after local risk adjustment. Of 18 operators, 10 had insufficient data to allow us to accept or reject the hypothesis of increased risk.

CONCLUSIONS

The local institution performed within national expectations, but 1 operator was identified as having poor performance, which prompted an in-depth review of that operator's cases. The review revealed that the operator had an unusually high number of patients who presented with risk factors not included in the risk-adjustment models. This study highlights the utility of risk-adjusted sequential probability ratio test as a method for outcomes monitoring and quality control in interventional cardiology.

Demonstrating safety through in-hospital mortality analysis following elective abdominal aortic aneurysm repair in England

Background

The aims were to assess the evidence that individual hospitals had mortality rates in excess of the national average after abdominal aortic aneurysm (AAA) repair and to develop an effective method for monitoring mortality using local data.

Methods

Hospital Episode Statistics identified patients undergoing elective infrarenal AAA repair. A technique was developed that compared individual hospital mortality rates with the mortality rate in the remainder of England. The strength of evidence that the death rate was less than elsewhere, and less than twice elsewhere, was quantified using a test of statistical significance. A moving average chart technique was devised using local data for mortality monitoring and comparison to the national average.

Results

For 30 hospitals, the mortality rate was significantly greater than elsewhere, and in three hospitals it was demonstrably greater than twice that in the remainder of England. The moving average chart appeared to provide a useful technique for local mortality monitoring.

Conclusion

Different mortality rates exist for AAA repair within England. Mortality can be monitored locally and compared with the national average.

From trees to wood and back: perspective on clinical data analysis in thoracic surgery

General thoracic surgery has been slow to embrace advanced statistical techniques to assess outcome and direct future therapy to individual patients (the "trees") based on aggregating data and discovering informative patterns in them (the "wood"). This is due in part to the nature and complexity of the practice. However, methods exist that deal with problems of small numbers, variable case volume, and heterogeneity of patients, diseases, and therapies. For effective analysis of these multiple variables, modern strategies must be used and pitfalls avoided. For generating new knowledge about appropriateness of therapy, the focus of analyses must include long-term outcome. Quality improvement efforts may be assisted by in-depth analysis of short-term outcome from a select number of centers to inform future universal efforts. Ultimately, these results need to feed back to improved management of the individual patient.

Transferring critically ill patients out of hospital improves the standardized mortality ratio: a simulation study

BACKGROUND

Transferring critically ill patients to other acute care hospitals may artificially impact benchmarking measures. We sought to quantify the effect of out-of-hospital transfers on the standardized mortality ratio (SMR), an outcome-based measure of ICU performance.

METHODS

We performed a cohort study and Monte Carlo simulation using data from 85 ICUs participating in the acute physiology and chronic health evaluation (APACHE) clinical information system from 2002 to 2003. The SMR (observed divided by expected hospital mortality) was calculated for each ICU using APACHE IV risk adjustment. A set number of patients was randomly assigned to be transferred out alive rather than experience their original outcome. The SMR was recalculated, and the mean simulated SMR was compared to the original.

RESULTS

The mean (+/- SD) baseline SMR was 1.06 +/- 0.19. In the simulation, increasing the number of transfers by 2% and 6% over baseline decreased the SMR by 0.10 +/- 0.03 and 0.14 +/- 0.03, respectively. At a 2% increase, 27 ICUs had a decrease in SMR of > 0.10, and two ICUs had a decrease in SMR of > 0.20. Transferring only one additional patient per month was enough to create a bias of > 0.1 in 27 ICUs.

CONCLUSIONS

Increasing the number of acute care transfers by a small amount can significantly bias the SMR, leading to incorrect inference about ICU quality. Sensitivity to the variation in hospital discharge practices greatly limits the use of the SMR as a quality measure.

Comparison of hospital episode statistics and central cardiac audit database in public reporting of congenital heart surgery mortality

OBJECTIVE

To verify or refute the value of hospital episode statistics (HES) in determining 30 day mortality after open congenital cardiac surgery in infants nationally in comparison with central cardiac audit database (CCAD) information.

DESIGN

External review of paediatric cardiac surgical outcomes in England (HES) and all UK units (CCAD), as derived from each database.

SETTING

Congenital heart surgery centres in the United Kingdom.

DATA SOURCES

HES for congenital heart surgery and corresponding information from CCAD for the period 1 April 2000 to 31 March 2002. HES was restricted to the 11 English centres; CCAD covered all 13 UK centres.

MAIN OUTCOME MEASURE

Mortality within 30 days of open heart surgery in infants aged under 12 months.

RESULTS

In a direct comparison for the years when data from the 11 English centres were available from both databases, HES omitted between 5% and 38% of infants operated on in each centre. A median 40% (range 0-73%) shortfall occurred in identification of deaths by HES. As a result, mean 30 day mortality was underestimated at 4% by HES as compared with 8% for CCAD. In CCAD, between 1% and 23% of outcomes were missing in nine of 11 English centres used in the comparison (predominantly those for overseas patients). Accordingly, CCAD mortality could also be underestimated. Oxford provided the most complete dataset to HES, including all deaths recorded by CCAD. From three years of CCAD, Oxford's infant mortality from open cardiac surgery (10%) was not statistically different from the mean for all 13 UK centres (8%), in marked contrast to the conclusions drawn from HES for two of those years.

CONCLUSIONS

Hospital episode statistics are unsatisfactory for the assessment of activity and outcomes in congenital heart surgery. The central cardiac audit database is more accurate and complete, but further work is needed to achieve fully comprehensive risk stratified mortality data. Given unresolved limitations in data quality, commercial organisations should reconsider placing centre specific or surgeon specific mortality data in the public domain.

Using Automated Clinical Data for Risk Adjustment

BACKGROUND

Clinically plausible risk-adjustment methods are needed to implement pay-for-performance protocols. Because billing data lacks clinical precision, may be gamed, and chart abstraction is costly, we sought to develop predictive models for mortality that maximally used automated laboratory data and intentionally minimized the use of administrative data (Laboratory Models). We also evaluated the additional value of vital signs and altered mental status (Full Models).

METHODS

Six models predicting in-hospital mortality for ischemic and hemorrhagic stroke, pneumonia, myocardial infarction, heart failure, and septicemia were derived from 194,903 admissions in 2000-2003 across 71 hospitals that imported laboratory data. Demographics, admission-based labs, International Classification of Diseases (ICD)-9 variables, vital signs, and altered mental status were sequentially entered as covariates. Models were validated using abstractions (629,490 admissions) from 195 hospitals. Finally, we constructed hierarchical models to compare hospital performance using the Laboratory Models and the Full Models.

RESULTS

Model c-statistics ranged from 0.81 to 0.89. As constructed, laboratory findings contributed more to the prediction of death compared with any other risk factor characteristic groups across most models except for stroke, where altered mental status was more important. Laboratory variables were between 2 and 67 times more important in predicting mortality than ICD-9 variables. The hospital-level risk-standardized mortality rates derived from the Laboratory Models were highly correlated with the results derived from the Full Models (average rho = 0.92).

CONCLUSIONS

Mortality can be well predicted using models that maximize reliance on objective pathophysiologic variables whereas minimizing input from billing data. Such models should be less susceptible to the vagaries of billing information and inexpensive to implement.

AAA benchmarking by Dr Foster: a cause for concern?

INTRODUCTION

The publication of interpretable performance data for hospitals is an important service. In November 2002, the medical benchmarking company Dr Foster published a league table based on the results of abdominal aortic aneurysm (AAA) repair. The purpose of our study was to establish the validity of the data used in benchmarking.

PATIENTS AND METHODS

Data on elective infra-renal AAA (IRAAA) repair was obtained from three sources. Data used by Dr Foster was based on the hospital PAS system. The databases for both Dr Foster and PAS were analysed and cross-referenced to the vascular unit database maintained by a separately employed audit co-ordinator.

RESULTS

Of 395 total aortic aneurysm repairs, 223 (56%) were identified as elective IRAAA repairs on the unit database. Of these, 125 were identified on the PAS database and 115 on the Dr Foster database. The number of deaths was the same in both the unit and Dr Foster databases (n = 11) but the Dr Foster database included deaths in patients who had undergone juxtarenal (n = 1), Type III TAAA (n = 2) and Type IV TAAA (n = 4) repairs and omitted 7 deaths following IRAAA. The sensitivity and specificity for the PAS dataset was 0.51 and 0.93, respectively. For Dr Foster, the results were worse with a sensitivity and specificity of 0.41 and 0.86, respectively. The accuracy of the data was 0.6 and 0.69 for Dr Foster and PAS, respectively. Standardised mortality ratios (SMRs) were used to rank hospitals. Dr Foster's published SMR for elective AAA repair for our unit was 160. The actual SMR was 67.

CONCLUSIONS

Robust and accurate published league tables should be supported and commended but currently available data appear to be misleading and may cause unnecessary concern to patients.

Comparison of Clinical and Administrative Data Sources for Hospital Coronary Artery Bypass Graft Surgery Report Cards

Background— Regardless of statistical methodology, public performance report cards must use the highest-quality validated data, preferably from a prospectively maintained clinical database. Using logistic regression and hierarchical models, we compared hospital cardiac surgery profiling results based on clinical data with those derived from contemporaneous administrative data.

Methods and Results— Fiscal year 2003 isolated coronary artery bypass grafting surgery results based on an audited and validated Massachusetts clinical registry were compared with those derived from a contemporaneous state administrative database, the latter using the inclusion/exclusion criteria and risk model of the Agency for Healthcare Research and Quality. There was a 27.4% disparity in isolated coronary artery bypass grafting surgery volume (4440 clinical, 5657 administrative), a 0.83% difference in observed in-hospital mortality (2.05% versus 2.88%), corresponding differences in risk-adjusted mortality calculated by various statistical methodologies, and 1 hospital classified as an outlier only with the administrative data–based approach. The discrepancies in volumes and risk-adjusted mortality were most notable for higher-volume programs that presumably perform a higher proportion of combined procedures that were misclassified as isolated coronary artery bypass grafting surgery in the administrative cohort. Subsequent analyses of a patient cohort common to both databases revealed the smoothing effect of hierarchical models, a 9% relative difference in mortality (2.21% versus 2.03%) resulting from nonstandardized mortality end points, and 1 hospital classified as an outlier using logistic regression but not using hierarchical regression.

Conclusions— Cardiac surgery report cards using administrative data are problematic compared with those derived from audited and validated clinical data, primarily because of case misclassification and nonstandardized end points.

Incorporating natural variation into IVF clinic league tables

BACKGROUND

More and more league tables are being published every day to rate the performance of health boards, hospitals and surgeons. However, they do not show the magnitude of uncertainty caused by natural variation.

METHODS

We propose a new method to present league tables in which the ratings are easy to interpret. Instead of just giving one score, we suggest the addition of best-case scenario and worst-case scenario scores. The true performance of a clinic, accounting for natural variation, is most likely to be between the best-case scenario and the worst-case scenario for its rating. These ratings can be computed easily, without any special software.

RESULTS

We illustrate our method based on data of Dutch IVF clinics from 2004. Six (out of 13) clinics shared a 'top of the league' position when considering the best-case scenario.

CONCLUSION

There is great uncertainty about the ratings. To show the magnitude of uncertainty, league tables should include the best-case scenario and the worst-case scenario ratings of each clinic.

The impact of unmeasured clinical variables on the accuracy of hospital report cards: a Monte Carlo study

PURPOSE

Hospital report cards are commonly produced using administrative data. The objective of this study was to determine the impact of unmeasured clinical data on the accuracy of hospitals' report cards.

METHODS

Monte Carlo simulations were based on both administrative and detailed clinical data for patients hospitalized with an acute myocardial infarction in Ontario, Canada. Data were simulated such that the true performance of each hospital was known. Both clinical and administrative risk scores were randomly generated for each patient. The ability of hospital report cards to correctly identify hospitals that truly had higher than acceptable mortality was compared when both clinical and administrative data were used and when only administrative data were used. By using Monte Carlo simulations, we were able to incrementally increase the divergence between the 2 risk scores.

RESULTS

In a wide range of settings, sensitivity and specificity of hospital report cards was only negligibly greater when both administrative and clinical data were used compared to when only administrative data were used.

CONCLUSIONS

Unmeasured clinical data have at most a minor impact on the accuracy of cardiac hospital report cards.

Public versus private institutional performance reporting: what is mandatory for quality improvement?

BACKGROUND

In the past 11 years, Ontario has generated institution-level performance report cards on outcomes of coronary artery bypass graft (CABG) surgery. The objective of this study was to evaluate the differences in patient characteristics and outcomes observed during the transition from no reporting to confidential, and ultimately public performance report cards for CABG surgery in a public health system.

METHODS

We used clinical and administrative data to assess crude, expected, and risk-adjusted 30-day mortality rates after isolated CABG surgery in Ontario for 67693 patients from September 1, 1991, to March 31, 2002. Confidence intervals on relative mortality reductions were determined by bootstrapping. We compared 30-day mortality trends to a control outcome (risk-adjusted 30-day all-cause readmission). We analyzed inhospital mortality trends for Ontario compared with the rest of Canada for the period from 1992 to 1998.

RESULTS

The risk-adjusted 30-day mortality rate decreased 29% (95% CI 21-39) from the era of no reporting (1991-1993) to confidential reporting (1994-1998). There was no further decrease with public reporting (1999-2001). The control outcome of 30-day readmission did not decrease across reporting eras. Inhospital mortality fell significantly faster in Ontario during the period of confidential reporting than in other parts of Canada.

CONCLUSION

Ontario CABG mortality outcomes improved sharply after provider results were confidentially disclosed at an institutional level. No such changes were seen for nondisclosed outcomes or regions outside Ontario. Further public reporting of outcomes had no discernible impact on performance. These results are consistent with the hypothesis that confidential disclosure of outcomes was sufficient to accelerate quality improvement in a public system with little competition for patients between hospitals.

Public reporting of hospital outcomes based on administrative data: risks and opportunities

In the wake of findings from the Bundaberg Hospital and Forster inquiries in Queensland, periodic public release of hospital performance reports has been recommended. A process for developing and releasing such reports is being established by Queensland Health, overseen by an independent expert panel. This recommendation presupposes that public reports based on routinely collected administrative data are accurate; that the public can access, correctly interpret and act upon report contents; that reports motivate hospital clinicians and managers to improve quality of care; and that there are no unintended adverse effects of public reporting. Available research suggests that primary data sources are often inaccurate and incomplete, that reports have low predictive value in detecting "outlier" hospitals, and that users experience difficulty in accessing and interpreting reports and tend to distrust their findings.

Using HIPE data as a research and planning tool: limitations and opportunities

BACKGROUND

The Hospital Inpatient Enquiry (HIPE) system is an important information source for research and health service planning activities. However, as it was not designed explicitly for these purposes, some limitations exist.

AIMS

To make recommendations that would increase the value of HIPE as a research and planning tool.

METHODS

Experiences of using HIPE for research and planning exercises were analysed so as to identify its limitations and their impact on research and planning.

RESULTS

Limitations were identified regarding data quality, policy issues and the general system.

CONCLUSIONS

To increase the utility of HIPE as a research and planning tool, a number of changes are recommended, including: expanding the system to cover private hospitals and outpatient and emergency services; adopting routine small area and socio-economic coding; adopting unique personal identifiers; publishing regular detailed reports with in-depth analyses; and considering making hospital identifiers available in certain circumstances.

New technology to enable personal monitoring and incident reporting can transform professional culture: the potential to favourably impact the future of health care

There have been recent exposures of poor health care performance in many countries with western health care systems. The poor performance has either related to poor or criminal practices routinely going undetected or to organizational indifference or hostility to staff raising concerns about perceived poor standards of care. The demonstration that routine performance data monitoring would have detected and prevented many of the deaths attributed to poor surgical standards in the Bristol Royal Infirmary paediatric cardiac surgery scandal and criminal behaviour in the Harold Shipman scandal has highlighted the need for routine data collection to demonstrate to both health care administrators and patients that minimum standards of clinical practice are being achieved. The recent proposal that surgical report cards represent an important minimum ethical standard for health care consent will force the medical profession to engage in the debate surrounding routine data collection for performance monitoring and other purposes. This article considers the cultural background to data collection in the medical profession and the cost implications of failing to improve data collection in the areas of performance monitoring and incident reporting. A potential solution developed by the Geelong hospital group and in use in Australia is proposed as a novel, technologically appropriate and working example of practical data collection. This model is endorsed by the professional specialties and supported by modern regulatory theory. The individual, local and system wide benefits of such personal professional data collection are outlined and the necessary prerequisites are detailed.

Improving the ICU: part 1

ICUs are a vital component of modern health care. Improving ICU performance requires that we shift from a paradigm that concentrates on individual performance to a different paradigm that emphasizes the need to assess and improve ICU systems and processes. This is the first part of a two-part treatise. It discusses existing problems in ICU care, and the methods for defining and measuring ICU performance.

Interstate comparisons of public hospital outputs using DRGs: are they fair?

OBJECTIVE

To assess whether there is variation among Australian States in the reporting and coding of important and relevant secondary diagnoses in public hospital data. Such variation is a potentially important problem because it may invalidate interstate (and other) comparisons of hospital outputs based on Diagnosis Related Groups (DRGs).

METHODS

Our outcome measure was the percentage of separations in the lowest-resource split for adjacent DRGs. To reduce potential bias due to interstate differences in the complexity of cases treated in public hospitals, we directly standardised by adjacent-DRG and analysed two subgroups of adjacent-DRGs where there is less discretion about the threshold for admission: obstetrics and major medical conditions.

RESULTS

There was important interstate variation in the percentage of separations in the lowest-resource split. The statistically significant differences represent a large number of medical records that might have been documented or coded differently if the separation had occurred in another State. For example, if Queensland had the same standardised percentage as South Australia, then an extra 10,100 separations in Queensland would have been allocated to a DRG with a higher cost weight.

CONCLUSIONS

The task of perfecting a national database is never complete and it makes sense to superimpose a permanent cycle of monitoring, followed by more detailed audits. The methods used in this paper could be routinely used to screen administrative hospital data to identify where detailed audits with feedback might be implemented with best effect. Unless interstate variation in the reporting and coding of important additional diagnoses is reduced, measuring public hospital outputs using DRGs will be of limited value at a national level.

Public Reporting and Case Selection for Percutaneous Coronary Interventions

OBJECTIVES

The purpose of this research was to determine the potential effect of public reporting on case selection for percutaneous coronary intervention (PCI).

BACKGROUND

Previous studies have suggested that public reporting of coronary artery bypass graft surgery (CABG) mortality might result in case selection bias and in denial of care to or out migration of high-risk patients. The potential effect of public reporting on case selection for PCI is unknown.

METHODS

We compared demographics, indications, and outcomes of 11,374 patients included in a multicenter (eight hospitals) PCI database in Michigan where no public reporting is present, with 69,048 patients in a statewide (34 hospitals) PCI database in New York, where public reporting is present. The primary end point was in-hospital mortality.

RESULTS

Patients in Michigan more frequently underwent PCI for acute myocardial infarction (14.4% vs. 8.7%, p < 0.0001) and cardiogenic shock (2.56% vs. 0.38%, p < 0.0001) than those in New York. The Michigan cohort also had a higher prevalence of congestive heart failure and extracardiac vascular disease. The unadjusted in-hospital mortality rate was significantly lower in New York than in Michigan (0.83% vs. 1.54%, p < 0.0001; odds ratio [OR] 0.54, 95% confidence interval [CI] 0.45 to 0.63). However, after adjustment for comorbidities, there was no significant difference in mortality between the two groups (adjusted OR 1.05, 95% CI 0.84 to 1.31, p = 0.70, c-statistic 0.88).

CONCLUSIONS

There are significant differences in case mix between patients undergoing PCI in Michigan and New York that result in marked differences in unadjusted mortality rates. A propensity in New York toward not intervening on higher-risk patients because of fear of public reporting of high mortality rates is a possible explanation for these differences.

Report cards and nursing homes

PURPOSE

We first describe which states have produced nursing home report cards; second, we compare what information is provided in these report cards; third, we identify data sources used to produce the report cards; and, finally, we examine seven factors previously shown to be associated with the usefulness of report-card information and provide several examples from current reporting efforts to illustrate how nursing home report cards could be improved.

DESIGN AND METHODS

We searched the Web sites for each state agency responsible for elder affairs-nursing homes. For those states identified as having a nursing home report card, we further examined the information presented.

RESULTS

We identified 19 states as having nursing home report cards (AZ, CO, FL, IL, IN, IO, MD, MA, MS, NV, NJ, NY, OH, PA, RI, TX, UT, VT, and WI). The information presented in these report cards differs quite substantially across states, although the data sources for report cards do not differ substantially. How the information is presented and our evaluation of the usefulness of the information is also highly varied.

IMPLICATIONS

Providing nursing home report-card information may be important in helping elders and their families choose a nursing facility. With 19 states identified in our research as providing nursing home report-card information on the World Wide Web, we were surprised and encouraged at this number of initiatives. We give some insight into the kinds of information that can be found on these report cards and what steps could be taken to improve how the information is presented.

An ethical framework for cardiac report cards: a qualitative study

Background

The recent proliferation of health care report cards, especially in cardiac care, has occurred in the absence of an ethical framework to guide in their development and implementation. An ethical framework is a consistent and comprehensive theoretical foundation in ethics, and is formed by integrating ethical theories, relevant literature, and other critical information (such as the views of stakeholders). An ethical framework in the context of cardiac care provides guidance for developing cardiac report cards (CRCs) that are relevant and legitimate to all stakeholders. The purpose of this study is to develop an ethical framework for CRCs.

Methods

Delphi technique – 13 panelists: 2 administrators, 2 cardiac nurses, 5 cardiac patients, 2 cardiologists, 1 member of the media, and 1 outcomes researcher. Panelists' views regarding the ethics of CRCs were analyzed and organized into themes.

Results

We have organized panelists' views into ten principles that emerged from the data: 1) improving quality of care, 2) informed understanding, 3) public accountability, 4) transparency, 5) equity, 6) access to information 7) quality of information, 8) multi-stakeholder collaboration, 9) legitimacy, and 10) evaluation and continuous quality improvement.

Conclusion

We have developed a framework to guide the development and dissemination of CRCs. This ethical framework can provide necessary guidance for those generating CRCs and may help them avoid a number of difficult issues associated with existing ones.

Evaluating the performance of an institution using an intensive care unit benchmark

OBJECTIVES

To describe the performances of selected intensive care units (ICUs) in a single institution using the Acute Physiology and Chronic Health Evaluation (APACHE) III benchmark and to propose interventions that may improve performance.

PATIENTS AND METHODS

In this retrospective study, we analyzed APACHE III data from critically ill patients admitted to ICUs at the Mayo Clinic in Rochester, Minn, between October 1994 and December 2003. We retrieved ICU performance measures based on first ICU day APACHE III values. Standardized ratios were defined as ratios of measured to predicted values. The primary performance measure was the standardized mortality ratio, and secondary performance measures were length of stay (LOS) ratios, low-risk monitor ICU admission rates, and ICU readmission rates. We calculated 95% confidence intervals (CIs) for each performance, graded as good, average, or poor.

RESULTS

Among 46,381 patients admitted during the study period, 57.5% were in surgical ICUs, 24.8% in a medical ICU, and 17.7% in a surgical-medical ICU. Low-risk monitoring accounted for 37.2% of admissions. Hospital standardized mortality ratios (95% CI) were 0.95 (0.90-0.99), 0.86 (0.81-0.91), and 0.70 (0.66-0.74) for medical, multispecialty, and surgical ICUs, respectively. Hospital LOS ratios (95% CI) were 0.83 (0.81-0.85), 0.91 (0.88-0.93), and 0.99 (0.97-1.00) for medical, multispecialty, and surgical ICUs, respectively. The ICU readmission rate for each ICU was higher than the 6.7% reported in the medical literature. Performances were good in mortality, average to good in LOS, average in low-risk admission, and poor in ICU readmission.

CONCLUSIONS

A national benchmarking database can highlight the strengths and weaknesses of ICUs. The performances of ICUs in a single institution may differ; therefore, the performance of each unit should be evaluated individually.

Consequences of impaired data quality on information retrieval in electronic patient records

OBJECTIVES

To assess the quality of specific information in electronic patient records and the consequences of sub-optimal data quality on automated information retrieval.

METHODS

Patient records were evaluated with respect to accuracy of data relevant for retrieval according to a source-oriented, time-oriented and concept-oriented view of the record. Retrieval effectiveness was estimated using various methods based on record structure, text based retrieval and combinations of these.

RESULTS

98.1% of record documents were consistent regarding author, 99.8% regarding department of origin and 90.9% regarding document date. Document type was definitely not consistent in 8% of the documents. Estimated recall was 97% with 50% precision for document retrieval on the basis of date, and varying from 31 to 100% for retrieval based on document type. Retrieval based on manually supplied semantic tags performed better than simple string-based methods and improved when combined with string-matching mechanisms.

CONCLUSIONS

Data attributes central for automated document retrieval in electronic patient records showed variable accuracy, with potentially negative consequences for basic record navigation. Text-based retrieval was inferior to methods based on data representing record structure. Quality of specific information elements suffered from lack of precise definitions and adequate mechanisms for quality assurance.