League tables of in vitro fertilisation clinics misinform patients

AAA benchmarking by Dr Foster: a cause for concern?

INTRODUCTION

The publication of interpretable performance data for hospitals is an important service. In November 2002, the medical benchmarking company Dr Foster published a league table based on the results of abdominal aortic aneurysm (AAA) repair. The purpose of our study was to establish the validity of the data used in benchmarking.

PATIENTS AND METHODS

Data on elective infra-renal AAA (IRAAA) repair was obtained from three sources. Data used by Dr Foster was based on the hospital PAS system. The databases for both Dr Foster and PAS were analysed and cross-referenced to the vascular unit database maintained by a separately employed audit co-ordinator.

RESULTS

Of 395 total aortic aneurysm repairs, 223 (56%) were identified as elective IRAAA repairs on the unit database. Of these, 125 were identified on the PAS database and 115 on the Dr Foster database. The number of deaths was the same in both the unit and Dr Foster databases (n = 11) but the Dr Foster database included deaths in patients who had undergone juxtarenal (n = 1), Type III TAAA (n = 2) and Type IV TAAA (n = 4) repairs and omitted 7 deaths following IRAAA. The sensitivity and specificity for the PAS dataset was 0.51 and 0.93, respectively. For Dr Foster, the results were worse with a sensitivity and specificity of 0.41 and 0.86, respectively. The accuracy of the data was 0.6 and 0.69 for Dr Foster and PAS, respectively. Standardised mortality ratios (SMRs) were used to rank hospitals. Dr Foster's published SMR for elective AAA repair for our unit was 160. The actual SMR was 67.

CONCLUSIONS

Robust and accurate published league tables should be supported and commended but currently available data appear to be misleading and may cause unnecessary concern to patients.

Treatment-related morbidity and hospital league tables: experience from a national audit of radiotherapy-induced morbidity in cervical carcinoma

Data are now available from a U.K. audit of survival and late morbidity following curative radiotherapy for cancer of the cervix treated in 1993. The complication rate per centre ranges from 0 to 67%. Although the frequency of complications following curative radiotherapy for cancer of the cervix might be considered to be an indicator of clinical performance, variation in treatment outcomes can be explained by sampling variability rather than real differences in quality of care. In the present study we have asked the question: could the disparity in complication rates between centres be no more or less than would be expected by chance? Our analysis suggests that this is the case, and for this reason it would be premature to use such outcome data to produce league tables or to assess institutional differences. Thus, ranking centres according to complication rate would not be valid, as the differences in rates observed are probably not significantly different from the national average. It is important that audit data are not used inappropriately and this analysis further highlights the need for reliable prospective collection of clinical information and the importance of considering sampling variability in interpreting the results of such studies.