Performance league tables: the NHS deserves better

Funnel plots a graphical instrument for the evaluation of population performance and quality of trauma care: a blueprint of implementation

BACKGROUND

Using patient outcomes to monitor medical centre performance has become an essential part of modern health care. However, classic league tables generally inflict stigmatization on centres rated as "poor performers", which has a negative effect on public trust and professional morale. In the present study, we aim to illustrate that funnel plots, including trends over time, can be used as a method to control the quality of data and to monitor and assure the quality of trauma care. Moreover, we aimed to present a set of regulations on how to interpret and act on underperformance or overperformance trends presented in funnel plots.

METHODS

A retrospective observational cohort study was performed using the Dutch National Trauma Registry (DNTR). Two separate datasets were created to assess the effects of healthy and multiple imputations to cope with missing values. Funnel plots displaying the performance of all trauma-receiving hospitals in 2020 were generated, and in-hospital mortality was used as the main indicator of centre performance. Indirect standardization was used to correct for differences in the types of cases. Comet plots were generated displaying the performance trends of two level-I trauma centres since 2017 and 2018.

RESULTS

Funnel plots based on data using healthy imputation for missing values can highlight centres lacking good data quality. A comet plot illustrates the performance trend over multiple years, which is more indicative of a centre's performance compared to a single measurement. Trends analysis offers the opportunity to closely monitor an individual centres' performance and direct evaluation of initiated improvement strategies.

CONCLUSION

This study describes the use of funnel and comet plots as a method to monitor and assure high-quality data and to evaluate trauma centre performance over multiple years. Moreover, this is the first study to provide a regulatory blueprint on how to interpret and act on the under- or overperformance of trauma centres. Further evaluations are needed to assess its functionality.

LEVEL OF EVIDENCE

Retrospective study, level III.

Hospital variation in admissions for low back pain following an emergency department presentation: a retrospective study

Background

One in 6 patients with low back pain (LBP) presenting to emergency departments (EDs) are subsequently admitted to hospital each year, making LBP the ninth most common reason for hospital admission in Australia. No studies have investigated and quantified the extent of clinical variation in hospital admission following an ED presentation for LBP.

Methods

We used routinely collected ED data from public hospitals within the state of New South Wales, Australia, to identify presentations of patients aged between 18 and 111 with a discharge diagnosis of LBP. We fitted a series of random effects multilevel logistic regression models adjusted by case-mix and hospital variables. The main outcome was the hospital-adjusted admission rate (HAAR). Data were presented as funnel plots with 95% and 99.8% confidence limits. Hospitals with a HAAR outside the 95% confidence limit were considered to have a HAAR significantly different to the state average.

Results

We identified 176,729 LBP presentations across 177 public hospital EDs and 44,549 hospital admissions (25.2%). The mean (SD) age was 51.8 (19.5) and 52% were female. Hospital factors explained 10% of the variation (ICC = 0.10), and the median odds ratio (MOR) was 2.03. We identified marked variation across hospitals, with HAAR ranging from 6.9 to 65.9%. After adjusting for hospital variables, there was still marked variation between hospitals with similar characteristics.

Conclusion

We found substantial variation in hospital admissions following a presentation to the ED due to LBP even after controlling by case-mix and hospital characteristics. Given the substantial costs associated with these admissions, our findings indicate the need to investigate sources of variation and to determine instances where the observed variation is warranted or unwarranted.

Supplementary information

The online version contains supplementary material available at 10.1186/s12913-022-08134-8.

Variation in clinical outcomes and process of care measures in community acquired pneumonia: a systematic review

Background

Variation in outcomes of patients with community acquired pneumonia (CAP) has been reported in some, but not all, studies. Although some variation is expected, unwarranted variation in healthcare impacts patient outcomes and equity of care. The aim of this systematic review was to: i) summarise current evidence on regional and inter-hospital variation in the clinical outcomes and process of care measures of patients hospitalised with CAP and ii) assess the strength of this evidence.

Methods

Databases were systematically searched from inception to February 2018 for relevant studies and data independently extracted by two investigators in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement. Included studies enrolled adults hospitalised with CAP and reported a measure of variation between two or more units in healthcare outcomes or process of care measures. Outcomes of interest were mortality, length of hospital stay (LOS) and re-admission rates. A structured synthesis of the studies was performed.

Results

Twenty-two studies were included in the analysis. The median number of units compared across studies was five (IQR 4–15). Evidence for variation in mortality between units was inconsistent; of eleven studies that performed statistical significance testing, five found significant variation. For LOS, of nine relevant studies, all found statistically significant variation. Four studies reported site of admission accounted for 1–24% of the total observed variation in LOS. A shorter LOS was not associated with increased mortality or readmission rates. For readmission, evidence was mixed; of seven studies, 4 found statistically significant variation. There was consistent evidence for variation in the use of intensive care, obtaining blood cultures on admission, receiving antibiotics within 8 h of admission and duration of intravenous antibiotics. Across all outcome measures, only one study accounted for natural variation between units in their analysis.

Conclusion

There is consistent evidence of moderate quality for significant variation in length of stay and process of care measures but not for in-patient mortality or hospital re-admission. Evidence linking variation in outcomes with variation in process of care measures was limited; where present no difference in mortality was detected despite POC variation. Adjustment for natural variation within studies was lacking; the proportion of observed variation due to chance is not quantified by existing evidence.

Monitoring of high-yield and periodical processes in health care

Statistical control charts have found valuable applications in health care, having been largely adopted from operations research in manufacturing. However, the most common types are not best-suited to monitor high-yield processes (outcomes comprising true/false fractions, 'near-zero') and periodical processes (characterized by sequences of single populations of finite sizes), but rather to monitor variable vital signs levels and, to a lesser degree, service performance indicators. We discuss control charts that are most suitable for fraction non-conforming measurements. We focus particularly on high-yield and periodical processes, i.e. range in which out-of-control conditions are expected and should be identified. For these conditions, we discuss control charts based on the family of hypergeometric distributions, explaining and comparing their application to more traditional alternatives with two health care case studies. We demonstrate that hypergeometric-type control charts provide higher sensitivity in timely identification of changing rare event fractions and are well-suited for monitoring of periodical processes, while remaining more resistant to false alarms, versus their alternatives.

Disentangling the contribution of hospitals and municipalities for understanding patient level differences in one-year mortality risk after hip-fracture: A cross-classified multilevel analysis in Sweden

Background

One-year mortality after hip-fracture is a widely used outcome measure when comparing hospital care performance. However, traditional analyses do not explicitly consider the referral of patients to municipality care after just a few days of hospitalization. Furthermore, traditional analyses investigates hospital (or municipality) variation in patient outcomes in isolation rather than as a component of the underlying patient variation. We therefore aimed to extend the traditional approach to simultaneously estimate both case-mix adjusted hospital and municipality comparisons in order to disentangle the amount of the total patient variation in clinical outcomes that was attributable to the hospital and municipality level, respectively.

Methods

We determined 1-year mortality risk in patients aged 65 or above with hip fractures registered in Sweden between 2011 and 2014. We performed cross-classified multilevel analysis with 54,999 patients nested within 54 hospitals and 290 municipalities. We adjusted for individual demographic, socioeconomic and clinical characteristics. To quantify the size of the hospital and municipality variation we calculated the variance partition coefficient (VPC) and the area under the receiver operator characteristic curve (AUC).

Results

The overall 1-year mortality rate was 25.1%. The case-mix adjusted rates varied from 21.7% to 26.5% for the 54 hospitals, and from 18.9% to 29.5% for the 290 municipalities. The VPC was just 0.2% for the hospital and just 0.1% for the municipality level. Patient sociodemographic and clinical characteristics were strong predictors of 1-year mortality (AUC = 0.716), but adding the hospital and municipality levels in the cross-classified model had a minor influence (AUC = 0.718).

Conclusions

Overall in Sweden, one-year mortality after hip-fracture is rather high. However, only a minor part of the patient variation is explained by the hospital and municipality levels. Therefore, a possible intervention should be nation-wide rather than directed to specific hospitals or municipalities.

Between-Hospital Variation in Revision Rates After Total Hip and Knee Arthroplasty in the Netherlands: Directing Quality-Improvement Initiatives

BACKGROUND

Variation in 1-year revision rates between Dutch hospitals after primary total hip and knee arthroplasty (THA and TKA) may direct quality-improvement initiatives if this variation accurately reflects true hospital differences. The aim of the present study was to assess the extent of variation, both overall and for specific indications, as well as the statistical reliability of ranking hospitals.

METHODS

All primary THAs and TKAs that were performed between January 2014 and December 2016 were included. Observed/expected (O/E) ratios regarding 1-year revision rates were depicted in a funnel plot with 95% control limits to identify outliers based on 1 or 3 years of data, both overall and by specific indication for revision. The expected number was calculated on the basis of patient mix with use of logistic regression models. The statistical reliability of ranking hospitals (rankability) on these outcomes indicates the percentage of total variation that is explained by "true" hospital differences rather than chance. Rankability was evaluated using fixed and random effects models, for overall revisions and specific indications for revision, including 1 versus 3 years of data.

RESULTS

The present study included 86,468 THAs and 73,077 TKAs from 97 and 98 hospitals, respectively. Thirteen hospitals performing THAs were identified as negative outliers (median O/E ratio, 1.9; interquartile range [IQR], 1.5-2.5), with 5 hospitals as outliers in multiple years. Eight negative outliers were identified for periprosthetic joint infection; 4, for dislocation; and 2, for prosthesis loosening. Seven hospitals performing TKAs were identified as negative outliers (median O/E ratio, 2.3; IQR, 2.2-2.8), with 2 hospitals as outliers in multiple years. Two negative outlier hospitals were identified for periprosthetic joint infection and 1 was identified for technical failures. The rankability for overall revisions was 62% (moderate) for THA and 46% (low) for TKA.

CONCLUSIONS

There was large between-hospital variation in 1-year revision rates after primary THA and TKA. For most outlier hospitals, a specific indication for revision could be identified as contributing to worse performance, particularly for THA; these findings are starting points for quality-improvement initiatives.

Hospital differences in mortality rates after hip fracture surgery in Denmark

Background

Thirty-day mortality after hip fracture is widely used when ranking hospital performance, but the reliability of such hospital ranking is seldom calculated. We aimed to quantify the variation in 30-day mortality across hospitals and to determine the hospital general contextual effect for understanding patient differences in 30-day mortality risk.

Methods

Patients aged ≥65 years with an incident hip fracture registered in the Danish Multidisciplinary Fracture Registry between 2007 and 2016 were identified (n=60,004). We estimated unadjusted and patient-mix adjusted risk of 30-day mortality in 32 hospitals. We performed a multilevel analysis of individual heterogeneity and discriminatory accuracy with patients nested within hospitals. We expressed the hospital general contextual effect by the median odds ratio (MOR), the area under the receiver operating characteristics curve and the variance partition coefficient (VPC).

Results

The overall 30-day mortality rate was 10%. Patient characteristics including high sociodemographic risk score, underweight, comorbidity, a subtrochanteric fracture, and living at a nursing home were strong predictors of 30-day mortality (area under the curve=0.728). The adjusted differences between hospital averages in 30-day mortality varied from 5% to 9% across the 32 hospitals, which correspond to a MOR of 1.18 (95% CI: 1.12-1.25). However, the hospital general context effect was low, as the VPC was below 1% and adding the hospital level to a single-level model with adjustment for patient-mix increased the area under the receiver operating characteristics curve by only 0.004 units.

Conclusions

Only minor hospital differences were found in 30-day mortality after hip fracture. Mortality after hip fracture needs to be lowered in Denmark but possible interventions should be patient oriented and universal rather than focused on specific hospitals.

Identifying exceptional cystic fibrosis care services: combining statistical process control with focus groups

Background

The Cystic Fibrosis (CF) Registry collects clinical data on all patients attending specialist CF centres in the UK. These data have been used to make comparisons between centres on key outcomes such as forced expiratory volume in 1 second (FEV1) using simple rankings, which promote the assumption that those with the highest measures provide ‘better’ care.

Objectives

To explore whether or not using statistical ‘process control’ charts that move away from league tables and adjusting for case mix (age, where appropriate; sex; CF genotype; pancreatic sufficiency; and socioeconomic status) could identify exceptional CF care services in terms of clinically meaningful outcomes. Then, using insight from patients and clinicians on what structures, processes and policies are necessary for delivering good CF care, to explore whether or not care is associated with observed differences in outcomes.

Design

Cross-sectional analyses.

Setting

Specialist CF centres in the UK.

Participants

Patients aged ≥ 6 years attending specialist CF centres and clinicians at these centres.

Main outcome measures

FEV1% predicted.

Data sources

Annual reviews taken from the UK CF Registry (2007–15).

Results

We studied FEV1 in many different ways and in different periods. In our analyses of both adult and paediatric centres, we observed that some centres showed repeated evidence of ‘special-cause variation’, with mean FEV1 being greater than the mean in some cases and lower than the mean in others. Some of these differences were explained by statistical adjustment for different measures of case mix, such as age, socioeconomic status, genotype and pancreatic sufficiency. After adjustment, there was some remaining evidence of special-cause variation for some centres. Our data at these centres suggest that there may be an association with the use of intravenous antibiotics. Workshops and focus groups with clinicians at paediatric and adult centres identified a number of structures, processes and policies that were felt to be associated with good care. From these, questionnaires for CF centre directors were developed and disseminated. However, the response rate was low, limiting the questionnaires’ use. Focus groups with patients to gain their insights into what is necessary for the delivery of good care identified themes similar to those identified by clinicians, and a patient questionnaire was developed based on these insights.

Limitations

Our data analyses suggest that differences in intravenous antibiotic usage may be associated with centre-level outcomes; this needs to be explored further in partnership with the centres. Our survey of centre directors yielded a low response, making it difficult to gain useful knowledge to inform further discussions with sites.

Conclusions

Our findings confirm that the CF Registry can be used to identify differences in clinical outcomes between centres and that case mix might explain some of these differences. As such, adjustment for case mix is essential when trying to understand how and why centres differ from the mean.

Future work

Future work will involve exploring with clinicians how care is delivered so that we can understand associations between care and outcomes. Patients will also be asked for their perspectives on the care they receive.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

A critique of the Uganda district league table using a normative health system performance assessment framework

Background

In 2003 the Uganda Ministry of Health (MoH) introduced the District League Table (DLT) to track district performance. This review of the DLT is intended to add to the evidence base on Health Systems Performance Assessment (HSPA) globally, with emphasis on Low and Middle Income Countries (LMICs), and provide recommendations for adjustments to the current Ugandan reality.

Methods

A normative HSPA framework was used to inform the development of a Key Informant Interview (KII) tool. Thirty Key Informants were interviewed, purposively selected from the Ugandan health system on the basis of having developed or used the DLT. KII data and information from published and grey literature on the Uganda health system was analyzed using deductive analysis.

Results

Stakeholder involvement in the development of the DLT was limited, including MoH officials and development partners, and a few district technical managers. Uganda policy documents articulate a conceptually broad health system whereas the DLT focuses on a healthcare system. The complexity and dynamism of the Uganda health system was insufficiently acknowledged by the HSPA framework. Though DLT objectives and indicators were articulated, there was no conceptual reference model and lack of clarity on the constitutive dimensions. The DLT mechanisms for change were not explicit. The DLT compared markedly different districts and did not identify factors behind observed performance. Uganda lacks a designated institutional unit for the analysis and presentation of HSPA data, and there are challenges in data quality and range.

Conclusions

The critique of the DLT using a normative model supported the development of recommendation for Uganda district HSPA and provides lessons for other LMICs. A similar approach can be used by researchers and policy makers elsewhere for the review and development of other frameworks.

Adjustments in Uganda district HSPA should consider: wider stakeholder involvement with more district managers including political, administrative and technical; better anchoring within the national health system framework; integration of the notion of complexity in the design of the framework; and emphasis on facilitating district decision-making and learning. There is need to improve data quality and range and additional approaches for data analysis and presentation.

Validation of quality indicators for end-of-life communication: results of a multicentre survey

BACKGROUND

The lack of validated quality indicators is a major barrier to improving end-of-life communication and decision-making. We sought to show the feasibility of and provide initial validation for a set of quality indicators related to end-of-life communication and decision-making.

METHODS

We administered a questionnaire to patients and their family members in 12 hospitals and asked them about advance care planning and goals-of-care discussions. Responses were used to calculate a quality indicator score. To validate this score, we determined its correlation with the concordance between the patients' expressed wishes and the medical order for life-sustaining treatments recorded in the hospital chart. We compared the correlation with concordance for the advance care planning component score with that for the goal-of-care discussion scores.

RESULTS

We enrolled 297 patients and 209 family members. At all sites, both overall quality indicators and individual domain scores were low and there was wide variability around the point estimates. The highest-ranking institution had an overall quality indicator score (95% confidence interval) of 40% (36%-44%) and the lowest had a score of 18% (11%-25%). There was a strong correlation between the overall quality indicator score and the concordance measure (r = 0.72, p = 0.008); the estimated correlation between the advance care planning score and the concordance measure (r = 0.35) was weaker than that between the goal-of-care discussion scores and the concordance measure (r = 0.53).

INTERPRETATION

Quality of end-of-life communication and decision-making appears low overall, with considerable variability across hospitals. The proposed quality indicator measure shows feasibility and partial validity. Study registration: ClinicalTrials.gov, no. NCT01362855.

Hospital variation in allogeneic transfusion and extended length of stay in primary elective hip and knee arthroplasty: a cross-sectional study

OBJECTIVES

Outcomes in total hip and knee arthroplasty (THA and TKA), such as allogeneic transfusions or extended length of stay (LoS), can be used to compare the performance of hospitals. However, there is much variation in these outcomes. This study aims to rank hospitals and to assess hospital differences of two outcomes in THA and TKA: allogeneic transfusions and extended LoS, and to additionally identify factors associated with these differences.

DESIGN

Cross-sectional medical record review study.

SETTING

Data were gathered in 23 Dutch hospitals.

PARTICIPANTS

1163 THA and 986 TKA patient admissions.

OUTCOMES

Hospitals were ranked based on their observed/expected (O/E) ratios regarding allogeneic transfusion and extended LoS percentages (extended LoS was defined by postoperative stay >4 days). To assess the reliability of these rankings, we calculated which percentage of the existing variation was based on differences between hospitals as compared with random variation (after adjustment for variation in patient characteristics). Associations between hospital-specific factors and O/E ratios were used to explore potential sources of differences.

RESULTS

The variation in O/E ratios between hospitals ranged from 0 to 4.4 for allogeneic transfusion, and from 0.08 to 2.7 for extended LoS. Variation in transfusion could in 21% be explained by hospital differences in THA and 34% in TKA. For extended LoS this was 71% in THA and 78% in TKA. Better performance (low O/E ratios) in transfusion was associated with more frequent tranexamic acid (TXA) use in TKA (R=-0.43, p=0.04). Better performance in extended LoS was associated with more frequent TXA use in THA (R=-0.45, p=0.03) and TKA (R=-0.65, p<0.001) and local infiltration analgesia (LIA) in TKA (R=-0.60, p=0.002).

CONCLUSIONS

Ranking hospitals based on allogeneic transfusion is unreliable due to small percentages of variation explained by hospital differences. Ranking based on extended LoS is more reliable. Hospitals using TXA and LIA have relatively fewer patients with transfusions and extended LoS.

Using nationwide ‘big data’ from linked electronic health records to help improve outcomes in cardiovascular diseases: 33 studies using methods from epidemiology, informatics, economics and social science in the ClinicAl disease research using LInked Bespoke studies and Electronic health Records (CALIBER) programme

Background

Electronic health records (EHRs), when linked across primary and secondary care and curated for research use, have the potential to improve our understanding of care quality and outcomes.

Objective

To evaluate new opportunities arising from linked EHRs for improving quality of care and outcomes for patients at risk of or with coronary disease across the patient journey.

Design

Epidemiological cohort, health informatics, health economics and ethnographic approaches were used.

Setting

230 NHS hospitals and 226 general practices in England and Wales.

Participants

Up to 2 million initially healthy adults, 100,000 people with stable coronary artery disease (SCAD) and up to 300,000 patients with acute coronary syndrome.

Main outcome measures

Quality of care, fatal and non-fatal cardiovascular disease (CVD) events.

Data platform and methods

We created a novel research platform [ClinicAl disease research using LInked Bespoke studies and Electronic health Records (CALIBER)] based on linkage of four major sources of EHR data in primary care and national registries. We carried out 33 complementary studies within the CALIBER framework. We developed a web-based clinical decision support system (CDSS) in hospital chest pain clinics. We established a novel consented prognostic clinical cohort of SCAD patients.

Results

CALIBER was successfully established as a valid research platform based on linked EHR data in nearly 2 million adults with > 600 EHR phenotypes implemented on the web portal (seehttps://caliberresearch.org/portal). Despite national guidance, key opportunities for investigation and treatment were missed across the patient journey, resulting in a worse prognosis for patients in the UK compared with patients in health systems in other countries. Our novel, contemporary, high-resolution studies showed heterogeneous associations for CVD risk factors across CVDs. The CDSS did not alter the decision-making behaviour of clinicians in chest pain clinics. Prognostic models using real-world data validly discriminated risk of death and events, and were used in cost-effectiveness decision models.

Conclusions

Emerging ‘big data’ opportunities arising from the linkage of records at different stages of a patient’s journey are vital to the generation of actionable insights into the diagnosis, risk stratification and cost-effective treatment of people at risk of, or with, CVD.

Future work

The vast majority of NHS data remain inaccessible to research and this hampers efforts to improve efficiency and quality of care and to drive innovation. We propose three priority directions for further research. First, there is an urgent need to ‘unlock’ more detailed data within hospitals for the scale of the UK’s 65 million population. Second, there is a need for scaled approaches to using EHRs to design and carry out trials, and interpret the implementation of trial results. Third, large-scale, disease agnostic genetic and biological collections linked to such EHRs are required in order to deliver precision medicine and to innovate discovery.

Study registration

CALIBER studies are registered as follows: study 2 – NCT01569139, study 4 – NCT02176174 and NCT01164371, study 5 – NCT01163513, studies 6 and 7 – NCT01804439, study 8 – NCT02285322, and studies 26–29 – NCT01162187. Optimising the Management of Angina is registered as Current Controlled Trials ISRCTN54381840.

Funding

The National Institute for Health Research (NIHR) Programme Grants for Applied Research programme (RP-PG-0407-10314) (all 33 studies) and additional funding from the Wellcome Trust (study 1), Medical Research Council Partnership grant (study 3), Servier (study 16), NIHR Research Methods Fellowship funding (study 19) and NIHR Research for Patient Benefit (study 33).

The use of control charts by laypeople and hospital decision-makers for guiding decision making

Graphs presenting healthcare data are increasingly available to support laypeople and hospital staff's decision making. When making these decisions, hospital staff should consider the role of chance-that is, random variation. Given random variation, decision-makers must distinguish signals (sometimes called special-cause data) from noise (common-cause data). Unfortunately, many graphs do not facilitate the statistical reasoning necessary to make such distinctions. Control charts are a less commonly used type of graph that support statistical thinking by including reference lines that separate data more likely to be signals from those more likely to be noise. The current work demonstrates for whom (laypeople and hospital staff) and when (treatment and investigative decisions) control charts strengthen data-driven decision making. We present two experiments that compare people's use of control and non-control charts to make decisions between hospitals (funnel charts vs. league tables) and to monitor changes across time (run charts with control lines vs. run charts without control lines). As expected, participants more accurately identified the outlying data using a control chart than using a non-control chart, but their ability to then apply that information to more complicated questions (e.g., where should I go for treatment?, and should I investigate?) was limited. The discussion highlights some common concerns about using control charts in hospital settings.

Considering chance in quality and safety performance measures: an analysis of performance reports by boards in English NHS trusts

OBJECTIVES

Hospital board members are asked to consider large amounts of quality and safety data with a duty to act on signals of poor performance. However, in order to do so it is necessary to distinguish signals from noise (chance). This article investigates whether data in English National Health Service (NHS) acute care hospital board papers are presented in a way that helps board members consider the role of chance in their decisions.

METHODS

Thirty English NHS trusts were selected at random and their board papers retrieved. Charts depicting quality and safety were identified. Categorical discriminations were then performed to document the methods used to present quality and safety data in board papers, with particular attention given to whether and how the charts depicted the role of chance, that is, by including control lines or error bars.

RESULTS

Thirty board papers, containing a total of 1488 charts, were sampled. Only 88 (6%) of these charts depicted the role of chance, and only 17 of the 30 board papers included any charts depicting the role of chance. Of the 88 charts that attempted to represent the role of chance, 16 included error bars and 72 included control lines. Only 6 (8%) of the 72 control charts indicated where the control lines had been set (eg, 2 vs 3 SDs).

CONCLUSIONS

Hospital board members are expected to consider large amounts of information. Control charts can help board members distinguish signals from noise, but often boards are not using them. We discuss demand-side and supply-side barriers that could be overcome to increase use of control charts in healthcare.

A comparison of hierarchical cluster analysis and league table rankings as methods for analysis and presentation of district health system performance data in Uganda

In 2003, the Uganda Ministry of Health introduced the district league table for district health system performance assessment. The league table presents district performance against a number of input, process and output indicators and a composite index to rank districts. This study explores the use of hierarchical cluster analysis for analysing and presenting district health systems performance data and compares this approach with the use of the league table in Uganda. Ministry of Health and district plans and reports, and published documents were used to provide information on the development and utilization of the Uganda district league table. Quantitative data were accessed from the Ministry of Health databases. Statistical analysis using SPSS version 20 and hierarchical cluster analysis, utilizing Wards' method was used. The hierarchical cluster analysis was conducted on the basis of seven clusters determined for each year from 2003 to 2010, ranging from a cluster of good through moderate-to-poor performers. The characteristics and membership of clusters varied from year to year and were determined by the identity and magnitude of performance of the individual variables. Criticisms of the league table include: perceived unfairness, as it did not take into consideration district peculiarities; and being oversummarized and not adequately informative. Clustering organizes the many data points into clusters of similar entities according to an agreed set of indicators and can provide the beginning point for identifying factors behind the observed performance of districts. Although league table ranking emphasize summation and external control, clustering has the potential to encourage a formative, learning approach. More research is required to shed more light on factors behind observed performance of the different clusters. Other countries especially low-income countries that share many similarities with Uganda can learn from these experiences.

Making quality registers supporting improvements: a systematic review of the data visualization in 5 quality registries

Traditionally, quality registries have been initiated, developed, and used by physicians essentially for research purposes. There is an unrealized opportunity to expand and strengthen the contribution of quality registries in health care quality improvement. This article aims to characterize quality registry annual reports regarding factors deemed important to process improvement. The 2012 annual reports of the 5 most highly developed Swedish quality registries were examined. Each of the 636 charts included was coded according to an abstraction form. Results show that league tables are highly prevalent, whereas funnel plots and control charts are rare. Health care quality is monitored over time on the basis of few and highly aggregated measurements, and it is usually measured using percentages. In conclusion, quality registry annual reports lack both the level of detail and the consideration of random variation necessary to being able to be systematically used in process improvement. Users of annual reports are recommended caution when discussing differences in quality, both over time and across health care providers, as they can be due to chance and insufficient guidance is provided on the reports in this regard. To better support process improvement, annual reports should thus be more detailed and give more consideration to random variation.

Prevalent but moderate variation across small geographic regions in patient nonadherence to evidence-based preventive therapies in older adults after acute myocardial infarction

BACKGROUND

Patient long-term adherence to β-blockers, HMG-CoA reductase inhibitors (statins), and angiotensin-converting enzyme inhibitors (ACEIs)/angiotensin receptor blockers (ARBs) after acute myocardial infarction (AMI) is alarmingly low. It is unclear how prevalent patient adherence may be across small geographic areas and whether this geographic prevalence may vary.

METHODS

This is a retrospective cohort study using Medicare service claims files from 2007 to 2009 with Medicare beneficiaries 65 years and above who were alive 30 days after the index AMI hospitalization between January 1, 2008 and December 31, 2008 (N=85,017). The adjusted proportions of patients adherent to β-blockers, statins, and ACEIs/ARBs, respectively, in the 12 months after discharge across the 306 Hospital Referral Regions (HRRs) were measured and compared by control chart. The intracluster correlation coefficient (ICC) and the additional prediction power from this small-area variation on individual patient adherence were assessed.

RESULTS

The adjusted proportion of patients adherent across HRRs ranged from 58% to 74% (median, 66%) for β-blockers, from 57% to 67% (median, 63%) for ACEIs/ARBs, and from 58% to 73% (median, 66%) for statins. The ICC was 0.053 (95% CI, 0.043-0.064) for β-blockers, 0.050 (95% CI, 0.039-0.061) for ACEIs/ARBs, and 0.041 (95% CI, 0.031-0.052) for statins. The adjusted proportion of patients adherent across HRRs increased the c-statistic by 0.01-0.02 (P < 0.0001).

CONCLUSIONS

Nonadherence to evidence-based preventive therapies post-AMI among older adults was prevalent across small geographic regions. Moderate small-area variation in patient adherence exists.

Exploring variations in childhood stunting in Nigeria using league table, control chart and spatial analysis

BACKGROUND

Stunting, linear growth retardation is the best measure of child health inequalities as it captures multiple dimensions of children's health, development and environment where they live. The developmental priorities and socially acceptable health norms and practices in various regions and states within Nigeria remains disaggregated and with this, comes the challenge of being able to ascertain which of the regions and states identifies with either high or low childhood stunting to further investigate the risk factors and make recommendations for action oriented policy decisions.

METHODS

We used data from the birth histories included in the 2008 Nigeria Demographic and Health Survey (DHS) to estimate childhood stunting. Stunting was defined as height for age below minus two standard deviations from the median height for age of the standard World Health Organization reference population. We plotted control charts of the proportion of childhood stunting for the 37 states (including federal capital, Abuja) in Nigeria. The Local Indicators of Spatial Association (LISA) were used as a measure of the overall clustering and is assessed by a test of a null hypothesis.

RESULTS

Childhood stunting is high in Nigeria with an average of about 39%. The percentage of children with stunting ranged from 11.5% in Anambra state to as high as 60% in Kebbi State. Ranking of states with respect to childhood stunting is as follows: Anambra and Lagos states had the least numbers with 11.5% and 16.8% respectively while Yobe, Zamfara, Katsina, Plateau and Kebbi had the highest (with more than 50% of their under-fives having stunted growth).

CONCLUSIONS

Childhood stunting is high in Nigeria and varied significantly across the states. The northern states have a higher proportion than the southern states. There is an urgent need for studies to explore factors that may be responsible for these special cause variations in childhood stunting in Nigeria.

Use of surgical-site infection rates to rank hospital performance across several types of surgery

BACKGROUND

Comparing and ranking hospitals based on health outcomes is becoming increasingly popular, although case-mix differences between hospitals and random variation are known to distort interpretation. The aim of this study was to explore whether surgical-site infection (SSI) rates are suitable for comparing hospitals, taking into account case-mix differences and random variation.

METHODS

Data from the national surveillance network in the Netherlands, on the eight most frequently registered types of surgery for the year 2009, were used to calculate SSI rates. The variation in SSI rate between hospitals was estimated with multivariable fixed- and random-effects logistic regression models to account for random variation and case mix. 'Rankability' (as the reliability of ranking) of the SSI rates was calculated by relating within-hospital variation to between-hospital variation.

RESULTS

Thirty-four hospitals reported on 13 629 patients, with overall SSI rates per surgical procedure varying between 0 and 15·1 per cent. Statistically significant differences in SSI rate between hospitals were found for colonic resection, caesarean section and for all operations combined. Rankability was 80 per cent for colonic resection but 0 per cent for caesarean section. Rankability was 8 per cent in all operations combined, as the differences in SSI rates were explained mainly by case mix.

CONCLUSION

When comparing SSI rates in all operations, differences between hospitals were explained by case mix. For individual types of surgery, case mix varied less between hospitals, and differences were explained largely by random variation. Although SSI rates may be used for monitoring quality improvement within hospitals, they should not be used for ranking hospitals.

Lessons from 2012: What the NHS Can Learn from Britain's Olympic Success

The 2012 London Olympic and Paralympic Games were widely regarded as an organisational and sporting success for the United Kingdom. Therefore, it is prudent to consider what other large, public endeavours might learn from the Games’ success.

Team GB worked to develop a positive team culture based around shared values. This is something the National Health Service (NHS) could learn from, as an organisation which can appear to lack this culture. The NHS should also work harder to adopt evidence-based practices, and to adopt them quickly, as is often the case in sport. Sport is the ultimate example of transparent results reporting, and the NHS ought to consider systematic reporting of risk-adjusted performance data, which may drive improved performance. The NHS should pay attention to the experiences of successful Olympic sports with centralised centres of excellence, and to medical data which suggests that better outcomes result from centres of excellence. The NHS and wider government should look to Olympic athletes and place more emphasis on prevention of disease by encouraging positive lifestyle choices. Finally, the NHS should develop private sector partnerships carefully.

We must look to gather knowledge and ideas from every area of life in pursuit of excellence in the NHS. Experience of the Olympics offers a number of instructive lessons.

What is the value of hospital mortality indicators, and are there ways to do better?

Monitoring hospital performance using patient safety indicators is one of the key components of healthcare reform in Australia. Mortality indicators, including the hospital standardised mortality ratio and deaths in low mortality diagnosis reference groups have been included in the core national hospital-based outcome indicator set recommended for local generation and review and public reporting. Although the face validity of mortality indicators such as these is high, an increasing number of studies have demonstrated that there are concerns regarding their internal, construct and criterion validity. Use of indicators with poor validity has the consequence of potentially incorrectly classifying hospitals as performance outliers and expenditure of limited hospital staff time on activities which may provide no gain to hospital quality and safety and may in fact cause damage to morale. This paper reviews the limitations of current approaches to monitoring hospital quality and safety performance using mortality indicators. It is argued that there are better approaches to improving performance than monitoring with mortality indicators generated from hospital administrative data. These approaches include use of epidemiologically sound, clinically relevant data from clinical-quality registries, better systems of audit, evidence-based bundles, checklists, simulators and application of the science of complex systems.

Funnel plots and risk-adjusted count data adverse events. A limitation of indirect standardisation

Reporting of hospital adverse event data is becoming increasingly mandated and this has motivated work on methods for the analysis and display of these data for groups of institutions. Currently, the method preferred by many workers is the funnel plot. Often, indirect standardisation is employed to produce these plots. It appears that, when used to display binary data such as surgical site infection or mortality data, the method is satisfactory. Increasingly, these data are risk-adjusted. However, risk adjustment of these data usually involves individual patients undergoing the same or similar procedures and the method does not appear to mislead. However, when dealing with count data such as bacteraemias it appears that this method can mislead, particularly where methods for risk adjustment of these data are used. Information about the hospitals or units of interest rather than individual patients is employed. For example, one hospital may have plastic and cardiac surgery units in which bacteraemias occur infrequently whereas another may provide treatment for renal failure (including transplantation) and have a large haematology-oncology unit (also including transplantation), each of which would expect higher bacteraemia rates. Moreover, the hospitals and units within them may differ substantially in size. It is well known that indirect standardisation can give biased results when denominators differ substantially. We illustrate this difficulty with risk-adjusted bacteraemia data from the Queensland Health Centre for Healthcare Infection, Surveillance and Prevention (CHRISP) database.

Creating a culture of clinical excellence in critical care nutrition: the 2008 "Best of the Best" award

OBJECTIVE

To develop, validate, and implement a system to reward top performers in critical care nutrition practice and to illuminate characteristics of top-performing intensive care units (ICUs).

DESIGN

An international, prospective, observational, cohort study conducted in May 2008.

SETTING

179 ICUs from 18 countries.

PATIENTS

2956 consecutively enrolled mechanically ventilated adult patients who stayed in the ICU for at least 72 hours.

INTERVENTIONS

To qualify for the "Best of the Best" (BOB) award, sites had to have implemented a nutrition protocol and contributed complete data on a minimum of 20 patients.

MEASUREMENTS AND MAIN RESULTS

Data on nutrition practices were collected from ICU admission to ICU discharge for a maximum of 12 days. Eligible sites were ranked based on their performance on the following 5 criteria: adequacy of provision of energy, use of enteral nutrition (EN), early initiation of EN, use of promotility drugs and small bowel feeding tubes, and adequate glycemic control. Of the 179 participating ICUs, 81 qualified for the BOB award. Overall, the average nutrition adequacy across sites was 56.2% (site range, 20.3%-90.1%). The top 10 performers were identified and publicly recognized. Regression analysis suggested that the presence of a dietitian in the ICU was associated with a high BOB award ranking, whereas being located in the United States or China, relative to other participating countries, was associated with worst performance.

CONCLUSIONS

There is variable performance with respect to critical care nutrition practices across the world.

The use of quality control performance charts to analyze cesarean delivery rates nationally

OBJECTIVE

To examine the use of quality control performance charts to analyze cesarean rates nationally.

METHODS

Information on cesarean rates was obtained for all 19 Irish maternity hospitals receiving state funding in 2009. All women who underwent cesarean delivery of a live or stillborn infant weighing 500 g or more between January 1 and December 31 were included. Deliveries were classified as elective or emergency. Individual hospitals were not identified in the analysis.

RESULTS

The mean rates per hospital of elective and emergency cesarean were 12.9±2.6% (n=9337) and 13.8±3.0% (n=9989), respectively-giving an overall mean rate of 26.7±4.2% (n=19326) per hospital. Cesarean rates were normally distributed. Using a quality control performance chart with a cutoff 2 standard deviations from the mean, 1 hospital was above the normal range for both total and elective cesareans, indicating that its pre-labor obstetric practices warrant clinical review. Another hospital had a mean emergency cesarean rate above the normal range, indicating that its labor ward practices warrant review.

CONCLUSION

Quality control performance charts can be used to analyze cesarean rates nationally and, thus, to identify hospitals at which obstetric practices should be reviewed.

Performance evaluations and league tables: do they capture variation between organizational units? An analysis of 5 Swedish pharmacological performance indicators

BACKGROUND

The use of league tables during the last decade has frequently been employed to assess quality in health care. However, few studies have attempted to assess quality by quantifying the variability across the organizational units or attempted to investigate whether the units are the correct context that really influences the outcome under study.

OBJECTIVES

To quantify the variation between different organizational units regarding 5 different Swedish national pharmacological performance indicators and to examine whether the organizational units under study are a valid construct of the context that influences the specific outcome.

RESEARCH DESIGN

A multilevel model with patients nested within health care units that in turn were nested within County councils was used. By using measures of variance (intraclass correlation [ICC]), we quantified the extent to which the 5 indicators of health care quality were conditioned by the specified units.

RESULTS

For all 5 studied indicators, the variation between county councils was small (ICC ranged from 2% to 7%), whereas the variation among health care units seemed to be more important (ICC ranged from 20% to 40%).

CONCLUSION

As the variation between county councils was small, using league tables for performance evaluation seems to be inappropriate. If league tables are to be presented, the relative size of the variation at the higher levels and an analysis regarding the possible influence of the context for the specific outcome should be included. This approach provides useful information for identifying relevant contexts to capture health care variation.

Do inter-hospital comparisons of in-hospital, acute myocardial infarction case-fatality rates serve the purpose of fostering quality improvement? An evaluative study

BACKGROUND

In-hospital case-fatality rates in patients, admitted for acute myocardial infarction (AMI-CFRs), are internationally used as a quality indicator. Attempting to encourage the hospitals to assume responsibility, the Belgian Ministry of Health decided to stimulate initiatives of quality improvement by means of a limited set of indicators, among which AMI-CFR, to be routinely analyzed. In this study we aimed, by determining the existence of inter-hospital differences in AMI-CFR, (1) to evaluate to which extent Belgian discharge records allow the assessment of quality of care in the field of AMI, and (2) to identify starting points for quality improvement.

METHODS

Hospital discharge records from all the Belgian short-term general hospitals in the period 2002-2005. The study population (N = 46,287) included patients aged 18 years and older, hospitalized for AMI. No unique patient identifier being present, we tried to track transferred patients. We assessed data quality through a comparison of MCD with data from two registers for acute coronary events and through transfer and sensitivity analyses. We compared AMI-CFRs across hospitals, using multivariable logistic regression models. In the main model hospitals, Charlson's co-morbidity index, age, gender and shock constituted the covariates. We carried out two types of analyses: a first one wherein transferred-out cases were excluded, to avoid double counting of patients when computing rates, and a second one with exclusion of all transferred cases, to allow the study of patients admitted into, treated in and discharged from the same hospital.

RESULTS

We identified problems regarding both the CFR's numerator and denominator.Sensitivity analyses revealed differential coding and/or case management practices. In the model with exclusion of transfer-out cases, the main determinants of AMI-CFR were cardiogenic shock (OR(adj) 23.0; 95% CI [20.9;25.2]), and five-year age groups OR(adj) 1.23; 95% CI [1.11;1.36]). Sizable inter-hospital and inter-type of hospital differences {(OR(comunity vs tertiary hospitals)1.36; 95% CI [1.34;1.39]) and (OR(intermediary vs tertiary hospitals)1.36; 95% CI [1.34;1.39])}, and nonconformities to guidelines for treatment were observed.

CONCLUSIONS

Despite established data quality shortcomings, the magnitude of the observed differences and the nonconformities constitute leads to quality improvement. However, to measure progress, ways to improve and routinely monitor data quality should be developed.

The National Bowel Cancer Audit: the risks and benefits of moving to open reporting of clinical outcomes

BACKGROUND

The government's proposals to openly report clinical outcomes poses challenges to the National Bowel Cancer Audit now funded by the UK department of health.

AIM

To identify the benefits and risks of open reporting and to propose ways the risks might be minimized.

METHODS

A review of the literature on clinical audit and the consequences of open reporting.

RESULTS

There are significant potential benefits of a national audit of bowel cancer including protecting patients from sub-standard care, providing clinicians with externally validated evidence of their performance, outcome data for clinical governance and evidence that increases in government expenditure are achieving improvements in survival from bowel cancer. These benefits will only be achieved if the audit captures most of the cases of bowel cancer in the UK, the data collected is complete and accurate, the results are risk adjusted and these are presented to the public in a way that is fair, clear and understandable. Involvement of clinicians who have confidence in the results of the audit and who actively compare their own results against a national standard is essential. It is suggested that a staged move to open reporting should minimise the risk of falsely identifying an outlying unit.

CONCLUSION

The fundamental aim of the National Bowel Cancer Audit is the pursuit of excellence by identification and adoption of best practice. This could achieve a continuous improvement in the care of all patients with bowel cancer in the UK. The ACPGBI suggests a safer way of transition to open reporting to avoid at least some of its pitfalls.

A guide for mental health clinicians to develop and undertake benchmarking activities

There is a growing expectation for staff to participate in benchmarking activities. If benchmarking projects are to be successful, managers and clinicians need to be aware of the steps involved. In this article, we identify key aspects of benchmarking and consider how clinicians and managers can respond to and meet contemporary requirements for the development of sound benchmarking relationships. Practicalities and issues that must be considered by benchmarking teams are also outlined. Before commencing a benchmarking project, ground rules and benchmarking agreements must be developed and ratified. An understandable benchmarking framework is required: one that is sufficiently robust for clinicians to engage in benchmarking activities and convince others that benchmarking has taken place. There is a need to build the capacity of clinicians in relation to benchmarking.

Surgical report cards: the myth and the reality

There seems no good reason for doctors to work in secret. Individual users of healthcare and the community in general, which ultimately bears the cost, are perfectly entitled to know how their health services and health providers are performing. The promulgation of surgical report cards has been hailed by some as a liberating step in the right direction. This paper seeks to analyse, from a clinician's perspective, the evolution and limitations of report cards. Ultimately, the importance of report cards will not be their immediate utility, which is minimal, but as a first step in a much wider and far more important debate about how we meaningfully measure the quality of health services and providers (including managers and bureaucrats), the likely cost of such an enterprise, how much we are willing and able to pay and how we reconcile the competing needs of information versus clinical and preventive care when all are competing for the same and inadequate pool of resources.

Analysing the performance ofin vitrofertilization clinics in the United Kingdom

During the past century, the manufacturing industry has achieved major successes in improving the quality of its products. An essential factor in these successes has been the use of Walter A. Shewhart's pioneering work in the economic control of variation, which culminated in the development of a simple yet powerful theory of variation, which classifies variation as having a common or special cause and thus guides the user to the most appropriate action to effect improvement. Using publicly available performance data, which includes percentage of live births and multiple births, for in vitro fertilization (IVF) clinics in the United Kingdom, we show a central role for Shewhart's approach in moving away from the limitations and controversies associated with performance league tables towards data analyses to support continual improvement. We outline strategies for dealing with common and special causes of variation in IVF clinic performance data.

Outcome prediction in critical care: the Simplified Acute Physiology Score models

PURPOSE OF REVIEW

Outcome prediction models measuring severity of illness of patients admitted to the intensive care unit should predict hospital mortality. This review describes the state-of-the-art of Simplified Acute Physiology Score models from the clinical and managerial perspectives. Methodological issues concerning the effects of differences between new samples and original databases in which the models were developed are considered.

RECENT FINDINGS

The progressive lack of fit of the Simplified Acute Physiology Score II in independent intensive care unit populations induced investigators to propose customizations and expansions as potential evolutions for Simplified Acute Physiology Score II. We do not know whether those solutions did solve the issue because there are no demonstrations of consistent good fit in new databases. The recently developed Simplified Acute Physiology Score 3 Admission Score with customization for geographical areas is discussed. The points shared by the Simplified Acute Physiology Score models and the pros and cons for each of them are introduced.

SUMMARY

Comparisons of intensive care unit performance should take into account not only the patient severity of illness, but also the effect of the 'intensive care unit variable', that is, differences in human resources, structure, equipment, management and organization of the intensive care unit. In the future, moving from patient and geographical area adjustment to resource use could allow the user to adjust for differences in healthcare provision.

The application of risk-adjusted control charts using the Paediatric Index of Mortality 2 for monitoring paediatric intensive care performance in Australia and New Zealand

OBJECTIVE

To assess the applicability of risk-adjusted sequential control charts using the Paediatric Index of Mortality version 2 for monitoring of the quality of paediatric intensive care.

DESIGN

Observational study.

SETTING

A registry of patient admissions to paediatric intensive care units (PICUs) in Australia and New Zealand.

PATIENTS

A total of 10,710 patients admitted to eight PICUs during a 24-month period.

MEASUREMENTS AND RESULTS

A series of risk-adjusted control charts was created for each PICU. Modified sequential probability ratio tests were used to test the hypothesis that the PICUs being monitored were 'out of control', where loss of control was arbitrarily defined as the odds of death exceeding twice the odds of dying as estimated by PIM2. In 24 months of monitoring, there was one alarm signal, suggesting the odds of deaths had doubled, and there was one signal, in another PICU, suggesting the odds of death had halved.

CONCLUSIONS

The major advantage of risk-adjusted sequential control charts is that the technique allows unit performance to be monitored continuously over time, rather than intermittently, with the aim of rapidly detecting a change in performance as soon as possible after it occurs. This technique is suitable for continuously screening for a change in outcome within a PICU over time and complements other methods of monitoring the quality of paediatric intensive care.

The impact of different prognostic models and their customization on institutional comparison of intensive care units

OBJECTIVES

To evaluate the influence of choice of a prognostic model and the effect of customization of these models on league tables (i.e., rank-order listing) in which intensive care units (ICUs) are ranked by standardized mortality ratios using Acute Physiology and Chronic Health Evaluation (APACHE) II, Simplified Acute Physiology Score (SAPS) II, and Mortality Probability Model II (MPM24II).

DESIGN

Retrospective analysis of prospectively collected data on ICU admissions.

SETTING

Forty Dutch ICUs.

PATIENTS

A data set from a national registry of 86,427 patients from January 2002 to October 2006.

INTERVENTIONS

The league tables associated with the different models were compared to evaluate their agreement. Bootstrapping was used to quantify the uncertainty in the ranks for ICUs. First, for each ICU the median rank and its 95% confidence interval were identified for each model. Then, for a given pair of models, for each ICU the median difference in rank and its associated 95% confidence interval were computed. A difference in rank for an ICU for a given pair of models was considered relevant if it was statistically significant and if one of the models would categorize this ICU as a performance outlier (excellent performer or very poor performer) while the other did not.

MEASUREMENTS AND MAIN RESULTS

For 20 ICUs, there was a significant difference in rank (2-19 positions) between one or more pairs of models. Three ICUs were rated as performance outliers by one of the models, while the other excluded this possibility with 95% certainty. Furthermore, for ten ICUs, one or more pairs of models classified these ICUs as performance outliers while the other model did not do so with certainty. Regarding the agreement between the original models and their customized versions, in all cases the median change in rank was three positions or less and the models fully agreed with respect to which ICUs should be classified as performance outliers.

CONCLUSIONS

Institutional comparison based on case-mix adjusted league tables is sensitive to the choice of prognostic model but not to customization of these models. League tables should always display the uncertainty associated with institutional ranks.

Accuracy of the surgeons' clinical prediction of perioperative complications using a visual analog scale

BACKGROUND

The ability to predict who will develop perioperative complications remains difficult because the etiology of adverse events is multifactorial. This study examines the preoperative and postoperative ability of the surgeon to predict complications and assesses the significance of a change in prediction.

METHODS

This was a prospective study of 1013 patients. The surgeon assessed the risk of a major complication on a 100-mm visual analog scale (VAS) immediately before and after surgery. When the VAS score was changed, the surgeon was asked to document why. Patients were assessed up to 30 days postoperatively.

RESULTS

Surgeons made a meaningful preoperative prediction of major complications (median score = 27 mm vs. 19 mm, p < 0.01), with an area under the receiver operating characteristic curve of 0.74 for mortality, 0.67 for major complications, and 0.63 for all complications. A change in the VAS score postoperatively was due to technical reasons in 74% of stated cases. An increased VAS score identified significantly more complications, but the improvement in the discrimination was small. When included in a multivariate model for predicting postoperative complications, the surgeon's VAS score functioned as an independent predictive variable and improved the predictive ability, goodness of fit, and discrimination of the model.

CONCLUSIONS

Clinical assessment of risk by the surgeon using a VAS score independently improves the prediction of perioperative complications. Including the unique contribution of the surgeon's clinical assessment should be considered in models designed to predict the risk of surgery.

Measuring the performance of urban healthcare services: results of an international experience

The objective of this paper is to apply a framework for country-level performance assessment to the cities of Montreal, Canada, and Barcelona, Spain, and to use this framework to explore and understand the differences in their health systems. The UK National Health Service Performance Assessment Framework was chosen. Its indicators went through a process of selection, adaptation and prioritisation. Most of them were calculated for the period 2001-3, with data obtained from epidemiological, activity and economic registries. Montreal has a higher number of old people living alone and with limitations on performing one or more activities of daily life, as well as longer hospital stays for several conditions, especially in the case of elderly patients. This highlights a lack of mid-term, long-term and home care services. Diabetes-avoidable hospitalisation rates are also significant in Montreal, and are likely to improve following reforms in primary care. Efficient health policies such as generic drug prescription and major ambulatory surgery are lower in Barcelona. Rates of caesarean deliveries are higher in Barcelona, owing to demographics and clinical practice. Waiting times for knee arthroplasty are longer in Barcelona, which has triggered a plan to reduce them. In both cities, avoidable mortality and the prevalence of smoking have been identified as areas for improvement through preventive services. In conclusion, performance assessment fits perfectly in an urban context, as it has been shown to be a useful tool in designing and monitoring the accomplishment of programmes in both cities, to assess the performance of the services delivered, and for use in policy development.

Effect of an acute medical admission unit on key quality indicators assessed by funnel plots

OBJECTIVE

To determine the effect of the introduction of an acute medical admissions unit (AMAU) on key quality efficiency and outcome indicator comparisons between medical teams as assessed by funnel plots.

METHODS

A retrospective analysis was performed of data relating to emergency medical patients admitted to St James' Hospital, Dublin between 1 January 2002 and 31 December 2004, using data on discharges from hospital recorded in the hospital in-patient enquiry system. The base year was 2002 during which patients were admitted to a variety of wards under the care of a named consultant physician. In 2003, two centrally located wards were reconfigured to function as an AMAU, and all emergency patients were admitted directly to this unit. The quality indicators examined between teams were length of stay (LOS) <30 days, LOS >30 days, and readmission rates.

RESULTS

The impact of the AMAU reduced overall hospital LOS from 7 days in 2002 to 5 days in 2003/04 (p<0.0001). There was no change in readmission rates between teams over the 3 year period, with all teams displaying expected variability within control (95%) limits. Overall, the performance in LOS, both short term and long term, was significantly improved (p<0.0001), and was less varied between medical teams between 2002 and 2003/04.

CONCLUSIONS

Introduction of the AMAU improved performance among medical teams in LOS, both short term and long term, with no change in readmissions. Funnel plots are a powerful graphical technique for presenting quality performance indicator variation between teams over time.

[The objective assessment of the quality of hospital care: dream or reality?]

One of the hottest topics of the international journals is the question: what is the effect of the public indicator-based quality assessment on the quality of care and on the decision of stakeholders, and which criteria should be applied for development of public quality assessment system. According to the international literature the paper discusses 6 topics: (1) the ability of indicators to distinguish providers from the point of view of quality; (2) the appropriateness of outcome indicators to assess providers; (3) the ability of league tables to rank providers; (4) the people's behaviour during choosing providers; (5) the impact of indicator-based public report; (6) recommendations for developing quality assessment system. Based on the literature review, the ability of indicators in distinguishing providers from the point of view of quality is doubtful primarily because of risk-adjustment problems. Other reasons are: the outcomes of care do not definitely refer to the quality of care process; the rankings of providers (league tables) based on more indicators are not reliable; people take into account mainly distance and the opinions of acquaintance when they choose providers; as a result of public reports the overall quality of care is declining. The publication of the results of measurement to assess providers has to be considered as a tool. For the purpose of helping people in choosing providers, the publication of patient satisfaction survey designed according to their preferences could achieve the desired effect. The quality improvement aims are definitely helped by the direct feedback to providers about the indicator values. Furthermore, much finer picture can be made if the standardized audits of care and organisational processes are inserted into external assessment procedures.

An analysis of hospital trauma care performance evaluation

OBJECTIVE

The purpose of this article was to report on a study on the hospital trauma care performance evaluation based on a database of trauma events of participating UK and European hospitals.

METHODS

Performance evaluation has become increasingly important in the quality assessment of health care in general and trauma care in particular. For many years, attempts to quantifying the performance of trauma care systems on a numerical scale have been developed and applied, including the use of Ws statistic. The Trauma Audit and Research Network collected and managed the data. We first investigated the currently used approaches in the evaluation of trauma care systems, and then proposed an alternative using a statistical control based approach for the comparison of different hospitals at one time. Different control charts and types of calculations were also proposed for the chronologic outcome chart, which plots the variation of trauma care within one hospital over time.

RESULTS

New graphical methods for hospital trauma care performance evaluation based on statistical process plots were developed and tested on the project database.

CONCLUSION

A control chart approach to the presentation of the outcome charts for hospital trauma care performance evaluation is presented in this article. The charts are more meaningful than the "caterpillar" plot traditionally used, and avoid the ranking of institutions into "league tables".