Comparing and ranking hospitals based on outcome: results from The Netherlands Stroke Survey

Variation Between Hospitals in Outcomes and Costs of IBD Care: Results From the IBD Value Study

BACKGROUND

Data on variation in outcomes and costs of the treatment of inflammatory bowel disease (IBD) can be used to identify areas for cost and quality improvement. It can also help healthcare providers learn from each other and strive for equity in care. We aimed to assess the variation in outcomes and costs of IBD care between hospitals.

METHODS

We conducted a 12-month cohort study in 8 hospitals in the Netherlands. Patients with IBD who were treated with biologics and new small molecules were included. The percentage of variation in outcomes (following the International Consortium for Health Outcomes Measurement standard set) and costs attributable to the treating hospital were analyzed with intraclass correlation coefficients (ICCs) from case mix-adjusted (generalized) linear mixed models.

RESULTS

We included 1010 patients (median age 45 years, 55% female). Clinicians reported high remission rates (83%), while patient-reported rates were lower (40%). During the 12-month follow-up, 5.2% of patients used prednisolone for more than 3 months. Hospital costs (outpatient, inpatient, and medication costs) were substantial (median: €8323 per 6 months), mainly attributed to advanced therapies (€6611). Most of the variation in outcomes and costs among patients could not be attributed to the treating hospitals, with ICCs typically between 0% and 2%. Instead, patient-level characteristics, often with ICCs above 50%, accounted for these variations.

CONCLUSIONS

Variation in outcomes and costs cannot be used to differentiate between hospitals for quality of care. Future quality improvement initiatives should look at differences in structure and process measures of care and implement patient-level interventions to improve quality of IBD care.

TRIAL REGISTRATION NUMBER

NL8276.

Process indicators outshine outcome measures: assessing hospital quality of care in breast cancer treatment in China

Reporting the results of quality indicators can narrow the gap in the quality of care between hospitals. While most studies rely on outcome indicators, they may not accurately measure the quality of care. Process indicators are not only strongly associated with treatment outcomes, but are also more sensitive to whether patients are treated accurately, enabling timely intervention. Our study aims to investigate whether process indicators provide a more reasonable assessment of hospital quality of care compared to outcome indicators. Data were sourced from the Specific Disease Medical Service Quality Management and Control System in China. A total of 113,942 patients with breast cancer treated in 298 hospitals between January 2019 and April 2023 were included in this retrospective study. The rankability of 11 process indicators was calculated and used as a weight to create a new composite indicator. The composite indicators and outcome measures were compared using the O/E ratio categories. Finally, in order to determine the impact of different years on the results, a sensitivity analysis was conducted using bootstrap sampling. The rankability ( ρ ) values of the eleven process indicators showed significant differences, with the highest ρ value for preoperative cytological or histological examination before surgery (0.919). The ρ value for the outcome indicator was 0.011. The rankability-weighting method yielded a comprehensive score ( ρ  = 0.883). The comparison with categorical results of the outcome indicator has different performance classifications for 113 hospitals (37.92%) for composite scores and 140 (46.98%) for preoperative cytological or histological examinationbefore surgery. Process indicators are more suitable than outcome indicators for assessing the quality of breast cancer care in hospitals. Healthcare providers can use process indicators to identify specific areas for improvement, thereby driving continuous quality improvement efforts.

Standardized mortality ratios for regionalized acute cardiovascular care

BACKGROUND

Standardized Mortality Ratios (SMRs) are case-mix adjusted mortality rates per hospital and are used to evaluate quality of care. However, acute care is increasingly organized on a regional level, with more severe patients admitted to specialized hospitals. We hypothesize that the current case-mix adjustment insufficiently captures differences in case-mix between non-specialized and specialized hospitals. We aim to improve the SMR by adding proxies of disease severity to the model and by calculating a regional SMR (RSMR) for acute cerebrovascular disease (CVD) and myocardial infarction (MI).

METHODS

We used data from the Dutch National Basic Registration of Hospital Care. We selected all admissions from 2016 to 2018. SMRs and RSMRs were calculated by dividing the observed in-hospital mortality by the expected in-hospital mortality. The expected in-hospital mortality was calculated using logistic regression with adjustment for age, sex, socioeconomic status, severity of main diagnosis, urgency of admission, Charlson comorbidity index, place of residence before admission, month/year of admission, and in-hospital mortality as outcome.

RESULTS

The IQR of hospital SMRs of CVD was 0.85-1.10, median 0.94, with higher SMRs for specialized hospitals (median 1.12, IQR 1.00-1.28, 71%-SMR > 1) than for non-specialized hospitals (median 0.92, IQR 0.82-1.07, 32%-SMR > 1). The IQR of RSMRs was 0.92-1.09, median 1.00. The IQR of hospital SMRs of MI was 0.76-1.14, median 0.98, with higher SMRs for specialized hospitals (median 1.00, IQR 0.89-1.25, 50%-SMR > 1 versus median 0.94, IQR 0.74-1.11, 44%-SMR > 1). The IQR of RSMRs was 0.90-1.08, median 1.00. Adjustment for proxies of disease severity mostly led to lower SMRs of specialized hospitals.

CONCLUSION

SMRs of acute regionally organized diseases do not only measure differences in quality of care between hospitals, but merely measure differences in case-mix between hospitals. Although the addition of proxies of disease severity improves the model to calculate SMRs, real disease severity scores would be preferred. However, such scores are not available in administrative data. As a consequence, the usefulness of the current SMR as quality indicator is very limited. RSMRs are potentially more useful, since they fit regional organization and might be a more valid representation of quality of care.

A joint international consensus statement for measuring quality of survival for patients with childhood cancer

The aim of treating childhood cancer remains to cure all. As survival rates improve, long-term health outcomes increasingly define quality of care. The International Childhood Cancer Outcome Project developed a set of core outcomes for most types of childhood cancers involving relevant international stakeholders (survivors; pediatric oncologists; other medical, nursing or paramedical care providers; and psychosocial or neurocognitive care providers) to allow outcome-based evaluation of childhood cancer care. A survey among healthcare providers (n = 87) and online focus groups of survivors (n = 22) resulted in unique candidate outcome lists for 17 types of childhood cancer (five hematological malignancies, four central nervous system tumors and eight solid tumors). In a two-round Delphi survey, 435 healthcare providers from 68 institutions internationally (response rates for round 1, 70-97%; round 2, 65-92%) contributed to the selection of four to eight physical core outcomes (for example, heart failure, subfertility and subsequent neoplasms) and three aspects of quality of life (physical, psychosocial and neurocognitive) per pediatric cancer subtype. Measurement instruments for the core outcomes consist of medical record abstraction, questionnaires and linkage with existing registries. This International Childhood Cancer Core Outcome Set represents outcomes of value to patients, survivors and healthcare providers and can be used to measure institutional progress and benchmark against peers.

The impact of surgical volume on hospital ranking using the standardized infection ratio

The Centers for Medicare and Medicaid Services require hospitals to report on quality metrics which are used to financially penalize those that perform in the lowest quartile. Surgical site infections (SSIs) are a critical component of the quality metrics that target healthcare-associated infections. However, the accuracy of such hospital profiling is highly affected by small surgical volumes which lead to a large amount of uncertainty in estimating standardized hospital-specific infection rates. Currently, hospitals with less than one expected SSI are excluded from rankings, but the effectiveness of this exclusion criterion is unknown. Tools that can quantify the classification accuracy and can determine the minimal surgical volume required for a desired level of accuracy are lacking. We investigate the effect of surgical volume on the accuracy of identifying poorly performing hospitals based on the standardized infection ratio and develop simulation-based algorithms for quantifying the classification accuracy. We apply our proposed method to data from HCA Healthcare (2014-2016) on SSIs in colon surgery patients. We estimate that for a procedure like colon surgery with an overall SSI rate of 3%, to rank hospitals in the HCA colon SSI dataset, hospitals that perform less than 200 procedures have a greater than 10% chance of being incorrectly assigned to the worst performing quartile. Minimum surgical volumes and predicted events criteria are required to make evaluating hospitals reliable, and these criteria vary by overall prevalence and between-hospital variability.

Estimation of treatment effects in observational stroke care data: comparison of statistical approaches

Introduction

Various statistical approaches can be used to deal with unmeasured confounding when estimating treatment effects in observational studies, each with its own pros and cons. This study aimed to compare treatment effects as estimated by different statistical approaches for two interventions in observational stroke care data.

Patients and methods

We used prospectively collected data from the MR CLEAN registry including all patients (n = 3279) with ischemic stroke who underwent endovascular treatment (EVT) from 2014 to 2017 in 17 Dutch hospitals. Treatment effects of two interventions – i.e., receiving an intravenous thrombolytic (IVT) and undergoing general anesthesia (GA) before EVT – on good functional outcome (modified Rankin Scale ≤2) were estimated. We used three statistical regression-based approaches that vary in assumptions regarding the source of unmeasured confounding: individual-level (two subtypes), ecological, and instrumental variable analyses. In the latter, the preference for using the interventions in each hospital was used as an instrument.

Results

Use of IVT (range 66–87%) and GA (range 0–93%) varied substantially between hospitals. For IVT, the individual-level (OR ~ 1.33) resulted in significant positive effect estimates whereas in instrumental variable analysis no significant treatment effect was found (OR 1.11; 95% CI 0.58–1.56). The ecological analysis indicated no statistically significant different likelihood (β = − 0.002%; P = 0.99) of good functional outcome at hospitals using IVT 1% more frequently. For GA, we found non-significant opposite directions of points estimates the treatment effect in the individual-level (ORs ~ 0.60) versus the instrumental variable approach (OR = 1.04). The ecological analysis also resulted in a non-significant negative association (0.03% lower probability).

Discussion and conclusion

Both magnitude and direction of the estimated treatment effects for both interventions depend strongly on the statistical approach and thus on the source of (unmeasured) confounding. These issues should be understood concerning the specific characteristics of data, before applying an approach and interpreting the results. Instrumental variable analysis might be considered when unobserved confounding and practice variation is expected in observational multicenter studies.

Hospital Variation in Time to Endovascular Treatment for Ischemic Stroke: What Is the Optimal Target for Improvement?

Background

Time to reperfusion in patients with ischemic stroke is strongly associated with functional outcome and may differ between hospitals and between patients within hospitals. Improvement in time to reperfusion can be guided by between‐hospital and within‐hospital comparisons and requires insight in specific targets for improvement. We aimed to quantify the variation in door‐to‐reperfusion time between and within Dutch intervention hospitals and to assess the contribution of different time intervals to this variation.

Methods and Results

We used data from the MR CLEAN (Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands) Registry. The door‐to‐reperfusion time was subdivided into time intervals, separately for direct patients (door‐to‐computed tomography, computed tomography‐to‐computed tomography angiography [CTA], CTA‐to‐groin, and groin‐to‐reperfusion times) and for transferred patients (door‐to‐groin and groin‐to‐reperfusion times). We used linear mixed models to distinguish the variation in door‐to‐reperfusion time between hospitals and between patients. The proportional change in variance was used to estimate the amount of variance explained by each time interval. We included 2855 patients of 17 hospitals providing endovascular treatment. Of these patients, 44% arrived directly at an endovascular treatment hospital. The between‐hospital variation in door‐to‐reperfusion time was 9%, and the within‐hospital variation was 91%. The contribution of case‐mix variables on the variation in door‐to‐reperfusion time was marginal (2%–7%). Of the between‐hospital variation, CTA‐to‐groin time explained 83%, whereas groin‐to‐reperfusion time explained 15%. Within‐hospital variation was mostly explained by CTA‐to‐groin time (33%) and groin‐to‐reperfusion time (42%). Similar results were found for transferred patients.

Conclusions

Door‐to‐reperfusion time varies between, but even more within, hospitals providing endovascular treatment for ischemic stroke. Quality of stroke care improvements should not only be guided by between‐hospital comparisons, but also aim to reduce variation between patients within a hospital, and should specifically focus on CTA‐to‐groin time and groin‐to‐reperfusion time.

Between-centre differences in care for in-hospital cardiac arrest: a prospective cohort study

BACKGROUND

Survival after in-hospital cardiac arrest is poor, but current literature shows substantial heterogeneity in reported survival rates. This study aims to evaluate care for patients suffering in-hospital cardiac arrest (IHCA) in the Netherlands by assessing between-hospital heterogeneity in outcomes and to explain this heterogeneity stemming from differences in case-mix or differences in quality of care.

METHODS

A prospective multicentre study was conducted comprising 14 centres. All IHCA patients were included. The adjusted variation in structure and process indicators of quality of care and outcomes (in-hospital mortality and cerebral performance category [CPC] scale) was assessed with mixed effects regression with centre as random intercept. Variation was quantified using the median odds ratio (MOR), representing the expected odds ratio for poor outcome between two randomly picked centres.

RESULTS

After excluding centres with less than 10 inclusions (2 centres), 701 patients were included of whom, 218 (32%) survived to hospital discharge. The unadjusted and case-mix adjusted MOR for mortality was 1.19 and 1.05, respectively. The unadjusted and adjusted MOR for CPC score was 1.24 and 1.19, respectively. In hospitals where personnel received cardiopulmonary resuscitation (CPR) training twice per year, 183 (64.7%) versus 290 (71.4%) patients died or were in a vegetative state, and 59 (20.8%) versus 68 (16.7%) patients showed full recovery (p < 0.001).

CONCLUSION

In the Netherlands, survival after IHCA is relatively high and between-centre differences in outcomes are small. The existing differences in survival are mainly attributable to differences in case-mix. Variation in neurological outcome is less attributable to case-mix.

Administrative Data Are Unreliable for Ranking Hospital Performance Based on Serious Complications After Spine Fusion

STUDY DESIGN

Retrospective analysis of administrative billing data.

OBJECTIVE

To evaluate the extent to which a metric of serious complications determined from administrative data can reliably profile hospital performance in spine fusion surgery.

SUMMARY OF BACKGROUND DATA

While payers are increasingly focused on implementing pay-for-performance measures, quality metrics must reliably reflect true differences in performance among the hospitals profiled.

METHODS

We used State Inpatient Databases from nine states to characterize serious complications after elective cervical and thoracolumbar fusion. Hierarchical logistic regression was used to risk-adjust differences in case mix, along with variability from low case volumes. The reliability of this risk-stratified complication rate (RSCR) was assessed as the variation between hospitals that was not due to chance alone, calculated separately by fusion type and year. Finally, we estimated the proportion of hospitals that had sufficient case volumes to obtain reliable (>0.7) complication estimates.

RESULTS

From 2010 to 2017 we identified 154,078 cervical and 213,133 thoracolumbar fusion surgeries. 4.2% of cervical fusion patients had a serious complication, and the median RSCR increased from 4.2% in 2010 to 5.5% in 2017. The reliability of the RSCR for cervical fusion was poor and varied substantially by year (range 0.04-0.28). Overall, 7.7% of thoracolumbar fusion patients experienced a serious complication, and the RSCR varied from 6.8% to 8.0% during the study period. Although still modest, the RSCR reliability was higher for thoracolumbar fusion (range 0.16-0.43). Depending on the study year, 0% to 4.5% of hospitals had sufficient cervical fusion case volume to report reliable (>0.7) estimates, whereas 15% to 36% of hospitals reached this threshold for thoracolumbar fusion.

CONCLUSION

A metric of serious complications was unreliable for benchmarking cervical fusion outcomes and only modestly reliable for thoracolumbar fusion. When assessed using administrative datasets, these measures appear inappropriate for high-stakes applications, such as public reporting or pay-for-performance.Level of Evidence: 3.

Ordinal outcome analysis improves the detection of between-hospital differences in outcome

Background

There is a growing interest in assessment of the quality of hospital care, based on outcome measures. Many quality of care comparisons rely on binary outcomes, for example mortality rates. Due to low numbers, the observed differences in outcome are partly subject to chance. We aimed to quantify the gain in efficiency by ordinal instead of binary outcome analyses for hospital comparisons. We analyzed patients with traumatic brain injury (TBI) and stroke as examples.

Methods

We sampled patients from two trials. We simulated ordinal and dichotomous outcomes based on the modified Rankin Scale (stroke) and Glasgow Outcome Scale (TBI) in scenarios with and without true differences between hospitals in outcome. The potential efficiency gain of ordinal outcomes, analyzed with ordinal logistic regression, compared to dichotomous outcomes, analyzed with binary logistic regression was expressed as the possible reduction in sample size while keeping the same statistical power to detect outliers.

Results

In the IMPACT study (9578 patients in 265 hospitals, mean number of patients per hospital = 36), the analysis of the ordinal scale rather than the dichotomized scale (‘unfavorable outcome’), allowed for up to 32% less patients in the analysis without a loss of power. In the PRACTISE trial (1657 patients in 12 hospitals, mean number of patients per hospital = 138), ordinal analysis allowed for 13% less patients. Compared to mortality, ordinal outcome analyses allowed for up to 37 to 63% less patients.

Conclusions

Ordinal analyses provide the statistical power of substantially larger studies which have been analyzed with dichotomization of endpoints. We advise to exploit ordinal outcome measures for hospital comparisons, in order to increase efficiency in quality of care measurements.

Trial registration

We do not report the results of a health care intervention.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-020-01185-7.

Improving quality of stroke care through benchmarking center performance: why focusing on outcomes is not enough

Background

Between-center variation in outcome may offer opportunities to identify variation in quality of care. By intervening on these quality differences, patient outcomes may be improved. However, whether observed differences in outcome reflect the true quality improvement potential is not known for many diseases. Therefore, we aimed to analyze the effect of differences in performance on structure and processes of care, and case-mix on between-center differences in outcome after endovascular treatment (EVT) for ischemic stroke.

Methods

In this observational cohort study, ischemic stroke patients who received EVT between 2014 and 2017 in all 17 Dutch EVT-centers were included. Primary outcome was the modified Rankin Scale, ranging from 0 (no symptoms) to 6 (death), at 90 days. We used random effect proportional odds regression modelling, to analyze the effect of differences in structure indicators (center volume and year of admission), process indicators (time to treatment and use of general anesthesia) and case-mix, by tracking changes in tau², which represents the amount of between-center variation in outcome.

Results

Three thousand two hundred seventy-nine patients were included. Performance on structure and process indicators varied significantly between EVT-centers (P < 0.001). Predicted probability of good functional outcome (modified Rankin Scale 0–2 at 90 days), which can be interpreted as an overall measure of a center’s case-mix, varied significantly between 17 and 50% across centers. The amount of between-center variation (tau²) was estimated at 0.040 in a model only accounting for random variation. This estimate more than doubled after adding case-mix variables (tau²: 0.086) to the model, while a small amount of between-center variation was explained by variation in performance on structure and process indicators (tau²: 0.081 and 0.089, respectively). This indicates that variation in case-mix affects the differences in outcome to a much larger extent.

Conclusions

Between-center variation in outcome of ischemic stroke patients mostly reflects differences in case-mix, rather than differences in structure or process of care. Since the latter two capture the real quality improvement potential, these should be used as indicators for comparing center performance. Especially when a strong association exists between those indicators and outcome, as is the case for time to treatment in ischemic stroke.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-020-05841-y.

Between-center and between-country differences in outcome after aneurysmal subarachnoid hemorrhage in the Subarachnoid Hemorrhage International Trialists (SAHIT) repository

OBJECTIVE

Differences in clinical outcomes between centers and countries may reflect variation in patient characteristics, diagnostic and therapeutic policies, or quality of care. The purpose of this study was to investigate the presence and magnitude of between-center and between-country differences in outcome after aneurysmal subarachnoid hemorrhage (aSAH).

METHODS

The authors analyzed data from 5972 aSAH patients enrolled in randomized clinical trials of 3 different treatments from the Subarachnoid Hemorrhage International Trialists (SAHIT) repository, including data from 179 centers and 20 countries. They used random effects logistic regression adjusted for patient characteristics and timing of aneurysm treatment to estimate between-center and between-country differences in unfavorable outcome, defined as a Glasgow Outcome Scale score of 1-3 (severe disability, vegetative state, or death) or modified Rankin Scale score of 4-6 (moderately severe disability, severe disability, or death) at 3 months. Between-center and between-country differences were quantified with the median odds ratio (MOR), which can be interpreted as the ratio of odds of unfavorable outcome between a typical high-risk and a typical low-risk center or country.

RESULTS

The proportion of patients with unfavorable outcome was 27% (n = 1599). The authors found substantial between-center differences (MOR 1.26, 95% CI 1.16-1.52), which could not be explained by patient characteristics and timing of aneurysm treatment (adjusted MOR 1.21, 95% CI 1.11-1.44). They observed no between-country differences (adjusted MOR 1.13, 95% CI 1.00-1.40).

CONCLUSIONS

Clinical outcomes after aSAH differ between centers. These differences could not be explained by patient characteristics or timing of aneurysm treatment. Further research is needed to confirm the presence of differences in outcome after aSAH between hospitals in more recent data and to investigate potential causes.

Quality indicators for patients with traumatic brain injury in European intensive care units: a CENTER-TBI study

BACKGROUND

The aim of this study is to validate a previously published consensus-based quality indicator set for the management of patients with traumatic brain injury (TBI) at intensive care units (ICUs) in Europe and to study its potential for quality measurement and improvement.

METHODS

Our analysis was based on 2006 adult patients admitted to 54 ICUs between 2014 and 2018, enrolled in the CENTER-TBI study. Indicator scores were calculated as percentage adherence for structure and process indicators and as event rates or median scores for outcome indicators. Feasibility was quantified by the completeness of the variables. Discriminability was determined by the between-centre variation, estimated with a random effect regression model adjusted for case-mix severity and quantified by the median odds ratio (MOR). Statistical uncertainty of outcome indicators was determined by the median number of events per centre, using a cut-off of 10.

RESULTS

A total of 26/42 indicators could be calculated from the CENTER-TBI database. Most quality indicators proved feasible to obtain with more than 70% completeness. Sub-optimal adherence was found for most quality indicators, ranging from 26 to 93% and 20 to 99% for structure and process indicators. Significant (p < 0.001) between-centre variation was found in seven process and five outcome indicators with MORs ranging from 1.51 to 4.14. Statistical uncertainty of outcome indicators was generally high; five out of seven had less than 10 events per centre.

CONCLUSIONS

Overall, nine structures, five processes, but none of the outcome indicators showed potential for quality improvement purposes for TBI patients in the ICU. Future research should focus on implementation efforts and continuous reevaluation of quality indicators.

TRIAL REGISTRATION

The core study was registered with ClinicalTrials.gov, number NCT02210221, registered on August 06, 2014, with Resource Identification Portal (RRID: SCR_015582).

Between-Hospital Variation in Revision Rates After Total Hip and Knee Arthroplasty in the Netherlands: Directing Quality-Improvement Initiatives

BACKGROUND

Variation in 1-year revision rates between Dutch hospitals after primary total hip and knee arthroplasty (THA and TKA) may direct quality-improvement initiatives if this variation accurately reflects true hospital differences. The aim of the present study was to assess the extent of variation, both overall and for specific indications, as well as the statistical reliability of ranking hospitals.

METHODS

All primary THAs and TKAs that were performed between January 2014 and December 2016 were included. Observed/expected (O/E) ratios regarding 1-year revision rates were depicted in a funnel plot with 95% control limits to identify outliers based on 1 or 3 years of data, both overall and by specific indication for revision. The expected number was calculated on the basis of patient mix with use of logistic regression models. The statistical reliability of ranking hospitals (rankability) on these outcomes indicates the percentage of total variation that is explained by "true" hospital differences rather than chance. Rankability was evaluated using fixed and random effects models, for overall revisions and specific indications for revision, including 1 versus 3 years of data.

RESULTS

The present study included 86,468 THAs and 73,077 TKAs from 97 and 98 hospitals, respectively. Thirteen hospitals performing THAs were identified as negative outliers (median O/E ratio, 1.9; interquartile range [IQR], 1.5-2.5), with 5 hospitals as outliers in multiple years. Eight negative outliers were identified for periprosthetic joint infection; 4, for dislocation; and 2, for prosthesis loosening. Seven hospitals performing TKAs were identified as negative outliers (median O/E ratio, 2.3; IQR, 2.2-2.8), with 2 hospitals as outliers in multiple years. Two negative outlier hospitals were identified for periprosthetic joint infection and 1 was identified for technical failures. The rankability for overall revisions was 62% (moderate) for THA and 46% (low) for TKA.

CONCLUSIONS

There was large between-hospital variation in 1-year revision rates after primary THA and TKA. For most outlier hospitals, a specific indication for revision could be identified as contributing to worse performance, particularly for THA; these findings are starting points for quality-improvement initiatives.

Ranking hospital performance based on individual indicators: can we increase reliability by creating composite indicators?

BACKGROUND

Report cards on the health care system increasingly report provider-specific performance on indicators that measure the quality of health care delivered. A natural reaction to the publishing of hospital-specific performance on a given indicator is to create 'league tables' that rank hospitals according to their performance. However, many indicators have been shown to have low to moderate rankability, meaning that they cannot be used to accurately rank hospitals. Our objective was to define conditions for improving the ability to rank hospitals by combining several binary indicators with low to moderate rankability.

METHODS

Monte Carlo simulations to examine the rankability of composite ordinal indicators created by pooling three binary indicators with low to moderate rankability. We considered scenarios in which the prevalences of the three binary indicators were 0.05, 0.10, and 0.25 and the within-hospital correlation between these indicators varied between - 0.25 and 0.90.

RESULTS

Creation of an ordinal indicator with high rankability was possible when the three component binary indicators were strongly correlated with one another (the within-hospital correlation in indicators was at least 0.5). When the binary indicators were independent or weakly correlated with one another (the within-hospital correlation in indicators was less than 0.5), the rankability of the composite ordinal indicator was often less than at least one of its binary components. The rankability of the composite indicator was most affected by the rankability of the most prevalent indicator and the magnitude of the within-hospital correlation between the indicators.

CONCLUSIONS

Pooling highly-correlated binary indicators can result in a composite ordinal indicator with high rankability. Otherwise, the composite ordinal indicator may have lower rankability than some of its constituent components. It is recommended that binary indicators be combined to increase rankability only if they represent the same concept of quality of care.

Ranking hospitals: do we gain reliability by using composite rather than individual indicators?

Background

Despite widespread use of quality indicators, it remains unclear to what extent they can reliably distinguish hospitals on true differences in performance. Rankability measures what part of variation in performance reflects ‘true’ hospital differences in outcomes versus random noise.

Objective

This study sought to assess whether combining data into composites or including data from multiple years improves the reliability of ranking quality indicators for hospital care.

Methods

Using the Dutch National Medical Registration (2007–2012) for stroke, colorectal carcinoma, heart failure, acute myocardial infarction and total hiparthroplasty (THA)/ total knee arthroplasty (TKA) in osteoarthritis (OA), we calculated the rankability for in-hospital mortality, 30-day acute readmission and prolonged length of stay (LOS) for single years and 3-year periods and for a dichotomous and ordinal composite measure in which mortality, readmission and prolonged LOS were combined. Rankability, defined as (between-hospital variation/between-hospital+within hospital variation)×100% is classified as low (<50%), moderate (50%–75%) and high (>75%).

Results

Admissions from 555 053 patients treated in 95 hospitals were included. The rankability for mortality was generally low or moderate, varying from less than 1% for patients with OA undergoing THA/TKA in 2011 to 71% for stroke in 2010. Rankability for acute readmission was low, except for acute myocardial infarction in 2009 (51%) and 2012 (62%). Rankability for prolonged LOS was at least moderate. Combining multiple years improved rankability but still remained low in eight cases for both mortality and acute readmission. Combining the individual indicators into the dichotomous composite, all diagnoses had at least moderate rankability (range: 51%–96%). For the ordinal composite, only heart failure had low rankability (46% in 2008) (range: 46%–95%).

Conclusion

Combining multiple years or into multiple indicators results in more reliable ranking of hospitals, particularly compared with mortality and acute readmission in single years, thereby improving the ability to detect true hospital differences. The composite measures provide more information and more reliable rankings than combining multiple years of individual indicators.

Volume-outcome revisited: The effect of hospital and surgeon volumes on multiple outcome measures in oesophago-gastric cancer surgery

Background

Most studies showing a volume outcome effect in resection surgery for oesophago-gastric cancer were conducted before the centralisation of clinical services. This study evaluated the relation between hospital- and surgeon volume and different risk-adjusted outcomes after oesophago-gastric (OG) cancer surgery in England between 2011 and 2013.

Methods

In data from the National Oesophago-Gastric Cancer Audit from the UK, multivariable random-effects logistic regression models were used to quantify the effect of surgeon and hospital volume on three outcomes: 30-day and 90-day mortality and anastomotic leakage. The models included patient risk factors to adjust for differences in case-mix among hospitals and surgeons. The between-cluster heterogeneity was estimated with the median odds ratio (MOR).

Results

The study included patients treated at 42 hospitals and 329 surgeons. The median (interquartile range) of the annual hospital and surgeon volumes were 110 patients (82 to 137) and 13 patients (8 to 19), respectively. The overall rates for 30-day and 90-day mortality were 2.3% and 4.4% respectively, and the anastomotic leakage was 6.3%. Higher hospital volume was associated with lower 30-day mortality (OR: 0.94; 95% CI: 0.91–0.98) and lower anastomotic leakage rates (OR: 0.96; 95% CI: 0.93–0.98) but not 90-day mortality. Higher surgeon volume was only associated with lower anastomotic leakage rates (OR: 0.81; 95% CI: 0.72–0.92). Hospital volume explained a part of the between-hospital variation in 30-day mortality whereas surgeon volume explained part of the between-hospital variation in anastomotic leakage.

Conclusions

In the setting of centralized O-G cancer surgery in England, we could still observe an effect of volume on short-term outcomes. However, the effect is inconsistent, depending on the type of outcome measure under consideration, and much smaller than in previous studies. Efforts to centralise O-G cancer services further should carefully address the effects of both hospital and surgeon volume on the range of outcome measures that are relevant to patients.

Surgical Site Infections: Volume-Outcome Relationship and Year-to-Year Stability of Performance Rankings

BACKGROUND

Surgical site infection (SSI) rates are publicly reported as quality metrics and increasingly used to determine financial reimbursement.

OBJECTIVE

To evaluate the volume-outcome relationship as well as the year-to-year stability of performance rankings following coronary artery bypass graft (CABG) surgery and hip arthroplasty.

RESEARCH DESIGN

We performed a retrospective cohort study of Medicare beneficiaries who underwent CABG surgery or hip arthroplasty at US hospitals from 2005 to 2011, with outcomes analyzed through March 2012. Nationally validated claims-based surveillance methods were used to assess for SSI within 90 days of surgery. The relationship between procedure volume and SSI rate was assessed using logistic regression and generalized additive modeling. Year-to-year stability of SSI rates was evaluated using logistic regression to assess hospitals' movement in and out of performance rankings linked to financial penalties.

RESULTS

Case-mix adjusted SSI risk based on claims was highest in hospitals performing <50 CABG/year and <200 hip arthroplasty/year compared with hospitals performing ≥200 procedures/year. At that same time, hospitals in the worst quartile in a given year based on claims had a low probability of remaining in that quartile the following year. This probability increased with volume, and when using 2 years' experience, but the highest probabilities were only 0.59 for CABG (95% confidence interval, 0.52-0.66) and 0.48 for hip arthroplasty (95% confidence interval, 0.42-0.55).

CONCLUSIONS

Aggregate SSI risk is highest in hospitals with low annual procedure volumes, yet these hospitals are currently excluded from quality reporting. Even for higher volume hospitals, year-to-year random variation makes past experience an unreliable estimator of current performance.

Hospital Variation in Functional Recovery After Stroke

Background—

Functional status is a key patient-centric outcome, but there are little data on whether functional recovery post-stroke varies among hospitals. This study examined the distribution of functional status 3 months after stroke, determined whether these outcomes vary among hospitals, and identified hospital characteristics associated with better (or worse) functional outcomes.

Methods and Results—

Observational analysis of the AVAIL study (Adherence Evaluation After Ischemic Stroke-Longitudinal) included 2083 ischemic stroke patients enrolled from 82 US hospitals participating in Get With The Guidelines-Stroke and AVAIL. The primary outcome was dependence or death at 3 months (modified Rankin Scale [mRS] score of 3–6). Secondary outcomes included functional dependence (mRS score of 3–5), disabled (mRS score of 2–5), and mRS evaluated as a continuous score. By 3 months post-discharge, 36.5% of patients were functionally dependent or dead. Rates of dependence or death varied widely by discharging hospitals (range: 0%–67%). After risk adjustment, patients had lower rates of 3-month dependence or death when treated at teaching hospitals (odds ratio, 0.72; 95% confidence interval, 0.54–0.96) and certified primary stroke centers (odds ratio, 0.69; 95% confidence interval, 0.53–0.91). In contrast, a composite measure of hospital-level adherence to acute stroke care performance metrics, stroke volume, and bed size was not associated with downstream patient functional status. Findings were robust across mRS end points and sensitivity analyses.

Conclusions—

One third of acute ischemic stroke patients were functionally dependent or dead 3 months postacute stroke; functional recovery rates varied considerably among hospitals, supporting the need to better determine which care processes can maximize functional outcomes.

Individual and Clustered Rankability of ICUs According to Case-Mix-Adjusted Mortality

OBJECTIVES

The performance of ICUs can be compared by ranking them into a league table according to their risk-adjusted mortality rate. The statistical quality of a league table can be expressed as its rankability, the percentage of variation between ICUs attributable to unexplained differences. We examine whether we can improve the rankability of our league table by using data from a longer period or by grouping ICUs with similar performance constructing a league table of clusters rather than individual ICUs.

DESIGN

We developed a league table for risk-adjusted mortality rate with its rankability. The effect of assessment period was determined using a resampling procedure. Hierarchical clustering was used to obtain clusters of similar ICUs.

PATIENTS

We used data from ICUs participating in the Dutch National Intensive Care Evaluation registry between 2011 and 2013.

MEASUREMENTS AND MAIN RESULTS

We constructed league tables using 157,394 admissions from 78 ICUs with risk-adjusted mortality rate between 5.9% and 13.9% per ICU over the inclusion period. The rankability was 73% for 2013 and 89% for the whole period 2011-2013. Rankability over the year 2013 increased till 98% when clustering ICUs, reaching an optimum at a league table of seven clusters.

CONCLUSIONS

We conclude that, when using data from a single year, the rankability of a league table of Dutch ICUs based on risk-adjusted mortality rate was unacceptably low. We could improve the rankability of this league table by increasing the period of data collection or by grouping similar ICUs into clusters and constructing a league table of clusters of ICUs rather than individual ICUs. Ranking clusters of ICUs could be useful for identifying possible differences in performance between clusters of ICUs.

Benchmarking operating room departments in the Netherlands

Purpose

– Benchmarking is increasingly considered a useful management instrument to improve performance in healthcare. The purpose of this paper is to assess if a nationwide long-term benchmarking collaborative between operating room (OR) departments of university medical centres in the Netherlands leads to benefits in OR management and to evaluate if the initiative meets the requirements of the 4P-model.

Design/methodology/approach

– The evaluation was based on the 4P-model (purposes, performance indicators, participating organisations, performance management system), developed in former studies. A mixed-methods design was applied, consisting of document study, observations, interviews as well as analysing OR performance data using SPSS statistics.

Findings

– Collaborative benchmarking has benefits different from mainly performance improvement and identification of performance gaps. It is interesting that, since 2004, the OR benchmarking initiative still endures after already existing for ten years. A key benefit was pointed out by all respondents as “the purpose of networking”, on top of the purposes recognised in the 4P-model. The networking events were found to make it easier for participants to contact and also visit one another. Apparently, such informal contacts were helpful in spreading knowledge, sharing policy documents and initiating improvement. This benchmark largely met all key conditions of the 4P-model.

Research limitations/implications

– The current study has the limitations accompanied with any qualitative research and particularly related to interviewing. Qualitative research findings must be viewed within the context of the conducted case study. The experiences in this university hospital context in the Netherlands might not be transferable to other (general) hospital settings or other countries. The number of conducted interviews is restricted; nevertheless, all other data sources are extensive.

Originality/value

– A collaborative approach in benchmarking can be effective because participants use its knowledge-sharing infrastructure which enables operational, tactical and strategic learning. Organisational learning is to the advantage of overall OR management. Benchmarking seems a useful instrument in enabling hospitals to learn from each other, to initiate performance improvements and catalyse knowledge-sharing.

Improved Prediction by Dynamic Modeling: An Exploratory Study in the Adult Cardiac Surgery Database of the Netherlands Association for Cardio-Thoracic Surgery

BACKGROUND

The predictive performance of static risk prediction models such as EuroSCORE deteriorates over time. We aimed to explore different methods for continuous updating of EuroSCORE (dynamic modeling) to improve risk prediction.

METHODS AND RESULTS

Data on adult cardiac surgery from 2007 to 2012 (n=95 240) were extracted from the Netherlands Association for Cardio-Thoracic Surgery database. The logistic EuroSCORE predicting in-hospital death was updated using 6 methods: recalibrating the intercept of the logistic regression model; recalibrating the intercept and joint effects of the prognostic factors; re-estimating all prognostic factor effects, re-estimating all prognostic factor effects, and applying shrinkage of the estimates; applying a test procedure to select either of these; and a Bayesian learning strategy. Models were updated with 1 or 3 years of data, in all cardiac surgery or within operation subgroups. Performance was tested in the subsequent year according to discrimination (area under the receiver operating curve, area under the curve) and calibration (calibration slope and calibration-in-the-large). Compared with the original EuroSCORE, all updating methods resulted in improved calibration-in-the-large (range -0.17 to 0.04 versus -1.13 to -0.97, ideally 0.0). Calibration slope (range 0.92-1.15) and discrimination (area under the curve range 0.83-0.87) were similar across methods. In small subgroups, such as aortic valve replacement and aortic valve replacement+coronary artery bypass grafting, extensive updating using 1 year of data led to poorer performance than using the original EuroSCORE. The choice of updating method had little effect on benchmarking results of all cardiac surgery.

CONCLUSIONS

Several methods for dynamic modeling may result in good discrimination and superior calibration compared with the original EuroSCORE. For large populations, all methods are appropriate. For smaller subgroups, it is recommended to use data from multiple years or a Bayesian approach.

Risk adjustment models for short-term outcomes after surgical resection for oesophagogastric cancer

BACKGROUND

Outcomes for oesophagogastric cancer surgery are compared with the aim of benchmarking quality of care. Adjusting for patient characteristics is crucial to avoid biased comparisons between providers. The study objective was to develop a case-mix adjustment model for comparing 30- and 90-day mortality and anastomotic leakage rates after oesophagogastric cancer resections.

METHODS

The study reviewed existing models, considered expert opinion and examined audit data in order to select predictors that were consequently used to develop a case-mix adjustment model for the National Oesophago-Gastric Cancer Audit, covering England and Wales. Models were developed on patients undergoing surgical resection between April 2011 and March 2013 using logistic regression. Model calibration and discrimination was quantified using a bootstrap procedure.

RESULTS

Most existing risk models for oesophagogastric resections were methodologically weak, outdated or based on detailed laboratory data that are not generally available. In 4882 patients with oesophagogastric cancer used for model development, 30- and 90-day mortality rates were 2·3 and 4·4 per cent respectively, and 6·2 per cent of patients developed an anastomotic leak. The internally validated models, based on predictors selected from the literature, showed moderate discrimination (area under the receiver operating characteristic (ROC) curve 0·646 for 30-day mortality, 0·664 for 90-day mortality and 0·587 for anastomotic leakage) and good calibration.

CONCLUSION

Based on available data, three case-mix adjustment models for postoperative outcomes in patients undergoing curative surgery for oesophagogastric cancer were developed. These models should be used for risk adjustment when assessing hospital performance in the National Health Service, and tested in other large health systems.

A Modelling Approach to Estimate the Impact of Sodium Reduction in Soups on Cardiovascular Health in the Netherlands

Hypertension is a major modifiable risk factor for cardiovascular disease and mortality, which could be lowered by reducing dietary sodium. The potential health impact of a product reformulation in the Netherlands was modelled, selecting packaged soups containing on average 25% less sodium as an example of an achievable product reformulation when implemented gradually. First, the blood pressure lowering resulting from sodium intake reduction was modelled. Second, the predicted blood pressure lowering was translated into potentially preventable incidence and mortality cases from stroke, acute myocardial infarction (AMI), angina pectoris, and heart failure (HF) implementing one year salt reduction. Finally, the potentially preventable subsequent lifetime Disability-Adjusted Life Years (DALYs) were calculated. The sodium reduction in soups might potentially reduce the incidence and mortality of stroke by approximately 0.5%, AMI and angina by 0.3%, and HF by 0.2%. The related burden of disease could be reduced by approximately 800 lifetime DALYs. This modelling approach can be used to provide insight into the potential public health impact of sodium reduction in specific food products. The data demonstrate that an achievable food product reformulation to reduce sodium can potentially benefit public health, albeit modest. When implemented across multiple product categories and countries, a significant health impact could be achieved.

Assessing the service quality of Iran military hospitals: Joint Commission International standards and Analytic Hierarchy Process (AHP) technique

BACKGROUND

Military hospitals are responsible for preserving, restoring and improving the health of not only armed forces, but also other people. According to the military organizations strategy, which is being a leader and pioneer in all areas, providing quality health services is one of the main goals of the military health care organizations. This study was aimed to evaluate the service quality of selected military hospitals in Iran based on the Joint Commission International (JCI) standards and comparing these hospitals with each other and ranking them using the analytic hierarchy process (AHP) technique in 2013.

MATERIALS AND METHODS

This was a cross-sectional and descriptive study conducted on five military hospitals, selected using the purposive sampling method, in 2013. Required data collected using checklists of accreditation standards and nominal group technique. AHP technique was used for prioritizing. Furthermore, Expert Choice 11.0 was used to analyze the collected data.

RESULTS

Among JCI standards, the standards of access to care and continuity of care (weight = 0.122), quality improvement and patient safety (weight = 0.121) and leadership and management (weight = 0.117) had the greatest importance, respectively. Furthermore, in the overall ranking, BGT (weight = 0.369), IHM (0.238), SAU (0.202), IHK (weight = 0.125) and SAB (weight = 0.066) ranked first to fifth, respectively.

CONCLUSION

AHP is an appropriate technique for measuring the overall performance of hospitals and their quality of services. It is a holistic approach that takes all hospital processes into consideration. The results of the present study can be used to improve hospitals performance through identifying areas, which are in need of focus for quality improvement and selecting strategies to improve service quality.

Ranking and rankability of hospital postoperative mortality rates in colorectal cancer surgery

OBJECTIVES

To examine to what extent random variation and variation in case-mix influence hospital rankings on the basis of mortality rates and to determine the suitability of mortality for ranking hospitals in colorectal surgery.

BACKGROUND

Comparing and ranking postoperative mortality rates between hospitals becomes increasingly popular. Differences in hospital case-mix, and chance variation related to caseload, may influence rankings. The suitability of mortality for rankings remains unclear.

METHODS

Data were derived from the Dutch Surgical Colorectal Audit. Hospital rankings based on fixed- and random-effects logistic regression models, unadjusted and adjusted for case-mix were compared with the percentile based on expected ranks (the chance that a hospital performs better than a random hospital). Rankability, measuring which part of variation between hospitals is not due to chance, was calculated.

RESULTS

Some 25,591 patients undergoing colorectal resections in 92 hospitals were evaluated. Postoperative mortality rates ranged between 0% and 8.8%. Adjustment for case-mix with a fixed-effects model caused large changes in rankings. A smaller additional effect on changes in rankings occurred after adjusting with a random-effects model, with lower volume hospitals moving toward the mean. Percentile based on expected ranks ranged between 10% and 85%. Rankability was 38%, meaning that 62% of hospital variation in mortality was due to chance.

CONCLUSIONS

Hospital ranks changed after case-mix adjustment and random-effects models, compared with unadjusted analysis. A large proportion of hospital variation in mortality was due to chance. Caution should be warranted when interpreting hospital rankings on the basis of postoperative mortality. Percentiles of expected ranks may help identify hospitals with exceptional performance.

Optimal age to start preventive measures in women with BRCA1/2 mutations or high familial breast cancer risk

Women from high-risk families consider preventive measures for breast cancer including screening. Guidelines on screening differ considerably regarding starting age. We investigated whether age at diagnosis in affected relatives is predictive for age at diagnosis. We analyzed the age of breast cancer detection of 1,304 first- and second-degree relatives of 314 BRCA1, 164 BRCA2 and 244 high-risk participants of the Dutch MRI-SCreening study. The within- and between-family variance in the relative's age at diagnosis was analyzed with a random effect linear regression model. We compared the starting age of screening based on risk-group (25 years for BRCA1, 30 years for BRCA2 and 35 years for familial risk), on family history, and on the model, which combines both. The findings were validated in 63 families from the UK-MARIBS study. Mean age at diagnosis in the relatives varied between families; 95% range of mean family ages was 35-55 in BRCA1-, 41-57 in BRCA2- and 44-60 in high-risk families. In all, 14% of the variance in age at diagnosis, in BRCA1 even 23%, was explained by family history, 7% by risk group. Determining start of screening based on the model and on risk-group gave similar results in terms of cancers missed and years of screening. The approach based on familial history only, missed more cancers and required more screening years in both the Dutch and the United Kingdom data sets. Age at breast cancer diagnosis is partly dependent on family history which may assist planning starting age for preventive measures.

Should measures of patient experience in primary care be adjusted for case mix? Evidence from the English General Practice Patient Survey

Objectives

Uncertainties exist about when and how best to adjust performance measures for case mix. Our aims are to quantify the impact of case-mix adjustment on practice-level scores in a national survey of patient experience, to identify why and when it may be useful to adjust for case mix, and to discuss unresolved policy issues regarding the use of case-mix adjustment in performance measurement in health care.

Design/setting

Secondary analysis of the 2009 English General Practice Patient Survey. Responses from 2 163 456 patients registered with 8267 primary care practices. Linear mixed effects models were used with practice included as a random effect and five case-mix variables (gender, age, race/ethnicity, deprivation, and self-reported health) as fixed effects.

Main outcome measures

Primary outcome was the impact of case-mix adjustment on practice-level means (adjusted minus unadjusted) and changes in practice percentile ranks for questions measuring patient experience in three domains of primary care: access; interpersonal care; anticipatory care planning, and overall satisfaction with primary care services.

Results

Depending on the survey measure selected, case-mix adjustment changed the rank of between 0.4% and 29.8% of practices by more than 10 percentile points. Adjusting for case-mix resulted in large increases in score for a small number of practices and small decreases in score for a larger number of practices. Practices with younger patients, more ethnic minority patients and patients living in more socio-economically deprived areas were more likely to gain from case-mix adjustment. Age and race/ethnicity were the most influential adjustors.

Conclusions

While its effect is modest for most practices, case-mix adjustment corrects significant underestimation of scores for a small proportion of practices serving vulnerable patients and may reduce the risk that providers would ‘cream-skim’ by not enrolling patients from vulnerable socio-demographic groups.

Limitations of Ranking Lists Based on Cardiac Surgery Mortality Rates

Background—

Ranking lists are a common way of reporting performance in cardiac surgery; however, rankings have shown to be imprecise, yet the extent of this imprecision is unknown. We aimed to determine the precision of, and fluctuations in, ranking lists in the comparison of cardiac surgery mortality rates.

Methods and Results—

Information on all adult cardiac surgery patients in all 16 cardiothoracic centers in The Netherlands from January 1, 2007, until December 31, 2009, was extracted from the database of the Netherlands Association for Cardio-Thoracic Surgery (n=46883). Ranks were assessed using crude and adjusted mortality rates, using a random effects logistic regression model. Risk adjustment was performed using the logistic EuroSCORE. Statistical precision of ranks was assessed with 95% confidence intervals. Additional analyses were performed for patients with isolated coronary artery bypass grafting. The ranking lists, based on mortality rates in 3 consecutive years, showed considerable reshuffling. When all data were pooled, the mean width of the 95% confidence intervals was 10 ranks using crude and 8 ranks using adjusted mortality rates. The large overlap of the confidence intervals across hospitals indicates that rank statistics were not materially different. Results were similar in the isolated coronary artery bypass grafting subgroup.

Conclusions—

Rankings are an imprecise statistical method to report cardiac surgery mortality rates and prone to (random) fluctuation. Hence, reshuffling of ranks can be expected solely due to chance. Therefore, we strongly discourage the use of ranking lists in the comparison of mortality rates.