On shrinkage and model extrapolation in the evaluation of clinical center performance

Modelling Volume-Outcome Relationships in Health Care

Despite the ongoing strong interest in associations between quality of care and the volume of health care providers, a unified statistical framework for analyzing them is missing, and many studies suffer from poor statistical modelling choices. We propose a flexible, additive mixed model for studying volume-outcome associations in health care that takes into account individual patient characteristics as well as provider-specific effects through a hierarchical approach. More specifically, we treat volume as a continuous variable, and its effect on the considered outcome is modeled as a smooth function. We take account of different case-mixes by including patient-specific risk factors and clustering on the provider level through random intercepts. This strategy enables us to extract a smooth volume effect as well as volume-independent provider effects. These two quantities can be compared directly in terms of their magnitude, which gives insight into the sources of variability of quality of care. Based on a causal DAG, we derive conditions under which the volume-effect can be interpreted as a causal effect. The paper provides confidence sets for each of the estimated quantities relying on joint estimation of all effects and parameters. Our approach is illustrated through simulation studies and an application to German health care data about mortality of very low birth weight infants.

Cross-Sectoral Comparisons of Process Quality Indicators of Health Care Across Residential Regions Using Restricted Mean Survival Time

BACKGROUND

Practice guidelines recommend patient management based on scientific evidence. Quality indicators gauge adherence to such recommendations and assess health care quality. They are usually defined as adverse event rates, which may not fully capture guideline adherence over time.

METHODS

For assessing process indicators where compliance to the recommended treatment can be assessed by evaluating a patient's trace in linked routine databases, we propose using restricted mean survival time or restricted mean time lost, which are applicable even in competing risk situations. We demonstrate their application by assessing the compliance of patients with acute myocardial infarction (AMI) to high-power statins over 12 months in Austria's political districts, using pseudo-observations and employing causal inference methods to achieve regional comparability.

RESULTS

We analyzed the compliance of 31,678 AMI patients from Austria's 116 political districts with index AMI between 2011 and 2015. The results revealed considerable compliance variations across districts but also plausible spatial similarities.

CONCLUSIONS

Restricted mean survival time and restricted mean time lost provide interpretable estimates of patients' expected time in compliance (lost), well-suited for risk-adjusted entity comparisons in the presence of (measurable) confounding, censoring, and competing risks.

Do Common Risk Adjustment Methods Do Their Job Well If Center Effects Are Correlated With the Center-Specific Mean Values of Patient Characteristics?

BACKGROUND

Direct and indirect standardization are well-established approaches to performing risk adjustment when comparing outcomes between healthcare providers. However, it is an open question whether they work well when there is an association between the center effects and the distributions of the patient characteristics in these centers.

OBJECTIVES AND METHODS

We try to shed further light on the impact of such an association. We construct an artificial case study with a single covariate, in which centers can be classified as performing above, on, or below average, and the center effects correlate with center-specific mean values of a patient characteristic, as a consequence of differential quality improvement. Based on this case study, direct standardization and indirect standardization-based on marginal as well as conditional models-are compared with respect to systematic differences between their results.

RESULTS

Systematic differences between the methods were observed. All methods produced results that partially reflect differences in mean age across the centers. This may mask the classification as above, on, or below average. The differences could be explained by an inspection of the parameter estimates in the models fitted.

CONCLUSIONS

In case of correlations of center effects with center-specific mean values of a covariate, different risk adjustment methods can produce systematically differing results. This suggests the routine use of sensitivity analyses. Center effects in a conditional model need not reflect the position of a center above or below average, questioning its use in defining the truth. Further empirical investigations are necessary to judge the practical relevance of these findings.

Prognostic score-based methods for estimating center effects based on survival probability: Application to post-kidney transplant survival

In evaluating the performance of different facilities or centers on survival outcomes, the standardized mortality ratio (SMR), which compares the observed to expected mortality has been widely used, particularly in the evaluation of kidney transplant centers. Despite its utility, the SMR may exaggerate center effects in settings where survival probability is relatively high. An example is one-year graft survival among U.S. kidney transplant recipients. We propose a novel approach to estimate center effects in terms of differences in survival probability (ie, each center versus a reference population). An essential component of the method is a prognostic score weighting technique, which permits accurately evaluating centers without necessarily specifying a correct survival model. Advantages of our approach over existing facility-profiling methods include a metric based on survival probability (greater clinical relevance than ratios of counts/rates); direct standardization (valid to compare between centers, unlike indirect standardization based methods, such as the SMR); and less reliance on correct model specification (since the assumed model is used to generate risk classes as opposed to fitted-value based 'expected' counts). We establish the asymptotic properties of the proposed weighted estimator and evaluate its finite-sample performance under a diverse set of simulation settings. The method is then applied to evaluate U.S. kidney transplant centers with respect to graft survival probability.

Limitations of the Inter-Unit Reliability: A Set of Practical Examples

Healthcare quality measures are statistics that serve to evaluate healthcare providers and identify those that need to improve their care. Before using these measures in clinical practice, developers and reviewers assess measure reliability, which describes the degree to which differences in the measure values reflect actual variation in healthcare quality, as opposed to random noise. The Inter-Unit Reliability (IUR) is a popular statistic for assessing reliability, and it describes the proportion of total variation in a measure that is attributable to between-provider variation. However, Kalbfleisch, He, Xia, and Li (2018) [Health Services and Outcomes Research Methodology, 18, 215-225] have argued that the IUR has a severe limitation in that some of the between-provider variation may be unrelated to quality of care. In this paper, we illustrate the practical implications of this limitation through several concrete examples. We show that certain best-practices in measure development, such as careful risk adjustment and exclusion of unstable measure values, can decrease the sample IUR value. These findings uncover potential negative consequences of discarding measures with IUR values below some arbitrary threshold.

Evaluations of statistical methods for outlier detection when benchmarking in clinical registries: a systematic review

OBJECTIVES

Benchmarking is common in clinical registries to support the improvement of health outcomes by identifying underperforming clinician or health service providers. Despite the rise in clinical registries and interest in publicly reporting benchmarking results, appropriate methods for benchmarking and outlier detection within clinical registries are not well established, and the current application of methods is inconsistent. The aim of this review was to determine the current statistical methods of outlier detection that have been evaluated in the context of clinical registry benchmarking.

DESIGN

A systematic search for studies evaluating the performance of methods to detect outliers when benchmarking in clinical registries was conducted in five databases: EMBASE, ProQuest, Scopus, Web of Science and Google Scholar. A modified healthcare modelling evaluation tool was used to assess quality; data extracted from each study were summarised and presented in a narrative synthesis.

RESULTS

Nineteen studies evaluating a variety of statistical methods in 20 clinical registries were included. The majority of studies conducted application studies comparing outliers without statistical performance assessment (79%), while only few studies used simulations to conduct more rigorous evaluations (21%). A common comparison was between random effects and fixed effects regression, which provided mixed results. Registry population coverage, provider case volume minimum and missing data handling were all poorly reported.

CONCLUSIONS

The optimal methods for detecting outliers when benchmarking clinical registry data remains unclear, and the use of different models may provide vastly different results. Further research is needed to address the unresolved methodological considerations and evaluate methods across a range of registry conditions.

PROSPERO REGISTRATION NUMBER

CRD42022296520.

Clinics Optimizing MEthadone Take-homes for opioid use disorder (COMET): Protocol for a stepped-wedge randomized trial to facilitate clinic level changes

INTRODUCTION

Regulatory changes made during the COVID-19 public health emergency (PHE) that relaxed criteria for take-home dosing (THD) of methadone offer an opportunity to improve quality of care with a lifesaving treatment. There is a pressing need for research to study the long-term effects of the new PHE THD rules and to test data-driven interventions to promote more effective adoption by opioid treatment programs (OTPs). We propose a two-phase project to develop and test a multidimensional intervention for OTPs that leverages information from large State administrative data.

METHODS AND ANALYSIS

We propose a two-phased project to develop then test a multidimensional OTP intervention to address clinical decision making, regulatory confusion, legal liability concerns, capacity for clinical practice change, and financial barriers to THD. The intervention will include OTP THD specific dashboards drawn from multiple State databases. The approach will be informed by the Health Equity Implementation Framework (HEIF). In phase 1, we will employ an explanatory sequential mixed methods design to combine analysis of large state administrative databases-Medicaid, treatment registry, THD reporting-with qualitative interviews to develop and refine the intervention. In phase 2, we will conduct a stepped-wedge trial over three years with 36 OTPs randomized to 6 cohorts of a six-month clinic-level intervention. The trial will test intervention effects on OTP-level implementation outcomes and patient outcomes (1) THD use; 2) retention in care; and 3) adverse healthcare events). We will specifically examine intervention effects for Black and Latinx clients. A concurrent triangulation mixed methods design will be used: quantitative and qualitative data collection will occur concurrently and results will be integrated after analysis of each. We will employ generalized linear mixed models (GLMMs) in the analysis of stepped-wedge trials. The primary outcome will be weekly or greater THD. The semi-structured interviews will be transcribed and analyzed with Dedoose to identify key facilitators, barriers, and experiences according to HEIF constructs using directed content analysis.

DISCUSSION

This multi-phase, embedded mixed methods project addresses a critical need to support long-term practice changes in methadone treatment for opioid use disorder following systemic changes emerging from the PHE-particularly for Black and Latinx individuals with opioid use disorder. By combining findings from analyses of large administrative data with lessons gleaned from qualitative interviews of OTPs that were flexible with THD and those that were not, we will build and test the intervention to coach clinics to increase flexibility with THD. The findings will inform policy at the local and national level.

MEASURING PERFORMANCE FOR END-OF-LIFE CARE

Although not without controversy, readmission is entrenched as a hospital quality metric with statistical analyses generally based on fitting a logistic-Normal generalized linear mixed model. Such analyses, however, ignore death as a competing risk, although doing so for clinical conditions with high mortality can have profound effects; a hospital's seemingly good performance for readmission may be an artifact of it having poor performance for mortality. in this paper we propose novel multivariate hospital-level performance measures for readmission and mortality that derive from framing the analysis as one of cluster-correlated semi-competing risks data. We also consider a number of profiling-related goals, including the identification of extreme performers and a bivariate classification of whether the hospital has higher-/lower-than-expected readmission and mortality rates via a Bayesian decision-theoretic approach that characterizes hospitals on the basis of minimizing the posterior expected loss for an appropriate loss function. in some settings, particularly if the number of hospitals is large, the computational burden may be prohibitive. To resolve this, we propose a series of analysis strategies that will be useful in practice. Throughout, the methods are illustrated with data from CMS on N = 17,685 patients diagnosed with pancreatic cancer between 2000-2012 at one of J = 264 hospitals in California.

Improving large-scale estimation and inference for profiling health care providers

Provider profiling has been recognized as a useful tool in monitoring health care quality, facilitating inter-provider care coordination, and improving medical cost-effectiveness. Existing methods often use generalized linear models with fixed provider effects, especially when profiling dialysis facilities. As the number of providers under evaluation escalates, the computational burden becomes formidable even for specially designed workstations. To address this challenge, we introduce a serial blockwise inversion Newton algorithm exploiting the block structure of the information matrix. A shared-memory divide-and-conquer algorithm is proposed to further boost computational efficiency. In addition to the computational challenge, the current literature lacks an appropriate inferential approach to detecting providers with outlying performance especially when small providers with extreme outcomes are present. In this context, traditional score and Wald tests relying on large-sample distributions of the test statistics lead to inaccurate approximations of the small-sample properties. In light of the inferential issue, we develop an exact test of provider effects using exact finite-sample distributions, with the Poisson-binomial distribution as a special case when the outcome is binary. Simulation analyses demonstrate improved estimation and inference over existing methods. The proposed methods are applied to profiling dialysis facilities based on emergency department encounters using a dialysis patient database from the Centers for Medicare & Medicaid Services.

Facility profiling under competing risks using multivariate prognostic scores: Application to kidneytransplant centers

The performance of health care facilities (e.g. hospitals, transplant centers, etc.) is often evaluated through time-to-event outcomes. In this paper, we consider the case where, for each subject, the failure event is due to one of several mutually exclusive causes (competing risks). Since the distribution of patient characteristics may differ greatly by the center, some form of covariate adjustment is generally necessary in order for center-specific outcomes to be accurately compared (to each other or to an overall average). We propose a weighting method for comparing facility-specific cumulative incidence functions to an overall average. The method directly standardizes each facility's non-parametric cumulative incidence function through a weight function constructed from a multivariate prognostic score. We formally define the center effects and derive large-sample properties of the proposed estimator. We evaluate the finite sample performance of the estimator through simulation. The proposed method is applied to the end-stage renal disease setting to evaluate the center-specific pre-transplant mortality and transplant cumulative incidence functions from the Scientific Registry of Transplant Recipients.

Can we trust the standardized mortality ratio? A formal analysis and evaluation based on axiomatic requirements

BACKGROUND

The standardized mortality ratio (SMR) is often used to assess and compare hospital performance. While it has been recognized that hospitals may differ in their SMRs due to differences in patient composition, there is a lack of rigorous analysis of this and other-largely unrecognized-properties of the SMR.

METHODS

This paper proposes five axiomatic requirements for adequate standardized mortality measures: strict monotonicity (monotone relation to actual mortality rates), case-mix insensitivity (independence of patient composition), scale insensitivity (independence of hospital size), equivalence principle (equal rating of hospitals with equal actual mortality rates in all patient groups), and dominance principle (better rating of unambiguously better performing hospitals). Given these axiomatic requirements, effects of variations in patient composition, hospital size, and actual and expected mortality rates on the SMR were examined using basic algebra and calculus. In this regard, we distinguished between standardization using expected mortality rates derived from a different dataset (external standardization) and standardization based on a dataset including the considered hospitals (internal standardization). The results were illustrated by hypothetical examples.

RESULTS

Under external standardization, the SMR fulfills the axiomatic requirements of strict monotonicity and scale insensitivity but violates the requirement of case-mix insensitivity, the equivalence principle, and the dominance principle. All axiomatic requirements not fulfilled under external standardization are also not fulfilled under internal standardization. In addition, the SMR under internal standardization is scale sensitive and violates the axiomatic requirement of strict monotonicity.

CONCLUSIONS

The SMR fulfills only two (none) out of the five proposed axiomatic requirements under external (internal) standardization. Generally, the SMRs of hospitals are differently affected by variations in case mix and actual and expected mortality rates unless the hospitals are identical in these characteristics. These properties hamper valid assessment and comparison of hospital performance based on the SMR.

Constructing inverse probability weights for institutional comparisons in healthcare

In comparing quality of care between hospitals, disease-specific quality indicators measure structural, process, or outcome elements related to the care of a particular condition. Such comparisons can be framed in terms of causal contrasts, answering the question of whether a patient (or a population of patients on average) would receive different care if treated at the care level of a different hospital. Fair comparisons have to be adjusted for patient case-mix, which is equivalent to controlling for confounding by the patient-level factors, including demographic factors, comorbidities, and disease progression. The methodological choice for such comparisons is usually between direct and indirect standardization methods. In this article, we discuss the alternative of inverse probability weighting as a tool for standardization in hospital comparisons. This involves fitting multinomial logistic hospital assignment models and using these to construct the inverse probability weights. The challenge in the present context is the presence of large number of hospitals being compared, many of which have a small patient volume. We propose methods to include small categories in the weighted analysis, as well as metrics and visualizations for checking the positivity/overlap and covariate balance in constructing such weights. The methods are illustrated in a running example using linked administrative data on surgical treatment of kidney cancer patients in Ontario.

Ranking hospitals when performance and risk factors are correlated: A simulation-based comparison of risk adjustment approaches for binary outcomes

Background

The conceptualization of hospital quality indicators usually includes some form of risk adjustment to account for hospital differences in case mix. For binary outcome variables like in-hospital mortality, frequently utilized risk adjusted measures include the standardized mortality ratio (SMR), the risk standardized mortality rate (RSMR), and excess risk (ER). All of these measures require the estimation of expected hospital mortality, which is often based on logistic regression models. In this context, an issue that is often neglected is correlation between hospital performance (e.g. care quality) and patient-specific risk factors. The objective of this study was to investigate the impact of such correlation on the adequacy of hospital rankings based on different measures and methods.

Methods

Using Monte Carlo simulation, the impact of correlation between hospital care quality and patient-specific risk factors on the adequacy of hospital rankings was assessed for SMR/RSMR, and ER based on logistic regression and random effects logistic regression. As an alternative method, fixed effects logistic regression with Firth correction was considered. The adequacies of the resulting hospital rankings were assessed by the shares of hospitals correctly classified into quintiles according to their true (unobserved) care qualities.

Results

The performance of risk adjustment approaches based on logistic regression and random effects logistic regression declined when correlation between care quality and a risk factor was induced. In contrast, fixed-effects-based estimations proved to be more robust. This was particularly true for fixed-effects-logistic-regression-based ER. In the absence of correlation between risk factors and care quality, all approaches showed similar performance.

Conclusions

Correlation between risk factors and hospital performance may severely bias hospital rankings based on logistic regression and random effects logistic regression. ER based on fixed effects logistic regression with Firth correction should be considered as an alternative approach to assess hospital performance.

Causal variance decompositions for institutional comparisons in healthcare

There is increasing interest in comparing institutions delivering healthcare in terms of disease-specific quality indicators (QIs) that capture processes or outcomes showing variations in the care provided. Such comparisons can be framed in terms of causal models, where adjusting for patient case-mix is analogous to controlling for confounding, and exposure is being treated in a given hospital, for instance. Our goal here is to help identify good QIs rather than comparing hospitals in terms of an already chosen QI, and so we focus on the presence and magnitude of overall variation in care between the hospitals rather than the pairwise differences between any two hospitals. We consider how the observed variation in care received at patient level can be decomposed into that causally explained by the hospital performance adjusting for the case-mix, the case-mix itself, and residual variation. For this purpose, we derive a three-way variance decomposition, with particular attention to its causal interpretation in terms of potential outcome variables. We propose model-based estimators for the decomposition, accommodating different link functions and either fixed or random effect models. We evaluate their performance in a simulation study and demonstrate their use in a real data application.

Causal Mediation Analysis for Standardized Mortality Ratios

Indirectly standardized mortality ratios (SMR) are often used to compare patient outcomes between health care providers as indicators of quality of care. Observed differences in the outcomes raise the question of whether these could be causally attributable to earlier processes or outcomes in the pathway of care that the patients received. Such pathways can be naturally addressed in a causal mediation analysis framework. Adopting causal mediation models allows the total provider effect on outcome to be decomposed into direct and indirect (mediated) effects. This in turn enables quantification of the improvement in patient outcomes due to a hypothetical intervention on the mediator. We formulate the effect decomposition for the indirectly standardized SMR when comparing to a health care system-wide average performance, propose novel model-based and semiparametric estimators for the decomposition, study the properties of these through simulations, and demonstrate their use through application to Ontario kidney cancer data.

Hospital and Intensive Care Unit Length of Stay for Injury Admissions: A Pan-Canadian Cohort Study

OBJECTIVE

To assess the variation in hospital and intensive care unit (ICU) length of stay (LOS) for injury admissions across Canadian provinces and to evaluate the relative contribution of patient case mix and treatment-related factors (intensity of care, complications, and discharge delays) to explaining observed variations.

BACKGROUND

Identifying unjustified interprovider variations in resource use and the determinants of such variations is an important step towards optimizing health care.

METHODS

We conducted a multicenter, retrospective cohort study on admissions for major trauma (injury severity score >12) to level I and II trauma centers across Canada (2006-2012). We used data from the Canadian National Trauma Registry linked to hospital discharge data to compare risk-adjusted hospital and ICU LOS across provinces.

RESULTS

Risk-adjusted hospital LOS was shortest in Ontario (10.0 days) and longest in Newfoundland and Labrador (16.1 days; P < 0.001). Risk-adjusted ICU LOS was shortest in Québec (4.4 days) and longest in Alberta (6.1 days; P < 0.001). Patient case-mix explained 32% and 8% of interhospital variations in hospital and ICU LOS, respectively, whereas treatment-related factors explained 63% and 22%.

CONCLUSIONS

We observed significant variation in risk-adjusted hospital and ICU LOS across trauma systems in Canada. Provider ranks on hospital LOS were not related to those observed for ICU LOS. Treatment-related factors explained more interhospital variation in LOS than patient case-mix. Results suggest that interventions targeting reductions in low-value procedures, prevention of adverse events, and better discharge planning may be most effective for optimizing LOS for injury admissions.

Evaluating center performance in the competing risks setting: Application to outcomes of wait-listed end-stage renal disease patients

It is often of interest to compare centers or healthcare providers on quality of care delivered. We consider the setting where evaluation of center performance on multiple competing events is of interest. We propose estimating center effects through cause-specific proportional hazards frailty models that allow correlation among a center's cause-specific effects. Estimation of our model proceeds via penalized partial likelihood and is implemented in R. To evaluate center performance, we also propose a directly standardized excess cumulative incidence (ECI) measure. Therefore, based on our proposed methods, practitioners can evaluate centers either through the cause-specific hazards or the cumulative incidence functions. We demonstrate, through simulations, the advantages of the proposed methods to detect outlying centers, by comparing the proposed methods and existing methods which assume uncorrelated random center effects. In addition, we develop a Correlation Score Test to test the null hypothesis that the competing event processes within a center are correlated. Using data from the Scientific Registry of Transplant Recipients, we apply our method to evaluate the performance of Organ Procurement Organizations on two competing risks: (i) receipt of a kidney transplant and (ii) death on the wait-list.

Acute stroke alert activation, emergency service use, and reperfusion therapy in Sweden

OBJECTIVES

Ambulance services and stroke alerts reduce the time from stroke onset to acute stroke diagnosis. We describe the use of stroke alerts and ambulance services in different hospitals and patient groups and their relationship with reperfusion therapy.

METHODS

This nationwide study included 49,907 patients admitted with acute stroke who were registered in The Swedish Stroke Register (Riksstroke) in 2011-2012.

RESULTS

The proportions of patients admitted as stroke alerts out of all acute stroke admissions varied from 12.2% to 45.7% in university hospitals (n = 9), 0.5% to 38.7% in specialized nonuniversity hospitals (n = 22), and 4.2% to 40.3% in community hospitals (n = 41). Younger age, atrial fibrillation (AF), living in an institution, reduced consciousness upon admission, and hemorrhagic stroke were factors associated with a higher probability of stroke alerts. Living alone, primary school education, non-European origin, previous stroke, diabetes, smoking, and dependency in activities of daily living (ADL) were associated with a lower probability of stroke alert. The proportion of patients arriving at the hospital by ambulance varied from 60.3% to 94.5%. Older age, living alone, primary school education, being born in a European country, previous stroke, AF, dependency in ADL, living in an institution, reduced consciousness upon admission, and hemorrhagic stroke were associated with ambulance services. Hospital stroke alert frequencies correlated strongly with reperfusion rates (r = .75).

CONCLUSION

Acute stroke alerts have a significant potential to improve stroke reperfusion rates. Prehospital stroke management varies conspicuously between hospitals and patient groups, and the elderly and patients living alone have a markedly reduced likelihood of stroke alerts.

Assessing Hospital Performance After Percutaneous Coronary Intervention Using Big Data

BACKGROUND

Although risk adjustment remains a cornerstone for comparing outcomes across hospitals, optimal strategies continue to evolve in the presence of many confounders. We compared conventional regression-based model to approaches particularly suited to leveraging big data.

METHODS AND RESULTS

We assessed hospital all-cause 30-day excess mortality risk among 8952 adults undergoing percutaneous coronary intervention between October 1, 2011, and September 30, 2012, in 24 Massachusetts hospitals using clinical registry data linked with billing data. We compared conventional logistic regression models with augmented inverse probability weighted estimators and targeted maximum likelihood estimators to generate more efficient and unbiased estimates of hospital effects. We also compared a clinically informed and a machine-learning approach to confounder selection, using elastic net penalized regression in the latter case. Hospital excess risk estimates range from -1.4% to 2.0% across methods and confounder sets. Some hospitals were consistently classified as low or as high excess mortality outliers; others changed classification depending on the method and confounder set used. Switching from the clinically selected list of 11 confounders to a full set of 225 confounders increased the estimation uncertainty by an average of 62% across methods as measured by confidence interval length. Agreement among methods ranged from fair, with a κ statistic of 0.39 (SE: 0.16), to perfect, with a κ of 1 (SE: 0.0).

CONCLUSIONS

Modern causal inference techniques should be more frequently adopted to leverage big data while minimizing bias in hospital performance assessments.

The Importance of Integrating Clinical Relevance and Statistical Significance in the Assessment of Quality of Care--Illustrated Using the Swedish Stroke Register

Background

When profiling hospital performance, quality inicators are commonly evaluated through hospital-specific adjusted means with confidence intervals. When identifying deviations from a norm, large hospitals can have statistically significant results even for clinically irrelevant deviations while important deviations in small hospitals can remain undiscovered. We have used data from the Swedish Stroke Register (Riksstroke) to illustrate the properties of a benchmarking method that integrates considerations of both clinical relevance and level of statistical significance.

Methods

The performance measure used was case-mix adjusted risk of death or dependency in activities of daily living within 3 months after stroke. A hospital was labeled as having outlying performance if its case-mix adjusted risk exceeded a benchmark value with a specified statistical confidence level. The benchmark was expressed relative to the population risk and should reflect the clinically relevant deviation that is to be detected. A simulation study based on Riksstroke patient data from 2008–2009 was performed to investigate the effect of the choice of the statistical confidence level and benchmark value on the diagnostic properties of the method.

Results

Simulations were based on 18,309 patients in 76 hospitals. The widely used setting, comparing 95% confidence intervals to the national average, resulted in low sensitivity (0.252) and high specificity (0.991). There were large variations in sensitivity and specificity for different requirements of statistical confidence. Lowering statistical confidence improved sensitivity with a relatively smaller loss of specificity. Variations due to different benchmark values were smaller, especially for sensitivity. This allows the choice of a clinically relevant benchmark to be driven by clinical factors without major concerns about sufficiently reliable evidence.

Conclusions

The study emphasizes the importance of combining clinical relevance and level of statistical confidence when profiling hospital performance. To guide the decision process a web-based tool that gives ROC-curves for different scenarios is provided.

On the practice of ignoring center-patient interactions in evaluating hospital performance

We evaluate the performance of medical centers based on a continuous or binary patient outcome (e.g., 30‐day mortality). Common practice adjusts for differences in patient mix through outcome regression models, which include patient‐specific baseline covariates (e.g., age and disease stage) besides center effects. Because a large number of centers may need to be evaluated, the typical model postulates that the effect of a center on outcome is constant over patient characteristics. This may be violated, for example, when some centers are specialized in children or geriatric patients. Including interactions between certain patient characteristics and the many fixed center effects in the model increases the risk for overfitting, however, and could imply a loss of power for detecting centers with deviating mortality. Therefore, we assess how the common practice of ignoring such interactions impacts the bias and precision of directly and indirectly standardized risks. The reassuring conclusion is that the common practice of working with the main effects of a center has minor impact on hospital evaluation, unless some centers actually perform substantially better on a specific group of patients and there is strong confounding through the corresponding patient characteristic. The bias is then driven by an interplay of the relative center size, the overlap between covariate distributions, and the magnitude of the interaction effect. Interestingly, the bias on indirectly standardized risks is smaller than on directly standardized risks. We illustrate our findings by simulation and in an analysis of 30‐day mortality on Riksstroke. © 2015 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.

Evaluating hospital performance based on excess cause-specific incidence

Formal evaluation of hospital performance in specific types of care is becoming an indispensable tool for quality assurance in the health care system. When the prime concern lies in reducing the risk of a cause-specific event, we propose to evaluate performance in terms of an average excess cumulative incidence, referring to the center's observed patient mix. Its intuitive interpretation helps give meaning to the evaluation results and facilitates the determination of important benchmarks for hospital performance. We apply it to the evaluation of cerebrovascular deaths after stroke in Swedish stroke centers, using data from Riksstroke, the Swedish stroke registry. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.