Logistic versus hierarchical modeling: an analysis of a statewide inpatient sample

Do Common Risk Adjustment Methods Do Their Job Well If Center Effects Are Correlated With the Center-Specific Mean Values of Patient Characteristics?

BACKGROUND

Direct and indirect standardization are well-established approaches to performing risk adjustment when comparing outcomes between healthcare providers. However, it is an open question whether they work well when there is an association between the center effects and the distributions of the patient characteristics in these centers.

OBJECTIVES AND METHODS

We try to shed further light on the impact of such an association. We construct an artificial case study with a single covariate, in which centers can be classified as performing above, on, or below average, and the center effects correlate with center-specific mean values of a patient characteristic, as a consequence of differential quality improvement. Based on this case study, direct standardization and indirect standardization-based on marginal as well as conditional models-are compared with respect to systematic differences between their results.

RESULTS

Systematic differences between the methods were observed. All methods produced results that partially reflect differences in mean age across the centers. This may mask the classification as above, on, or below average. The differences could be explained by an inspection of the parameter estimates in the models fitted.

CONCLUSIONS

In case of correlations of center effects with center-specific mean values of a covariate, different risk adjustment methods can produce systematically differing results. This suggests the routine use of sensitivity analyses. Center effects in a conditional model need not reflect the position of a center above or below average, questioning its use in defining the truth. Further empirical investigations are necessary to judge the practical relevance of these findings.

Cross-classified Multilevel Analysis of Individual Heterogeneity and Discriminatory Accuracy (MAIHDA) to evaluate hospital performance: the case of hospital differences in patient survival after acute myocardial infarction

OBJECTIVE

To describe a novel strategy, Multilevel Analysis of Individual Heterogeneity and Discriminatory Accuracy (MAIHDA) to evaluate hospital performance, by analysing differences in 30-day mortality after a first-ever acute myocardial infarction (AMI) in Sweden.

DESIGN

Cross-classified study.

SETTING

68 Swedish hospitals.

PARTICIPANTS

43 247 patients admitted between 2007 and 2009, with a first-ever AMI.

PRIMARY AND SECONDARY OUTCOME MEASURES

We evaluate hospital performance by analysing differences in 30-day mortality after a first-ever AMI using a cross-classified multilevel analysis. We classified the patients into 10 categories according to a risk score (RS) for 30-day mortality and created 680 strata defined by combining hospital and RS categories.

RESULTS

In the cross-classified multilevel analysis the overall RS adjusted hospital 30-day mortality in Sweden was 4.78% and the between-hospital variation was very small (variance partition coefficient (VPC)=0.70%, area under the curve (AUC)=0.54). The benchmark value was therefore achieved by all hospitals. However, as expected, there were large differences between the RS categories (VPC=34.13%, AUC=0.77) CONCLUSIONS: MAIHDA is a useful tool to evaluate hospital performance. The benefit of this novel approach to adjusting for patient RS is that it allowed one to estimate separate VPCs and AUC statistics to simultaneously evaluate the influence of RS categories and hospital differences on mortality. At the time of our analysis, all hospitals in Sweden were performing homogeneously well. That is, the benchmark target for 30-day mortality was fully achieved and there were not relevant hospital differences. Therefore, possible quality interventions should be universal and oriented to maintain the high hospital quality of care.

Multi-level models for heart failure patients' 30-day mortality and readmission rates: the relation between patient and hospital factors in administrative data

Background

This study aims at gathering evidence about the relation between 30-day mortality and 30-day unplanned readmission and patient and hospital factors. By definition, we refer to 30-day mortality and 30-day unplanned readmission as the number of deaths and non-programmed hospitalizations for any cause within 30 days after the incident heart failure (HF). In particular, the focus is on the role played by hospital-level factors.

Methods

A multi-level logistic model that combines patient- and hospital-level covariates has been developed to better disentangle the role played by the two groups of covariates. Later on, hospital outliers in term of better-than-expected/worst-than-expected performers have been identified by comparing expected cases vs. observed cases. Hospitals performance in terms of 30-day mortality and 30-day unplanned readmission rates have been visualized through the creation of funnel plots. Covariates have been selected coherently to past literature. Data comes from the hospital discharge forms for Heart Failure patients in the Lombardy Region (Northern Italy). Considering incident cases for HF in the timespan 2010–2012, 78,907 records for adult patients from 117 hospitals have been collected after quality checks.

Results

Our results show that 30-day mortality and 30-day unplanned readmissions are explained by hospital-level covariates, paving the way for the design and implementation of evidence-based improvement strategies. While the percentage of surgical DRG (OR = 1.001; CI (1.000–1.002)) and the hospital type of structure (Research hospitals vs. non-research public hospitals (OR = 0.62; CI (0.48–0.80)) and Non-research private hospitals vs. non-research hospitals OR = 0.75; CI (0.63–0.90)) are significant for mortality, the mean length of stay (OR = 0.96; CI (0.95–0.98)) is significant for unplanned readmission, showing that mortality and readmission rates might be improved through different strategies.

Conclusion

Our results confirm that hospital-level covariates do affect quality of care, and that 30-day mortality and 30-day unplanned readmission are affected by different managerial choices. This confirms that hospitals should be accountable for their “added value” to quality of care.

Observed to expected or logistic regression to identify hospitals with high or low 30-day mortality?

INTRODUCTION

A common quality indicator for monitoring and comparing hospitals is based on death within 30 days of admission. An important use is to determine whether a hospital has higher or lower mortality than other hospitals. Thus, the ability to identify such outliers correctly is essential. Two approaches for detection are: 1) calculating the ratio of observed to expected number of deaths (OE) per hospital and 2) including all hospitals in a logistic regression (LR) comparing each hospital to a form of average over all hospitals. The aim of this study was to compare OE and LR with respect to correctly identifying 30-day mortality outliers. Modifications of the methods, i.e., variance corrected approach of OE (OE-Faris), bias corrected LR (LR-Firth), and trimmed mean variants of LR and LR-Firth were also studied.

MATERIALS AND METHODS

To study the properties of OE and LR and their variants, we performed a simulation study by generating patient data from hospitals with known outlier status (low mortality, high mortality, non-outlier). Data from simulated scenarios with varying number of hospitals, hospital volume, and mortality outlier status, were analysed by the different methods and compared by level of significance (ability to falsely claim an outlier) and power (ability to reveal an outlier). Moreover, administrative data for patients with acute myocardial infarction (AMI), stroke, and hip fracture from Norwegian hospitals for 2012-2014 were analysed.

RESULTS

None of the methods achieved the nominal (test) level of significance for both low and high mortality outliers. For low mortality outliers, the levels of significance were increased four- to fivefold for OE and OE-Faris. For high mortality outliers, OE and OE-Faris, LR 25% trimmed and LR-Firth 10% and 25% trimmed maintained approximately the nominal level. The methods agreed with respect to outlier status for 94.1% of the AMI hospitals, 98.0% of the stroke, and 97.8% of the hip fracture hospitals.

CONCLUSION

We recommend, on the balance, LR-Firth 10% or 25% trimmed for detection of both low and high mortality outliers.

Control limits to identify outlying hospitals based on risk-stratification

Outcome indicators are routinely used to compare hospitals with respect to quality of care. Indicators might be based on observed proportions of adverse events (binary outcomes) or observed averages of e.g. lengths or costs of hospital stays (continuous outcomes). These observed values are compared with expected ones in an average hospital, which might be estimated from a reference sample and should be appropriately adjusted for the case mix of patients. One possibility to achieve a reliable adjustment is to stratify the patients according to their risks, where each patient belongs to one and only one stratum. Control limits calculated under the null hypothesis of an average hospital, allowing to decide whether a discrepancy between an observed and an expected value might be explained by chance or not, are then plotted around the indicator, such that hospitals falling above those control limits are detected as being statistically worse than an average hospital. Calculation of valid control limits is however not always obvious. In this article, we propose a simple and unified framework to calculate such control limits when adjustment is based on stratification, where we allow to distinguish and disentangle the variability explained by stratification and the variability due to chance, where we take into account the uncertainty about the estimation of the expected values, and where it is possible not only to detect those hospitals which are statistically worse, but also those which are statistically much worse than an average hospital. The method applies both to binary and continuous outcomes and is illustrated on Swiss hospital discharge data.

The US hospital standardised mortality ratio: Retrospective database study of Massachusetts hospitals

Objectives

To present a case-mix adjustment model that can be used to calculate Massachusetts hospital standardised mortality ratios and can be further adapted for other state-wide data-sets.

Design

We used binary logistic regression models to predict the probability of death and to calculate the hospital standardised mortality ratios. Independent variables were patient sociodemographic characteristics (such as age, gender) and healthcare details (such as admission source). Statistical performance was evaluated using c statistics, Brier score and the Hosmer–Lemeshow test.

Setting

Massachusetts hospitals providing care to patients over financial years 2005/6 to 2007/8.

Patients

1,073,122 patients admitted to Massachusetts hospitals corresponding to 36 hospital standardised mortality ratio diagnosis groups that account for 80% of in-hospital deaths nationally.

Main outcome measures

Adjusted in-hospital mortality rates and hospital standardised mortality ratios.

Results

The significant factors determining in-hospital mortality included age, admission type, primary diagnosis, the Charlson index and do-not-resuscitate status. The Massachusetts hospital standardised mortality ratios for acute (non-specialist) hospitals ranged from 60.3 (95% confidence limits 52.7–68.6) to 130.3 (116.1–145.8). The reference standard hospital standardised mortality ratio is 100 with the values below and above 100 suggesting either random or special cause variation. The model was characterised by excellent discrimination (c statistic 0.87), high accuracy (Brier statistics 0.03) and close agreement between predicted and observed mortality rates.

Conclusions

We have developed a case-mix model to give insight into mortality rates for patients served by hospitals in Massachusetts. Our analysis indicates that this technique would be applicable and relevant to Massachusetts hospital care as well as to other US hospitals.

Fixed effects modelling for provider mortality outcomes: Analysis of the Australia and New Zealand Intensive Care Society (ANZICS) Adult Patient Data-base

BACKGROUND

Risk adjusted mortality for intensive care units (ICU) is usually estimated via logistic regression. Random effects (RE) or hierarchical models have been advocated to estimate provider risk-adjusted mortality on the basis that standard estimators increase false outlier classification. The utility of fixed effects (FE) estimators (separate ICU-specific intercepts) has not been fully explored.

METHODS

Using a cohort from the Australian and New Zealand Intensive Care Society Adult Patient Database, 2009-2010, the model fit of different logistic estimators (FE, random-intercept and random-coefficient) was characterised: Bayesian Information Criterion (BIC; lower values better), receiver-operator characteristic curve area (AUC) and Hosmer-Lemeshow (H-L) statistic. ICU standardised hospital mortality ratios (SMR) and 95%CI were compared between models. ICU site performance (FE), relative to the grand observation-weighted mean (GO-WM) on odds ratio (OR), risk ratio (RR) and probability scales were assessed using model-based average marginal effects (AME).

RESULTS

The data set consisted of 145355 patients in 128 ICUs, years 2009 (47.5%) & 2010 (52.5%), with mean(SD) age 60.9(18.8) years, 56% male and ICU and hospital mortalities of 7.0% and 10.9% respectively. The FE model had a BIC = 64058, AUC = 0.90 and an H-L statistic P-value = 0.22. The best-fitting random-intercept model had a BIC = 64457, AUC = 0.90 and H-L statistic P-value = 0.32 and random-coefficient model, BIC = 64556, AUC = 0.90 and H-L statistic P-value = 0.28. Across ICUs and over years no outliers (SMR 95% CI excluding null-value = 1) were identified and no model difference in SMR spread or 95%CI span was demonstrated. Using AME (OR and RR scale), ICU site-specific estimates diverged from the GO-WM, and the effect spread decreased over calendar years. On the probability scale, a majority of ICUs demonstrated calendar year decrease, but in the for-profit sector, this trend was reversed.

CONCLUSIONS

The FE estimator had model advantage compared with conventional RE models. Using AME, between and over-year ICU site-effects were easily characterised.

Classifying hospitals as mortality outliers: logistic versus hierarchical logistic models

The use of hierarchical logistic regression for provider profiling has been recommended due to the clustering of patients within hospitals, but has some associated difficulties. We assess changes in hospital outlier status based on standard logistic versus hierarchical logistic modelling of mortality. The study population consisted of all patients admitted to acute, non-specialist hospitals in England between 2007 and 2011 with a primary diagnosis of acute myocardial infarction, acute cerebrovascular disease or fracture of neck of femur or a primary procedure of coronary artery bypass graft or repair of abdominal aortic aneurysm. We compared standardised mortality ratios (SMRs) from non-hierarchical models with SMRs from hierarchical models, without and with shrinkage estimates of the predicted probabilities (Model 1 and Model 2). The SMRs from standard logistic and hierarchical models were highly statistically significantly correlated (r > 0.91, p = 0.01). More outliers were recorded in the standard logistic regression than hierarchical modelling only when using shrinkage estimates (Model 2): 21 hospitals (out of a cumulative number of 565 pairs of hospitals under study) changed from a low outlier and 8 hospitals changed from a high outlier based on the logistic regression to a not-an-outlier based on shrinkage estimates. Both standard logistic and hierarchical modelling have identified nearly the same hospitals as mortality outliers. The choice of methodological approach should, however, also consider whether the modelling aim is judgment or improvement, as shrinkage may be more appropriate for the former than the latter.

Evaluating the performance of Australian and New Zealand intensive care units in 2009 and 2010

The Australian and New Zealand Intensive Care Society Adult Patient Database (ANZICS APD) is one of the largest databases of its kind in the world and collects individual admissions' data from intensive care units (ICUs) around Australia and New Zealand. Use of this database for monitoring and comparing the performance of ICUs, quantified by the standardised mortality ratio, poses several theoretical and computational challenges, which are addressed in this paper. In particular, the expected number of deaths must be appropriately estimated, the ICU casemix adjustment must be adequate, statistical variation must be fully accounted for, and appropriate adjustment for multiple comparisons must be made. Typically, one or more of these issues have been neglected in ICU comparison studies. Our approach to the analysis proceeds by fitting a random coefficient hierarchical logistic regression model for the inhospital death of each patient, with patients clustered within ICUs. We anticipate the majority of ICUs will be estimated as performing 'usually' after adjusting for important clinical covariates. We take as a starting point the ideas in Ohlssen et al and estimate an appropriate null model that we expect these ICUs to follow, taking a frequentist rather than a Bayesian approach. This methodology allows us to rigorously account for the aforementioned statistical issues and to determine if there are any ICUs contributing to the Australian and New Zealand Intensive Care Society database that have comparatively unusual performance. In addition to investigating the yearly performance of the ICUs, we also estimate changes in individual ICU performance between 2009 and 2010 by adjusting for regression-to-the-mean.