Improving the statistical approach to health care provider profiling

Emergency Medical Service Agency Practices and Cardiac Arrest Survival

Importance

Survival for out-of-hospital cardiac arrest (OHCA) varies widely across emergency medical service (EMS) agencies in the US. However, little is known about which EMS practices are associated with higher agency-level survival.

Objective

To identify resuscitation practices associated with favorable neurological survival for OHCA at EMS agencies.

Design, Setting, and Participants

This cohort study surveyed EMS agencies participating in the Cardiac Arrest Registry to Enhance Survival (CARES) with 10 or more OHCAs annually during January 2015 to December 2019; data analyses were performed from April to October 2023.

Exposure

Survey of resuscitation practices at EMS agencies.

Main Outcomes and Measures

Risk-standardized rates of favorable neurological survival for OHCA at each EMS agency were estimated using hierarchical logistic regression. Multivariable linear regression then examined the association of EMS practices with rates of risk-standardized favorable neurological survival.

Results

Of 577 eligible EMS agencies, 470 agencies (81.5%) completed the survey. The mean (SD) rate of risk-standardized favorable neurological survival was 8.1% (1.8%). A total of 7 EMS practices across 3 domains (training, cardiopulmonary resuscitation [CPR], and transport) were associated with higher rates of risk-standardized favorable neurological survival. EMS agencies with higher favorable neurological survival rates were more likely to use simulation to assess CPR competency (β = 0.54; P = .05), perform frequent reassessment (at least once every 6 months) of CPR competency in new staff (β = 0.51; P = .04), use full multiperson scenario simulation for ongoing skills training (β = 0.48; P = .01), perform simulation training at least every 6 months (β = 0.63; P < .001), and conduct training in the use of mechanical CPR devices at least once annually (β = 0.43; P = .04). EMS agencies with higher risk-standardized favorable neurological survival were also more likely to use CPR feedback devices (β = 0.58; P = .007) and to transport patients to a designated cardiac arrest or ST-segment elevation myocardial infarction receiving center (β = 0.57; P = .003). Adoption of more than half (≥4) of the 7 practices was more common at EMS agencies in the highest quartile of favorable neurological survival rates (70 of 118 agencies [59.3%]) vs the lowest quartile (42 of 118 agencies [35.6%]) (P < .001).

Conclusions and Relevance

In a national registry for OHCA, 7 practices associated with higher rates of favorable neurological survival were identified at EMS agencies. Given wide variability in neurological survival across EMS agencies, these findings provide initial insights into EMS practices associated with top-performing EMS agencies in OHCA survival. Future studies are needed to validate these findings and identify best practices for EMS agencies.

Administrative Model for Profiling Hospital Performance on Coronary Artery Bypass Graft Surgery: Based on the Chinese Hospital Quality Monitoring System

BACKGROUND

We aimed to develop an administrative model to profile the performance on the outcomes of coronary artery bypass grafting across hospitals in China.

METHODS AND RESULTS

This retrospective study was based on the Chinese Hospital Quality Monitoring System (HQMS) from 2016 to 2020. The coronary artery bypass grafting cases were identified by procedure code, and those of 2016 to 2017 were randomly divided into modeling and validation cohorts, while those in other years were used to ensure the model stability across years. The outcome was discharge status as "death or withdrawal," and that withdrawal referred to discharge without medical advice when patients were in the terminal stage but reluctant to die in the hospital. Candidate covariates were mainly identified by diagnoses or procedures codes. Patient-level logistic models and hospital-level hierarchical models were established. A total of 203 010 coronary artery bypass grafts in 699 hospitals were included, with 60 704 and 20 233 cases in the modeling and validation cohorts and 40 423, 42 698, and 38 952 in the years 2018, 2019, and 2020, respectively. The death or withdrawal rate was 3.4%. The areas under the curve were 0.746 and 0.729 in the patient-level models of modeling and validation cohorts, respectively, with good calibration and stability across years. Hospital-specific risk-standardized death or withdrawal rates were 2.61% (interquartile range, 1.87%-3.99%) and 2.63% (interquartile range, 1.97%-3.44%) in the modeling and validation cohorts, which were highly correlated (correlation coefficient, 0.96; P<0.001). Between-hospital variations were distinguished among hospitals of different volumes and across years.

CONCLUSIONS

The administrative model based on Hospital Quality Monitoring System could profile hospital performance on coronary artery bypass grafting in China.

Change in Hospital Risk-standardized Stroke Mortality Performance With and Without the Passive Surveillance Stroke Severity Score

BACKGROUND

Adjustment for baseline stroke severity is necessary for accurate assessment of hospital performance. We evaluated whether adjusting for the Passive Surveillance Stroke SeVerity (PaSSV) score, a measure of stroke severity derived using administrative data, changed hospital-specific estimated 30-day risk-standardized mortality rate (RSMR) after stroke.

METHODS

We used linked administrative data to identify adults who were hospitalized with ischemic stroke or intracerebral hemorrhage across 157 hospitals in Ontario, Canada between 2014 and 2019. We fitted a random effects logistic regression model using Markov Chain Monte Carlo methods to estimate hospital-specific 30-day RSMR and 95% credible intervals with adjustment for age, sex, Charlson comorbidity index, and stroke type. In a separate model, we additionally adjusted for stroke severity using PaSSV. Hospitals were defined as low-performing, average-performing, or high-performing depending on whether the RSMR and 95% credible interval were above, overlapping, or below the cohort's crude mortality rate.

RESULTS

We identified 65,082 patients [48.0% were female, the median age (25th,75th percentiles) was 76 years (65,84), and 86.4% had an ischemic stroke]. The crude 30-day all-cause mortality rate was 14.1%. The inclusion of PaSSV in the model reclassified 18.5% (n=29) of the hospitals. Of the 143 hospitals initially classified as average-performing, after adjustment for PaSSV, 20 were reclassified as high-performing and 8 were reclassified as low-performing. Of the 4 hospitals initially classified as low-performing, 1 was reclassified as high-performing. All 10 hospitals initially classified as high-performing remained unchanged.

CONCLUSION

PaSSV may be useful for risk-adjusting mortality when comparing hospital performance. External validation of our findings in other jurisdictions is needed.

Inappropriate Prescribing to Older Patients by Nurse Practitioners and Primary Care Physicians

BACKGROUND

Many U.S. states have legislated to allow nurse practitioners (NPs) to independently prescribe drugs. Critics contend that these moves will adversely affect quality of care.

OBJECTIVE

To compare rates of inappropriate prescribing among NPs and primary care physicians.

DESIGN

Rates of inappropriate prescribing were calculated and compared for 23 669 NPs and 50 060 primary care physicians who wrote prescriptions for 100 or more patients per year, with adjustment for practice experience, patient volume and risk, clinical setting, year, and state.

SETTING

29 states that had granted NPs prescriptive authority by 2019.

PATIENTS

Medicare Part D beneficiaries aged 65 years or older in 2013 to 2019.

MEASUREMENTS

Inappropriate prescriptions, defined as drugs that typically should not be prescribed for adults aged 65 years or older, according to the American Geriatrics Society's Beers Criteria.

RESULTS

Mean rates of inappropriate prescribing by NPs and primary care physicians were virtually identical (adjusted odds ratio, 0.99 [95% CI, 0.97 to 1.01]; crude rates, 1.63 vs. 1.69 per 100 prescriptions; adjusted rates, 1.66 vs. 1.68). However, NPs were overrepresented among clinicians with the highest and lowest rates of inappropriate prescribing. For both types of practitioners, discrepancies in inappropriate prescribing rates across states tended to be larger than discrepancies between these practitioners within states.

LIMITATION

The Beers Criteria addresses the appropriateness of a selected subset of drugs and may not be valid in some clinical settings.

CONCLUSION

Nurse practitioners were no more likely than physicians to prescribe inappropriately to older patients. Broad efforts to improve the performance of all clinicians who prescribe may be more effective than limiting independent prescriptive authority to physicians.

PRIMARY FUNDING SOURCE

The Robert Wood Johnson Foundation and National Science Foundation.

Cardiac Arrest Survival at Emergency Medical Service Agencies in Catchment Areas With Primarily Black and Hispanic Populations

Importance

Black and Hispanic patients are less likely to survive an out-of-hospital cardiac arrest (OHCA) than White patients. Given the central importance of emergency medical service (EMS) agencies in prehospital care, a better understanding of OHCA survival at EMS agencies that work in Black and Hispanic communities and White communities is needed to address OHCA disparities.

Objective

To examine whether EMS agencies serving catchment areas with primarily Black and Hispanic populations (Black and Hispanic catchment areas) have different rates of OHCA survival than agencies serving catchment areas with primarily White populations (White catchment areas).

Design, Setting, and Participants

A cohort study including adults with nontraumatic OHCA from January 1, 2015, to December 31, 2019, in the Cardiac Arrest Registry to Enhance Survival was conducted. Data analysis was conducted from August 17, 2022, to July 7, 2023.

Exposure

Emergency medical service agencies, categorized as working in catchment areas where the combination of Black and Hispanic residents made up more than 50% of the population or where White residents made up more than 50% of the population.

Main Outcomes and Measures

The unit of analysis was the EMS agency. The primary outcome was agency-level risk-standardized survival rates (RSSRs) to hospital admission for OHCA at each EMS agency, which were calculated using hierarchical logistic regression and compared between agencies serving Black and Hispanic and White catchment areas. Whether differences in OHCA survival were explained by EMS and first responder measures was evaluated with additional adjustment for these factors.

Results

Among 764 EMS agencies representing 258 342 OHCAs, 82 EMS agencies (10.7%) had a Black and Hispanic catchment area. Overall median age of the patients was 63.0 (IQR, 52.0-75.0) years, 36.1% were women, and 63.9% were men. Overall, the mean (SD) RSSR was 27.5% (3.6%), with lower survival at EMS agencies with Black and Hispanic catchment areas (25.8% [3.6%]) compared with agencies with White catchment areas (27.7% [3.5%]; P < .001). Among the 82 EMS agencies with Black and Hispanic catchment areas, a disproportionately higher number (32 [39.0%]) was in the lowest survival quartile, whereas a lower number (12 [14.6%]) was in the highest survival quartile. Additional adjustment for EMS response times, EMS termination of resuscitation rates, and first responder rates of initiating cardiopulmonary resuscitation or applying an automated external defibrillator before EMS arrival did not meaningfully attenuate differences in RSSRs between agencies with Black and Hispanic compared with White catchment areas (mean [SD] RSSRs after adjustment, 25.9% [3.3%] vs 27.7% [3.1%]; P < .001).

Conclusions and Relevance

Risk-standardized survival rates for OHCA were 1.9% lower at EMS agencies working in Black and Hispanic catchment areas than in White catchment areas. This difference was not explained by EMS response times, rates of EMS termination of resuscitation, or first responder rates of initiating cardiopulmonary resuscitation or applying an automated external defibrillator. These findings suggest there is a need for further assessment of these discrepancies.

Sample size calculations for indirect standardization

Indirect standardization, and its associated parameter the standardized incidence ratio, is a commonly-used tool in hospital profiling for comparing the incidence of negative outcomes between an index hospital and a larger population of reference hospitals, while adjusting for confounding covariates. In statistical inference of the standardized incidence ratio, traditional methods often assume the covariate distribution of the index hospital to be known. This assumption severely compromises one's ability to compute required sample sizes for high-powered indirect standardization, as in contexts where sample size calculation is desired, there are usually no means of knowing this distribution. This paper presents novel statistical methodology to perform sample size calculation for the standardized incidence ratio without knowing the covariate distribution of the index hospital and without collecting information from the index hospital to estimate this covariate distribution. We apply our methods to simulation studies and to real hospitals, to assess both its capabilities in a vacuum and in comparison to traditional assumptions of indirect standardization.

Updating the model for Risk-Standardizing survival for In-Hospital cardiac arrest to facilitate hospital comparisons

BACKGROUND

Risk-standardized survival rates (RSSR) for in-hospital cardiac arrest (IHCA) have been widely used for hospital benchmarking and research. The novel coronavirus 2019 (COVID-19) pandemic has led to a substantial decline in IHCA survival as COVID-19 infection is associated with markedly lower survival. Therefore, there is a need to update the model for computing RSSRs for IHCA given the COVID-19 pandemic.

METHODS

Within Get With The Guidelines®-Resuscitation, we identified 53,922 adult patients with IHCA from March, 2020 to December, 2021 (the COVID-19 era). Using hierarchical logistic regression, we derived and validated an updated model for survival to hospital discharge and compared the performance of this updated RSSR model with the previous model.

RESULTS

The survival rate was 21.0% and 20.8% for the derivation and validation cohorts, respectively. The model had good discrimination (C-statistic 0.72) and excellent calibration. The updated parsimonious model comprised 13 variables-all 9 predictors in the original model as well as 4 additional predictors, including COVID-19 infection status. When applied to data from the pre-pandemic period of 2018-2019, there was a strong correlation (r = 0.993) between RSSRs obtained from the updated and the previous models.

CONCLUSION

We have derived and validated an updated model to risk-standardize hospital rates of survival for IHCA. The updated model yielded RSSRs that were similar to the initial model for IHCAs in the pre-pandemic period and can be used for supporting ongoing efforts to benchmark hospitals and facilitate research that uses data from either before or after the emergence of COVID-19.

Marginal indirect standardization using latent clustering on multiple hospitals

A method was introduced in 2018 of performing indirect standardization for hospital profiling when only the marginal distributions of confounding variables are observed for the index hospital but the full joint covariate distribution is available for the reference hospitals (Wang et al, J Am Stat Assoc 2018; 114:662-630). The method constructs a synthetic comparison hospital using a weighted combination of reference hospitals, with weights assumed to follow a Dirichlet distribution with equal concentration parameters. In this article, we propose a novel method that improves upon the approach in a previous study (Wang et al, J Am Stat Assoc 2018; 114:662-630), by assuming the existence of latent classes among reference hospitals to allow for unequal Dirichlet concentration parameters. The latent class memberships, and thus the hospital weights, are informed by hospital-level characteristics. Our new method results in less biased point estimates and narrower uncertainty intervals for the standardized incidence ratio compared with the existing approach. We show the superiority of our novel methods in an application to a study on prevalence of high-radiation computed tomography exams, as well as in a simulation of the same medical context.

Hospital-level variation in follow-up strategies after percutaneous coronary intervention, revealed in health claims data of Korea

This study sought to determine hospital variation in the use of follow-up stress testing (FUST) and invasive coronary angiography (FUCAG) after percutaneous coronary intervention (PCI). The claims records of 150,580 Korean patients who received PCI in 128 hospitals between 2008 and 2015 were analyzed. Patient were considered to have undergone FUST and FUCAG, when these testings were performed within two years after discharge from the index hospitalization. Hierarchical generalized linear and frailty models were used to evaluate binary and time-to-event outcomes. Hospital-level risk-standardized FUCAG and FUST rates were highly variable across the hospitals (median, 0.41; interquartile range [IQR], 0.27–0.59; median, 0.22; IQR, 0.08–0.39, respectively). The performances of various models predicting the likelihood of FUCAG and FUST were compared, and the best performance was observed with the models adjusted for patient case mix and individual hospital effects as random effects (receiver operating characteristic curves, 0.72 for FUCAG; 0.82 for FUST). The intraclass correlation coefficients of the models (0.41 and 0.68, respectively) indicated that a considerable proportion of the observed variation was related to individual institutional effects. Higher hospital-level FUCAG and FUST rates were not preventive of death or myocardial infarction. Increased repeat revascularizations were observed in hospitals with higher FUCAG rates.

Examining Hospital Variation on Multiple Indicators of Stroke Quality of Care

BACKGROUND

Provider profiling involves comparing the performance of hospitals on indicators of quality of care. Typically, provider profiling examines the performance of hospitals on each quality indicator in isolation. Consequently, one cannot formally examine whether hospitals that have poor performance on one indicator also have poor performance on a second indicator.

METHODS

We used Bayesian multivariate response random effects logistic regression model to simultaneously examine variation and covariation in multiple binary indicators across hospitals. We considered 7 binary patient-level indicators of quality of care for patients presenting to hospital with a diagnosis of acute stroke. We examined between-hospital variation in these 7 indicators across 86 hospitals in Ontario, Canada.

RESULTS

The number of patients eligible for each indicator ranged from 1321 to 14 079. There were 7 pairs of indicators for which there was a strong correlation between a hospital's performance on each of the 2 indicators. Twenty-nine of the 86 hospitals had a probability higher than 0.90 of having worse performance than average on at least 4 of the 7 indicators. Seven of the 86 of hospitals had a probability higher than 0.90 of having worse performance than average on at least 5 indicators. Fourteen of the 86 of hospitals had a probability higher than 0.50 of having worse performance than average on at least 6 indicators. No hospitals had a probability higher than 0.50 of having worse performance than average on all 7 indicators.

CONCLUSIONS

These findings suggest that there are a small number of hospitals that perform poorly on at least half of the quality indicators, and that certain indicators tend to cluster together. The described methods allow for targeting quality improvement initiatives at these hospitals.

Association between coronary artery bypass graft center volume and year-to-year outcome variability: New York and California statewide analysis

OBJECTIVE

We evaluated whether volume-based, rather than time-based, annual reporting of center outcomes for coronary artery bypass grafting may improve inference of quality, assuming that large center-level year-to-year outcome variability is related to statistical noise.

METHODS

We analyzed 2012 to 2016 data on isolated coronary artery bypass grafting using statewide outcome reports from New York and California. Annual changes in center-level observed-to-expected mortality ratio represented stability of year-to-year outcomes. Cubic spline fit related the annual observed-to-expected ratio change and center volume. Volume above the inflection point of the spline curve indicated centers with low year-to-year change in outcome. We compared observed-to-expected ratio changes between centers below and above the volume threshold and observed-to-expected ratio changes between consecutive annual and biennial measurements.

RESULTS

There were 155 centers with median annual volume of 89 (interquartile range, 55-160) for isolated coronary artery bypass grafting. The inflection point of observed-to-expected ratio variability was observed at 111 cases/year. Median year-to-year observed-to-expected ratio change for centers performing less than 111 cases (62 centers) was greater at 0.83 (0.26-1.59) compared with centers performing 111 cases or more (93 centers) at 0.49 (022-0.87) (P < .001). By aggregating the outcome over 2 years, centers above the 111-case threshold increased from 93 centers (60%) to 118 centers (76%), but the median observed-to-expected change for all centers was similar between annual aggregates at 0.70 (0.26-1.22) compared with observed-to-expected change between biennial aggregates at 0.54 (0.23-1.02) (P = .095).

CONCLUSIONS

Center-level, risk-adjusted coronary artery bypass grafting mortality varies significantly from one year to the next. Reporting outcomes by specific case volume may complement annual reports.

National Variation in Congenital Heart Surgery Outcomes

BACKGROUND

Optimal strategies to improve national congenital heart surgery outcomes and reduce variability across hospitals remain unclear. Many policy and quality improvement efforts have focused primarily on higher-risk patients and mortality alone. Improving our understanding of both morbidity and mortality and current variation across the spectrum of complexity would better inform future efforts.

METHODS

Hospitals participating in the Society of Thoracic Surgeons Congenital Heart Surgery Database (2014-2017) were included. Case mix-adjusted operative mortality, major complications, and postoperative length of stay were evaluated using Bayesian models. Hospital variation was quantified by the interdecile ratio (IDR, upper versus lower 10%) and 95% credible intervals (CrIs). Stratified analyses were performed by risk group (Society of Thoracic Surgeons-European Association for Cardiothoracic Surgery [STAT] category) and simulations evaluated the potential impact of reductions in variation.

RESULTS

A total of 102 hospitals (n=84 407) were included, representing ≈85% of US congenital heart programs. STAT category 1 to 3 (lower risk) operations comprised 74% of cases. All outcomes varied significantly across hospitals: adjusted mortality by 3-fold (upper versus lower decile 5.0% versus 1.6%, IDR 3.1 [95% CrI 2.5-3.7]), mean length of stay by 1.8-fold (19.2 versus 10.5 days, IDR 1.8 [95% CrI 1.8-1.9]), and major complications by >3-fold (23.5% versus 7.0%, IDR 3.4 [95% CrI 3.0-3.8]). The degree of variation was similar or greater for low- versus high-risk cases across outcomes, eg, ≈3-fold mortality variation across hospitals for STAT 1 to 3 (IDR 3.0 [95% CrI 2.1-4.2]) and STAT 4 or 5 (IDR 3.1 [95% CrI 2.4-3.9]) cases. High-volume hospitals had less variability across outcomes and risk categories. Simulations suggested potential reductions in deaths (n=282), major complications (n=1539), and length of stay (101 183 days) over the 4-year study period if all hospitals were to perform at the current median or better, with 37% to 60% of the improvement related to the STAT 1 to 3 (lower risk) group across outcomes.

CONCLUSIONS

We demonstrate significant hospital variation in morbidity and mortality after congenital heart surgery. Contrary to traditional thinking, a substantial portion of potential improvements that could be realized on a national scale were related to variability among lower-risk cases. These findings suggest modifications to our current approaches to optimize care and outcomes in this population are needed.

Comparing a multivariate response Bayesian random effects logistic regression model with a latent variable item response theory model for provider profiling on multiple binary indicators simultaneously

Provider profiling entails comparing the performance of hospitals on indicators of quality of care. Many common indicators of healthcare quality are binary (eg, short‐term mortality, use of appropriate medications). Typically, provider profiling examines the variation in each indicator in isolation across hospitals. We developed Bayesian multivariate response random effects logistic regression models that allow one to simultaneously examine variation and covariation in multiple binary indicators across hospitals. Use of this model allows for (i) determining the probability that a hospital has poor performance on a single indicator; (ii) determining the probability that a hospital has poor performance on multiple indicators simultaneously; (iii) determining, by using the Mahalanobis distance, how far the performance of a given hospital is from that of an average hospital. We illustrate the utility of the method by applying it to 10 881 patients hospitalized with acute myocardial infarction at 102 hospitals. We considered six binary patient‐level indicators of quality of care: use of reperfusion, assessment of left ventricular ejection fraction, measurement of cardiac troponins, use of acetylsalicylic acid within 6 hours of hospital arrival, use of beta‐blockers within 12 hours of hospital arrival, and survival to 30 days after hospital admission. When considering the five measures evaluating processes of care, we found that there was a strong correlation between a hospital's performance on one indicator and its performance on a second indicator for five of the 10 possible comparisons. We compared inferences made using this approach with those obtained using a latent variable item response theory model.

Risk-Standardizing Rates of Return of Spontaneous Circulation for In-Hospital Cardiac Arrest to Facilitate Hospital Comparisons

Background

Sustained return of spontaneous circulation (ROSC) is the most proximal and direct assessment of acute resuscitation quality in hospitals. However, validated tools to benchmark hospital rates for ROSC after in‐hospital cardiac arrest currently do not exist.

Methods and Results

Within the national Get With The Guidelines‐Resuscitation registry, we identified 83 206 patients admitted from 335 hospitals from 2014 to 2017 with in‐hospital cardiac arrest. Using hierarchical logistic regression, we derived and validated a model for ROSC, defined as spontaneous and sustained ROSC for ≥20 consecutive minutes, from 24 pre‐arrest variables and calculated rates of risk‐standardized ROSC for in‐hospital cardiac arrest for each hospital. Overall, rates of ROSC were 72.0% and 72.7% for the derivation and validation cohorts, respectively. The model in the derivation cohort had moderate discrimination (C‐statistic 0.643) and excellent calibration (R² of 0.996). Seventeen variables were associated with ROSC, and a parsimonious model retained 10 variables. Before risk‐adjustment, the median hospital ROSC rate was 70.5% (interquartile range: 64.7–76.9%; range: 33.3–89.6%). After adjustment, the distribution of risk‐standardized ROSC rates was narrower: median of 71.9% (interquartile range: 68.2–76.4%; range: 42.2–84.6%). Overall, 56 (16.7%) of 335 hospitals had at least a 10% absolute change in percentile rank after risk standardization: 27 (8.0%) with a ≥10% negative percentile change and 29 (8.7%) with a ≥10% positive percentile change.

Conclusions

We have derived and validated a model to risk‐standardize hospital rates of ROSC for in‐hospital cardiac arrest. Use of this model can support efforts to compare acute resuscitation survival across hospitals to facilitate quality improvement.

Association between job-related stress and experience of presenteeism among Korean workers stratified on the presence of depression

Background

Presenteeism refers to the phenomenon of working while sick. Its development can be attributed to not only somatic symptoms but also underlying social agreements and workplace atmosphere. In this study, we analyzed presenteeism among workers from various industries, focusing on job-related stress with stratification on the presence of depression.

Methods

We conducted the study with data from questionnaires filled in by different enterprises enrolled in the Federation of Korean Trade Unions. Workers' depressive symptoms were investigated using the Patient Health Questionnaire-2, while questions on job-related stress and presenteeism were derived from the short form of the Korean Occupational Stress Scale and the official Korean version of the Work-Productivity and Activity Impairment Questionnaire-General Health, respectively. Multilevel logistic analysis was conducted to determine the statistical differences derived from the differences between companies.

Results

In total, 930 participants (753 men and 177 women) from 59 enterprises participated in the research. We conducted multilevel logistic regression to determine the association between the variables and presenteeism, with stratification by the presence of depression. Higher job demands and higher interpersonal conflict showed significantly elevated odds ratios (ORs) in univariate models and in the multivariate multilevel model. In the final model of total population, fully adjusted by general and work-related characteristics, higher job demands (OR: 3.29, 95% confidence interval [CI]: 2.08-5.21) and interpersonal conflict (OR: 1.87, 95% CI: 1.29-2.71) had significantly higher ORs-a tendency that remained in participants without depression.

Conclusions

This study reflected the factors associated with presenteeism among workers from various enterprises. The findings revealed that job-related stress was closely related to presenteeism in both the total population and in the population without depression. Thus, it emphasized interventions for managing job stress among workers to reduce presenteeism in general workers' population.

Profiling Hospital Performance on the Basis of Readmission After Transcatheter Aortic Valve Replacement in Ontario, Canada

Background

Readmission rates are a widely accepted quality indicator. Our objective was to develop models for calculating case‐mixed adjusted readmission rates after transcatheter aortic valve replacement for the purpose of profiling hospitals.

Methods and Results

In this population‐based study in Ontario, Canada, we identified all transcatheter aortic valve replacement procedures between April 1, 2012, and March 31, 2016. For each hospital, we first calculated 30‐day and 1‐year risk‐standardized (predicted versus expected) readmission rates, using 2‐level hierarchical logistic regression models, including clustering of patients within hospitals. We also calculated the risk‐adjusted (observed versus expected) readmission rates, accounting for the competing risk of death using a Fine‐Gray competing risk model. We categorized hospitals into 3 groups: those performing worse than expected, those performing better than expected, or those performing as expected, on the basis of whether the 95% CI was above, below, or included the provincial average readmission rate respectively. Our cohort consisted of 2129 transcatheter aortic valve replacement procedures performed at 10 hospitals. The observed readmission rate was 15.4% at 30 days and 44.2% at 1 year, with a range of 10.9% to 21.7% and 38.8% to 55.0%, respectively, across hospitals. Incorporating the competing risk of death translated into meaningful different results between models; as such, we concluded that the risk‐adjusted readmission rate was the preferred metric. On the basis of the 30‐day risk‐adjusted readmission rate, all hospitals performed as expected, with a 95% CI that included the provincial average. However, we found that there was significant variation in 1‐year risk‐adjusted readmission rate.

Conclusions

There is significant interhospital variation in 1‐year adjusted readmission rates among hospitals, suggesting that this should be a focus for quality improvement efforts in transcatheter aortic valve replacement.

Role of Hospital Volumes in Identifying Low-Performing and High-Performing Aortic and Mitral Valve Surgical Centers in the United States

Importance

Identifying high-performing surgical valve centers with the best surgical outcomes is challenging. Hospital surgical volume is a frequently used surrogate for outcomes. However, its ability to distinguish low-performing and high-performing hospitals remains unknown.

Objective

To examine the association of hospital procedure volume with hospital performance for aortic and mitral valve (MV) surgical procedures.

Design, Setting, and Participants

Within an all-payer nationally representative data set of inpatient hospitalizations, this study identified 682 unique hospitals performing surgical aortic valve replacement (SAVR) and MV replacement and repair with or without coronary artery bypass grafting (CABG) between 2007 and 2011. Procedural outcomes were further assessed for a 10-year period (2005-2014) to assess representativeness of study period.

Main Outcomes and Measures

In-hospital risk-standardized mortality rate (RSMR) calculated using hierarchical models and an empirical bayesian approach with volume-based shrinkage that allowed for reliability adjustment.

Results

At 682 US hospitals, 70 295 SAVR, 19 913 MV replacement, and 17 037 MV repair procedures were performed between 2007 and 2011, with a median annual volume of 43 (interquartile range [IQR], 23-76) SAVR, 13 (IQR, 6-22) MV replacement, and 9 (IQR, 4-19) MV repair procedures. Of 225 SAVR hospitals in the highest-volume tertile, 34.7% and 36.0% were in the highest-RSMR tertile for SAVR + CABG and isolated SAVR procedures, respectively, while 21.5% and 17.5% of the 228 SAVR hospitals in the lowest-volume tertile were in the lowest respective RSMR tertile. Similarly, 36.8% and 43.5% of hospitals in the highest tertile of volume for MV replacement and repair, respectively, were in the corresponding highest-RSMR tertile, and 17.4% and 11.2% of the low-volume hospitals were in the lowest-RSMR tertile for MV replacement and repair, respectively. There was limited correlation between outcomes for SAVR and MV procedures at an institution. If solely volume-based tertiles were used to categorize hospitals for quality, 44.7% of all valve hospitals would be misclassified (as either low performing or high performing) when assessing performance based on tertiles of RSMR.

Conclusions and Relevance

Hospital procedure volume alone frequently misclassifies hospital performance with regard to risk-standardized outcomes after aortic and MV surgical procedures. Valve surgery quality improvement endeavors should focus on a more comprehensive assessment that includes risk-adjusted outcomes rather than hospital volume alone.

Development of a Congenital Heart Surgery Composite Quality Metric: Part 2-Analytic Methods

BACKGROUND

We describe the statistical methods and results related to development of the first congenital heart surgery composite quality measure.

METHODS

The composite measure was developed using The Society of Thoracic Surgeons Congenital Heart Surgery Database (2012 to 2015), Bayesian hierarchical modeling, and the current Society of Thoracic Surgeons risk model for case-mix adjustment. It consists of a mortality domain (operative mortality) and morbidity domain (major complications and postoperative length of stay). We evaluated several potential weighting schemes and properties of the final composite measure, including reliability (signal-to-noise ratio) and hospital classification in various performance categories.

RESULTS

Overall, 100 hospitals (78,425 operations) were included. Each adjusted metric included in the composite varied across hospitals: operative mortality (median, 3.1%; 10th to 90th percentile, 2.1% to 4.4%) major complications (median 11.7%, 10th to 90th percentile, 6.4% to 17.4%), and length of stay (median, 7.0 days; 10th to 90th percentile, 5.9 to 8.2 days). In the final composite weighting scheme selected, mortality had the greatest influence, followed by major complications and length of stay (correlation with overall composite score of 0.87, 0.69, and 0.47, respectively). Reliability of the composite measure was 0.73 compared with 0.59 for mortality alone. The distribution of hospitals across composite measure performance categories (defined by whether the 95% credible interval overlapped The Society of Thoracic Surgeons average) was 75% (same as expected), 9% (worse than expected), and 16% (better than expected).

CONCLUSIONS

This congenital heart surgery composite measure incorporates aspects of both morbidity and mortality, has clinical face validity, and greater ability to discriminate hospital performance compared with mortality alone. Ongoing efforts will support the use of the composite measure in benchmarking and quality improvement activities.

Profiling Hospital Performance Based on Mortality After Transcatheter Aortic Valve Replacement in Ontario, Canada

BACKGROUND

Public reporting of hospital-level outcomes is increasingly common as a means to target quality improvement strategies to ensure the delivery of optimal care. Despite the rapid dissemination of transcatheter aortic valve replacement (TAVR), there is a paucity of reliable case-mix adjustment models for hospital profiling in TAVR. Our objective was to develop and evaluate different models for calculating risk-standardized all-cause mortality rates (RSMRs) post-TAVR.

METHODS AND RESULTS

In this population-based study in Ontario, Canada, we identified all patients who underwent a TAVR procedure between April 1, 2012, and March 31, 2016. For each hospital, we calculated 30-day and 1-year RSMR, using 2-level hierarchical logistic regression models that accounted for patient-specific demographic and clinical characteristics, as well as the clustering of patients within the same hospital using a hospital-specific random effects. We classified each hospital into one of 3 groups: performing worse than expected, better than expected, or performing as expected, based on whether the 95% CI of the RSMR was above, below, or included the provincial average mortality rate, respectively. Our cohort consisted of 2129 TAVR procedures performed at 10 hospitals. The observed mortality was 7.0% at 30 days and 16.4% at 1 year, with a range of 4% to 10% and 8% to 22%, respectively, across hospitals. We developed case-mix adjustment models using 28 clinically relevant variables. Using 30-day and 1-year RSMR to profile each hospital, we found that all hospitals performed as expected, with 95% CI that included the provincial average.

CONCLUSIONS

We found no significant interhospital variation in RSMR among hospitals, suggesting that quality improvement efforts should be directed at aspects other than the variation in observed mortality.

Joint Indirect Standardization when Only Marginal Distributions are Observed in the Index Population

It is a common interest in medicine to determine whether a hospital meets a benchmark created from an aggregate reference population, after accounting for differences in distributions of multiple covariates. Due to the difficulties of collecting individual-level data, however, it is often the case that only marginal distributions of the covariates are available, making covariate-adjusted comparison challenging. We propose and evaluate a novel approach for conducting indirect standardization when only marginal covariate distributions of the studied hospital are known, but complete information is available for the reference hospitals. We do this with the aid of two existing methods: iterative proportional fit, which estimates the cells of a contingency table when only marginal sums are known, and synthetic control methods, which create a counterfactual control group using a weighted combination of potential control groups. The proper application of these existing methods for indirect standardization would require accounting for the statistical uncertainties induced by a situation where no individual-level data is collected from the studied population. We address this need with a novel method which uses a random Dirichlet parametrization of the synthetic control weights to estimate uncertainty intervals for the standard incidence ratio. We demonstrate our novel methods by estimating hospital-level standardized incidence ratios for comparing the adjusted probability of computed tomography examinations with high radiations doses, relative to a reference standard and we evalauate out methods in a simulation study.

Detecting and Visualizing Outliers in Provider Profiling Using Funnel Plots and Mixed Effects Models-An Example from Prescription Claims Data

When prescribing a drug for a patient, a physician also has to consider economic aspects. We were interested in the feasibility and validity of profiling based on funnel plots and mixed effect models for the surveillance of German ambulatory care physicians’ prescribing. We analyzed prescriptions issued to patients with a health insurance card attending neurologists’ and psychiatrists’ ambulatory practices in the German federal state of Saarland. The German National Association of Statutory Health Insurance Physicians developed a prescribing assessment scheme (PAS) which contains a systematic appraisal of the benefit of drugs for so far 12 different indications. The drugs have been classified on the basis of their clinical evidence as “standard”, “reserve” or “third level” medication. We had 152.583 prescriptions in 56 practices available for analysis. A total of 38.796 patients received these prescriptions. The funnel plot approach with additive correction for overdispersion was almost equivalent to a mixed effects model which directly took the multilevel structure of the data into account. In the first case three practices were labeled as outliers, the mixed effects model resulted in two outliers. We suggest that both techniques should be routinely applied within a surveillance system of prescription claims data.

Variation in Empiric Coverage Versus Detection of Methicillin-Resistant Staphylococcus aureus and Pseudomonas aeruginosa in Hospitalizations for Community-Onset Pneumonia Across 128 US Veterans Affairs Medical Centers

OBJECTIVE To examine variation in antibiotic coverage and detection of resistant pathogens in community-onset pneumonia. DESIGN Cross-sectional study. SETTING A total of 128 hospitals in the Veterans Affairs health system. PARTICIPANTS Hospitalizations with a principal diagnosis of pneumonia from 2009 through 2010. METHODS We examined proportions of hospitalizations with empiric antibiotic coverage for methicillin-resistant Staphylococcus aureus (MRSA) and Pseudomonas aeruginosa (PAER) and with initial detection in blood or respiratory cultures. We compared lowest- versus highest-decile hospitals, and we estimated adjusted probabilities (AP) for patient- and hospital-level factors predicting coverage and detection using hierarchical regression modeling. RESULTS Among 38,473 hospitalizations, empiric coverage varied widely across hospitals (MRSA lowest vs highest, 8.2% vs 42.0%; PAER lowest vs highest, 13.9% vs 44.4%). Detection rates also varied (MRSA lowest vs highest, 0.5% vs 3.6%; PAER lowest vs highest, 0.6% vs 3.7%). Whereas coverage was greatest among patients with recent hospitalizations (AP for anti-MRSA, 54%; AP for anti-PAER, 59%) and long-term care (AP for anti-MRSA, 60%; AP for anti-PAER, 66%), detection was greatest in patients with a previous history of a positive culture (AP for MRSA, 7.9%; AP for PAER, 11.9%) and in hospitals with a high prevalence of the organism in pneumonia (AP for MRSA, 3.9%; AP for PAER, 3.2%). Low hospital complexity and rural setting were strong negative predictors of coverage but not of detection. CONCLUSIONS Hospitals demonstrated widespread variation in both coverage and detection of MRSA and PAER, but probability of coverage correlated poorly with probability of detection. Factors associated with empiric coverage (eg, healthcare exposure) were different from those associated with detection (eg, microbiology history). Providing microbiology data during empiric antibiotic decision making could better align coverage to risk for resistant pathogens and could promote more judicious use of broad-spectrum antibiotics. Infect Control Hosp Epidemiol 2017;38:937-944.

On-demand Reporting of Risk-adjusted and Smoothed Rates for Quality Profiling in ACS NSQIP

BACKGROUND

Surgical quality improvement depends on hospitals having accurate and timely information about comparative performance. Profiling accuracy is improved by risk adjustment and shrinkage adjustment to stabilize estimates. These adjustments are included in ACS NSQIP reports, where hospital odds ratios (OR) are estimated using hierarchical models built on contemporaneous data. However, the timeliness of feedback remains an issue.

STUDY DESIGN

We describe an alternative, nonhierarchical approach, which yields risk- and shrinkage-adjusted rates. In contrast to our "Traditional" NSQIP method, this approach uses preexisting equations, built on historical data, which permits hospitals to have near immediate access to profiling results. We compared our traditional method to this new "on-demand" approach with respect to outlier determinations, kappa statistics, and correlations between logged OR and standardized rates, for 12 models (4 surgical groups by 3 outcomes).

RESULTS

When both methods used the same contemporaneous data, there were similar numbers of hospital outliers and correlations between logged OR and standardized rates were high. However, larger differences were observed when the effect of contemporaneous versus historical data was added to differences in statistical methodology.

CONCLUSIONS

The on-demand, nonhierarchical approach provides results similar to the traditional hierarchical method and offers immediacy, an "over-time" perspective, application to a broader range of models and data subsets, and reporting of more easily understood rates. Although the nonhierarchical method results are now available "on-demand" in a web-based application, the hierarchical approach has advantages, which support its continued periodic publication as the gold standard for hospital profiling in the program.

Ultrasound lineal measurements predict ventricular volume in posthaemorrhagic ventricular dilatation in preterm infants

AIM

Posthaemorrhagic ventricular dilatation (PHVD) is monitored by conventional two-dimensional ultrasound (2DUS). The aims of this study were to determine the volume of the lateral ventricles using three-dimensional ultrasound (3DUS) in preterm infants with PHVD and to evaluate the relationship between volume and linear measurements.

METHODS

Serial 2DUSs and 3DUSs were performed on preterm infants with PHVD admitted to the neonatal intensive care unit at Puerta del Mar Hospital, Cádiz, Spain, from January 2013 to December 2014. The ventricular index, anterior horn width and thalamo-occipital distance were used as ventricular lineal measurements. Ventricular volume was calculated offline.

RESULTS

Serial ultrasounds from seven preterm infants were measured. Each linear measurement was significantly associated with volume, and an equation was obtained through a significant multilevel mixed-effects lineal regression model: ventricular volume (cm³ ) = -11.02 + 0.668*VI + 0.817*AHW + 0.256*TOD. Intra-observer and interobserver agreement was excellent with an intraclass correlation coefficient of 0.99.

CONCLUSION

Lateral ventricular volumes of preterm infants with PHVD could be reliably determined using 3DUS. Ventricular volume could be accurately estimated using three lineal measurements. More studies are needed to address the importance of volume determination in PHVD.

Cost Variation in Diabetes Care across Dutch Care Groups?

OBJECTIVE

The introduction of bundled payment for diabetes care in the Netherlands led to the origination of care groups. This study explored to what extent variation in health care costs per patient can be attributed to the performance of care groups. Furthermore, the commonly applied simple mean aggregation was compared with the more advanced generalized linear mixed model (GLMM) to benchmark health care costs per patient between care groups.

DATA SOURCE

Dutch 2009 nationwide insurance claims data of diabetes type 2 patients (104,544 patients, 50 care groups).

STUDY DESIGN

Both a simple mean aggregation and a GLMM approach was applied to rank care groups, using two different health care costs variables: total treatment health care costs and diabetes-specific specialist care costs per diabetes patient.

PRINCIPAL FINDINGS

Care groups varied slightly in the first and mainly in the second indicator. Care group variation was not explained by composition. Although the ranking methods were correlated, some care groups' rank positions differed, with consequences on the top-10 and the low-10 positions.

CONCLUSIONS

Differences between care groups exist when an appropriate indicator and a sophisticated aggregation technique is used. Currently applied benchmarking may have unfair consequences for some care groups.

Geographic Variation Immediate and Delayed Breast Reconstruction Utilization in Ontario, Canada and Plastic Surgeon Availability: A Population-Based Observational Study

BACKGROUND

Utilization of breast reconstruction (BR) is low in many jurisdictions. We studied the geographical and surgical workforce factors that contribute to access and use of BR using a small area analysis approach with a geographical unit of analysis.

METHODS

We linked administrative data from Ontario Canada to calculate the age-standardized rates for immediate BR (IBR) (same time as mastectomy) between 2002 and 2011, and delayed BR (DBR) (within 3 years of mastectomy) for each county. The influence of plastic surgeon access on variation in county rates of BR was examined using Poisson random effects models.

RESULTS

12,663 women underwent mastectomy in Ontario; 2,948 had BR within 3 years (23.3%). Over 50% of the counties had no access to any plastic surgeon. County IBR rates ranged from 0 to 21.5%; plastic surgeon access explained 46% of geographic variation (p<0.0001). IBR rates in counties with very low, low, and moderate access to plastic surgeons were significantly less than counties with high access (relative rate [RR] 0.48 [95% confidence interval (CI) 0.35-0.66], RR 0.61 [CI 0.43-0.87] and RR 0.70 [CI 0.52-0.96], respectively) after adjusting for age and county socioeconomic characteristics. For DBR, while there was less geographic variation, very low access counties demonstrated reduced rates (RR 0.60 [CI 0.47-0.76]).

INTERPRETATION

Geographic access to a plastic surgeon is a major determinant of BR. Targeted interventions for regions without high access to plastic surgeons may improve overall rates and reduce geographic disparities in care, particularly for IBR.

Composite Measures of Health Care Provider Performance: A Description of Approaches

CONTEXT

Since the Institute of Medicine's 2001 report Crossing the Quality Chasm, there has been a rapid proliferation of quality measures used in quality-monitoring, provider-profiling, and pay-for-performance (P4P) programs. Al-though individual performance measures are useful for identifying specific processes and outcomes for improvement and tracking progress, they do not easily provide an accessible overview of performance. Composite measures aggregate individual performance measures into a summary score. By reducing the amount of data that must be processed, they facilitate (1) benchmarking of an organization's performance, encouraging quality improvement initiatives to match performance against high-performing organizations, and (2) profiling and P4P programs based on an organization's overall performance.

METHODS

We describe different approaches to creating composite measures,discuss their advantages and disadvantages, and provide examples of their use.

FINDINGS

The major issues in creating composite measures are (1) whether to aggregate measures at the patient level through all-or-none approaches or the facility level, using one of the several possible weighting schemes; (2) when combining measures on different scales, how to rescale measures (using z scores,range percentages, ranks, or 5-star categorizations); and (3) whether to use shrinkage estimators, which increase precision by smoothing rates from smaller facilities but also decrease transparency.

CONCLUSIONS

Because provider rankings and rewards under P4P programs may be sensitive to both context and the data, careful analysis is warranted before deciding to implement a particular method. A better understanding of both when and where to use composite measures and the incentives created by composite measures are likely to be important areas of research as the use of composite measures grows.

Hospital variation in survival after pediatric in-hospital cardiac arrest

BACKGROUND

Although survival after in-hospital cardiac arrest is likely to vary among hospitals caring for children,validated methods to risk-standardize pediatric survival rates across sites do not currently exist.

METHODS AND RESULTS

From 2006 to 2010, within the American Heart Association's Get With the Guidelines-Resuscitation registry for in-hospital cardiac arrest, we identified 1551 cardiac arrests in children (<18 years). Using multivariable hierarchical logistic regression, we developed and validated a model to predict survival to hospital discharge and calculated risk-standardized rates of cardiac arrest survival for hospitals with a minimum of 10 pediatric cardiac arrest cases. A total of 13 patient-level predictors were identified: age, sex, cardiac arrest rhythm, location of arrest, mechanical ventilation, acute nonstroke neurological event, major trauma, hypotension, metabolic or electrolyte abnormalities, renal insufficiency, sepsis, illness category, and need for intravenous vasoactive agents prior to the arrest. The model had good discrimination (C-statistic of 0.71), confirmed by bootstrap validation (validation C-statistic of 0.69). Among 30 hospitals with ≥10 cardiac arrests, unadjusted hospital survival rates varied considerably (median, 37%; interquartile range, 24-42%; range, 0-61%). After risk-standardization, the range of hospital survival rates narrowed (median, 37%; interquartile range, 33-38%; range, 29-48%), but variation in survival persisted.

CONCLUSIONS

Using a national registry, we developed and validated a model to predict survival after in-hospital cardiac arrest in children. After risk-standardization, significant variation in survival rates across hospitals remained. Leveraging these models, future studies can identify best practices at high-performing hospitals to improve survival outcomes for pediatric cardiac arrest. (

A probability metric for identifying high-performing facilities: an application for pay-for-performance programs

BACKGROUND

Two approaches are commonly used for identifying high-performing facilities on a performance measure: one, that the facility is in a top quantile (eg, quintile or quartile); and two, that a confidence interval is below (or above) the average of the measure for all facilities. This type of yes/no designation often does not do well in distinguishing high-performing from average-performing facilities.

OBJECTIVE

To illustrate an alternative continuous-valued metric for profiling facilities--the probability a facility is in a top quantile--and show the implications of using this metric for profiling and pay-for-performance.

METHODS

We created a composite measure of quality from fiscal year 2007 data based on 28 quality indicators from 112 Veterans Health Administration nursing homes. A Bayesian hierarchical multivariate normal-binomial model was used to estimate shrunken rates of the 28 quality indicators, which were combined into a composite measure using opportunity-based weights. Rates were estimated using Markov Chain Monte Carlo methods as implemented in WinBUGS. The probability metric was calculated from the simulation replications.

RESULTS

Our probability metric allowed better discrimination of high performers than the point or interval estimate of the composite score. In a pay-for-performance program, a smaller top quantile (eg, a quintile) resulted in more resources being allocated to the highest performers, whereas a larger top quantile (eg, being above the median) distinguished less among high performers and allocated more resources to average performers.

CONCLUSION

The probability metric has potential but needs to be evaluated by stakeholders in different types of delivery systems.

Classification accuracy of claims-based methods for identifying providers failing to meet performance targets

Quality assessment is critical for healthcare reform, but data sources are lacking for measurement of many important healthcare outcomes. With over 49 million people covered by Medicare as of 2010, Medicare claims data offer a potentially valuable source that could be used in targeted health care quality improvement efforts. However, little is known about the operating characteristics of provider profiling methods using claims-based outcome measures that may estimate provider performance with error. Motivated by the example of screening mammography performance, we compared approaches to identifying providers failing to meet guideline targets using Medicare claims data. We used data from the Breast Cancer Surveillance Consortium and linked Medicare claims to compare claims-based and clinical estimates of cancer detection rate. We then demonstrated the performance of claim-based estimates across a broad range of operating characteristics using simulation studies. We found that identification of poor performing providers was extremely sensitive to algorithm specificity, with no approach identifying more than 65% of poor performing providers when claims-based measures had specificity of 0.995 or less. We conclude that claims have the potential to contribute important information on healthcare outcomes to quality improvement efforts. However, to achieve this potential, development of highly accurate claims-based outcome measures should remain a priority.

Level-adjusted funnel plots based on predicted marginal expectations: an application to prophylactic antibiotics in gallstone surgery

Funnel plots are widely used to visualize grouped data, for example, in institutional comparison. This paper extends the concept to a multi-level setting, displaying one level at a time, adjusted for the other levels, as well as for covariates at all levels. These level-adjusted funnel plots are based on a Markov chain Monte Carlo fit of a random effects model, translating the estimated model parameters to predicted marginal expectations. Working within the estimation framework, we accommodate outlying institutions using heavy-tailed random effects distributions. We also develop computer-efficient methods to compute predicted probabilities in the case of dichotomous outcome data and various random effect distributions. We apply the method to a data set on prophylactic antibiotics in gallstone surgery.

Fixed effects modelling for provider mortality outcomes: Analysis of the Australia and New Zealand Intensive Care Society (ANZICS) Adult Patient Data-base

BACKGROUND

Risk adjusted mortality for intensive care units (ICU) is usually estimated via logistic regression. Random effects (RE) or hierarchical models have been advocated to estimate provider risk-adjusted mortality on the basis that standard estimators increase false outlier classification. The utility of fixed effects (FE) estimators (separate ICU-specific intercepts) has not been fully explored.

METHODS

Using a cohort from the Australian and New Zealand Intensive Care Society Adult Patient Database, 2009-2010, the model fit of different logistic estimators (FE, random-intercept and random-coefficient) was characterised: Bayesian Information Criterion (BIC; lower values better), receiver-operator characteristic curve area (AUC) and Hosmer-Lemeshow (H-L) statistic. ICU standardised hospital mortality ratios (SMR) and 95%CI were compared between models. ICU site performance (FE), relative to the grand observation-weighted mean (GO-WM) on odds ratio (OR), risk ratio (RR) and probability scales were assessed using model-based average marginal effects (AME).

RESULTS

The data set consisted of 145355 patients in 128 ICUs, years 2009 (47.5%) & 2010 (52.5%), with mean(SD) age 60.9(18.8) years, 56% male and ICU and hospital mortalities of 7.0% and 10.9% respectively. The FE model had a BIC = 64058, AUC = 0.90 and an H-L statistic P-value = 0.22. The best-fitting random-intercept model had a BIC = 64457, AUC = 0.90 and H-L statistic P-value = 0.32 and random-coefficient model, BIC = 64556, AUC = 0.90 and H-L statistic P-value = 0.28. Across ICUs and over years no outliers (SMR 95% CI excluding null-value = 1) were identified and no model difference in SMR spread or 95%CI span was demonstrated. Using AME (OR and RR scale), ICU site-specific estimates diverged from the GO-WM, and the effect spread decreased over calendar years. On the probability scale, a majority of ICUs demonstrated calendar year decrease, but in the for-profit sector, this trend was reversed.

CONCLUSIONS

The FE estimator had model advantage compared with conventional RE models. Using AME, between and over-year ICU site-effects were easily characterised.

A scientific registry of transplant recipients bayesian method for identifying underperforming transplant programs

In response to recommendations from a recent consensus conference and from the Committee of Presidents of Statistical Societies, the Scientific Registry of Transplant Recipients explored the use of Bayesian hierarchical, mixed-effects models in assessing transplant program performance in the United States. Identification of underperforming centers based on 1-year patient and graft survival using a Bayesian approach was compared with current observed-to-expected methods. Fewer small-volume programs (<10 transplants per 2.5-year period) were identified as underperforming with the Bayesian method than with the current method, and more mid-volume programs (10-249 transplants per 2.5-year period) were identified. Simulation studies identified optimal Bayesian-based flagging thresholds that maximize true positives while holding false positive flagging rates to approximately 5% regardless of program volume. Compared against previous program surveillance actions from the Organ Procurement and Transplantation Network Membership and Professional Standards Committee, the Bayesian method would have reduced the number of false positive program identifications by 50% for kidney, 35% for liver, 43% for heart and 57% for lung programs, while preserving true positives for, respectively, 96%, 71%, 58% and 83% of programs identified by the current method. We conclude that Bayesian methods to identify underperformance improve identification of programs that need review while minimizing false flags.

Prevalent but moderate variation across small geographic regions in patient nonadherence to evidence-based preventive therapies in older adults after acute myocardial infarction

BACKGROUND

Patient long-term adherence to β-blockers, HMG-CoA reductase inhibitors (statins), and angiotensin-converting enzyme inhibitors (ACEIs)/angiotensin receptor blockers (ARBs) after acute myocardial infarction (AMI) is alarmingly low. It is unclear how prevalent patient adherence may be across small geographic areas and whether this geographic prevalence may vary.

METHODS

This is a retrospective cohort study using Medicare service claims files from 2007 to 2009 with Medicare beneficiaries 65 years and above who were alive 30 days after the index AMI hospitalization between January 1, 2008 and December 31, 2008 (N=85,017). The adjusted proportions of patients adherent to β-blockers, statins, and ACEIs/ARBs, respectively, in the 12 months after discharge across the 306 Hospital Referral Regions (HRRs) were measured and compared by control chart. The intracluster correlation coefficient (ICC) and the additional prediction power from this small-area variation on individual patient adherence were assessed.

RESULTS

The adjusted proportion of patients adherent across HRRs ranged from 58% to 74% (median, 66%) for β-blockers, from 57% to 67% (median, 63%) for ACEIs/ARBs, and from 58% to 73% (median, 66%) for statins. The ICC was 0.053 (95% CI, 0.043-0.064) for β-blockers, 0.050 (95% CI, 0.039-0.061) for ACEIs/ARBs, and 0.041 (95% CI, 0.031-0.052) for statins. The adjusted proportion of patients adherent across HRRs increased the c-statistic by 0.01-0.02 (P < 0.0001).

CONCLUSIONS

Nonadherence to evidence-based preventive therapies post-AMI among older adults was prevalent across small geographic regions. Moderate small-area variation in patient adherence exists.

On the accuracy of classifying hospitals on their performance measures

The evaluation, comparison, and public report of health care provider performance is essential to improving the quality of health care. Hospitals, as one type of provider, are often classified into quality tiers (e.g., top or suboptimal) based on their performance data for various purposes. However, potential misclassification might lead to detrimental effects for both consumers and payers. Although such risk has been highlighted by applied health services researchers, a systematic investigation of statistical approaches has been lacking. We assess and compare the expected accuracy of several commonly used classification methods: unadjusted hospital-level averages, shrinkage estimators under a random-effects model accommodating between-hospital variation, and two others based on posterior probabilities. Assuming that performance data follow a classic one-way random-effects model with unequal sample size per hospital, we derive accuracy formulae for these classification approaches and gain insight into how the misclassification might be affected by various factors such as reliability of the data, hospital-level sample size distribution, and cutoff values between quality tiers. The case of binary performance data is also explored using Monte Carlo simulation strategies. We apply the methods to real data and discuss the practical implications.

Statistical benchmarks for health care provider performance assessment: a comparison of standard approaches to a hierarchical Bayesian histogram-based method

OBJECTIVE

Examine how widely used statistical benchmarks of health care provider performance compare with histogram-based statistical benchmarks obtained via hierarchical Bayesian modeling.

DATA SOURCES

Publicly available data from 3,240 hospitals during April 2009-March 2010 on two process-of-care measures reported on the Medicare Hospital Compare website.

STUDY DESIGN

Secondary data analyses of two process-of-care measures comparing statistical benchmark estimates and threshold exceedance determinations under various combinations of hospital performance measure estimates and benchmarking approaches.

PRINCIPAL FINDINGS

Statistical benchmarking approaches for determining top 10 percent performance varied with respect to which hospitals exceeded the performance benchmark; such differences were not found at the 50 percent threshold. Benchmarks derived from the histogram of provider performance under hierarchical Bayesian modeling provide a compromise between benchmarks based on direct (raw) estimates, which are overdispersed relative to the true distribution of provider performance and prone to high variance for small providers, and posterior mean provider performance, for which over-shrinkage and under-dispersion relative to the true provider performance distribution is a concern.

CONCLUSIONS

Given the rewards and penalties associated with characterizing top performance, the ability of statistical benchmarks to summarize key features of the provider performance distribution should be examined.

Evaluation of support group interventions for children in troubled families: study protocol for a quasi-experimental control group study

Background

Support groups for children in troubled families are available in a majority of Swedish municipalities. They are used as a preventive effort for children in families with different parental problems such as addiction to alcohol/other drugs, mental illness, domestic violence, divorce situations, or even imprisonment. Children from families with these problems are a well-known at-risk group for various mental health and social problems. Support groups aim at strengthening children’s coping behaviour, to improve their mental health and to prevent a negative psycho-social development. To date, evaluations using a control-group study design are scarce. The aim of the current study is to evaluate the effects of support groups. This paper describes the design of an effectiveness study, initially intended as a randomized controlled trial, but instead is pursued as a quasi-experimental study using a non-randomized control group.

Methods/design

The aim is to include 116 children, aged 7–13 years and one parent/another closely related adult, in the study. Participants are recruited via existing support groups in the Stockholm county district and are allocated either into an intervention group or a waiting list control group, representing care as usual. The assessment consists of questionnaires that are to be filled in at baseline and at four months following the baseline. Additionally, the intervention group completes a 12-month follow-up. The outcomes include the Strength and Difficulties Questionnaire (SDQ S11-16), the Kids Coping Scale, the “Ladder of life” which measures overall life satisfaction, and “Jag tycker jag är” (I think I am) which measures self-perception and self-esteem. The parents complete the SDQ P4-16 (parent-report version) and the Swedish scale “Familjeklimat” (Family Climate), which measures the emotional climate in the family.

Discussion

There is a need for evaluating the effects of support groups targeted to children from troubled families. This quasi-experimental study therefore makes an important contribution to this novel field of research. In the article various problems related to pursuing a study with children at risk are discussed.

Trial registration

ISRCTN52310507

Association between a hospital's quality performance for in-hospital cardiac arrest and common medical conditions

BACKGROUND

Public reporting on hospital quality has been widely adopted for common medical conditions. Adding a measure of inpatient survival after cardiac arrest is being considered. It is unknown whether this measure would be redundant, given evidence that hospital organization and culture can have hospital-wide effects on quality. Therefore, we sought to evaluate the correlation between inpatient survival after cardiac arrest and 30-day risk-standardized mortality rates for common medical conditions.

METHODS AND RESULTS

Using data between 2007 and 2010 from a national in-hospital cardiac arrest registry, we calculated risk-standardized in-hospital survival rates for cardiac arrest at each hospital. We obtained risk-standardized 30-day mortality rates for acute myocardial infarction, heart failure, and pneumonia from Hospital Compare for the same period. The relationship between a hospital's performance on cardiac arrest and these other medical conditions was assessed using weighted Pearson correlation coefficients. Among 26 270 patients with in-hospital cardiac arrest at 130 hospitals, survival rates varied across hospitals, with a median risk-standardized hospital survival rate of 22.1% and an interquartile range of 19.7% to 24.2%. There were no significant correlations between a hospital's outcomes for its cardiac arrest patients and its patients admitted for acute myocardial infarction (correlation, -0.12; P=0.16), heart failure (correlation, -0.05; P=0.57), or pneumonia (correlation, -0.15; P=0.10).

CONCLUSIONS

Hospitals that performed better on publicly reported outcomes for 3 common medical conditions did not necessarily have better cardiac arrest survival rates. Public reporting on cardiac arrest outcomes could provide new information about hospital quality.