Report cards on cardiac surgeons. Assessing New York State's approach

The Impact of Congenital Cardiac Surgery Fellowship on Training and Practice

BACKGROUND

In 2007, congenital cardiac surgery became a recognized fellowship by the American Council of Graduate Medical Education (ACGME). Beginning in 2023, the fellowship transitioned from a 1-year to a 2-year program. Our objective is to provide current benchmarks by surveying current training programs and assessing characteristics contributing to career success.

METHODS

This was a survey-based study in which tailored questionnaires were distributed to program directors (PDs) and graduates of the ACGME accredited training programs. Data collection included responses to multiple-choice and open-ended questions relevant to didactics, operative training, training center characteristics, mentorship, and employment characteristics. Results were analyzed using summary statistics and subgroup and multivariable analyses.

RESULTS

The survey yielded responses from 13 of 15 PDs (86%) and 41 of 101 graduates (41%) from ACGME accredited programs. Perceptions among PDs and graduates were somewhat discordant, with PDs more optimistic than graduates. Of PDs, 77% (n = 10) believed current training adequately prepares fellows and is successful in securing employment for graduates. The responses from graduates demonstrated 30% (n = 12) were dissatisfied with operative experience and 24% (n = 10) with overall training. Being supported during the first 5 years of practice was significantly associated with retention in congenital cardiac surgery and greater practicing case volumes.

CONCLUSIONS

Dichotomous views exist between graduates and PDs regarding success in training. Mentorship during the early career was associated with increased case volumes, career satisfaction, and retention in the congenital cardiac surgery field. Educational bodies should incorporate these elements during training and after graduation.

An econometric approach to aggregating multiple cardiovascular outcomes in German hospitals

OBJECTIVE

Development of an aggregate quality index to evaluate hospital performance in cardiovascular events treatment.

METHODS

We applied a two-stage regression approach using an accelerated failure time model based on variance weights to estimate hospital quality over four cardiovascular interventions: elective coronary bypass graft, elective cardiac resynchronization therapy, and emergency treatment for acute myocardial infarction. Mortality and readmissions were used as outcomes. For the estimation we used data from a statutory health insurer in Germany from 2005 to 2016.

RESULTS

The precision-based weights calculated in the first stage were higher for mortality than for readmissions. In general, teaching hospitals performed better in our ranking of hospital quality compared to non-teaching hospitals, as did private not-for-profit hospitals compared to hospitals with public or private for-profit ownership.

DISCUSSION

The proposed approach is a new method to aggregate single hospital quality outcomes using objective, precision-based weights. Likelihood-based accelerated failure time models make use of existing data more efficiently compared to widely used models relying on dichotomized data. The main advantage of the variance-based weights approach is that the extent to which an indicator contributes to the aggregate index depends on the amount of its variance.

A Novel Approach to Evaluating Cardiac Surgery Providers: An Alternative to the RAMR

OBJECTIVE

We propose an alternative to the Risk Adjusted Mortality Rate (RAMR), about which we identify four serious concerns. We apply our method to cardiac surgery.

DESIGN

We present a methodology that uses the upper and lower tail probabilities (UTP/LTP) of the binomial distribution to screen for poor/high performing providers.

STUDY SETTING

The New York State Department of Health (NYS DOH) publicly releases data on all cardiac surgery patients in the state. We download cardiac surgery data from the NYS DOH website for the years 2011 through 2013. The state's objective is to identify poorly performing hospitals and surgeons and thereby reduce deaths. NYS employs the RAMR.

RESULTS

The UTP/LTP approach agrees with the RAMR in its classification of all 132 surgeons and all 40 hospitals. However, performance is a continuous construct and strict categorization can lead to failure to identify marginal providers.

CONCLUSIONS

Our methodology addresses all four concerns regarding the RAMR. The UTP/LTP approach avoids inappropriate hypothesis testing and is consistent with standard statistical theory and practice in its approach to case volume. It does not require confidence intervals and it applies to all providers regardless of case volume.

The impact of surgical training on early and long-term outcomes after isolated aortic valve surgery

 

OBJECTIVES

Patients presenting with more comorbidities, requiring more complex cardiac surgical procedures and an increase in public scrutiny are impacting on training programme because of the perceived risk of worse outcomes. Hence, we aimed to provide evidence that trainees as the first operator can achieve comparable results to consultants when performing isolated surgical aortic valve replacement.

METHODS

From 1996 to 2017, 2919 patients underwent surgical aortic valve replacement at the Bristol Heart Institute, operated on by either a consultant (n = 2220) or a trainee (n = 870) as the first operator. Propensity score matching was used to adjust for imbalance in the baseline characteristics of the 2 groups.

RESULTS

Over a 21-year period, the proportion of trainee cases dropped from 41.5% to 25.9%. No differences in the rates and risk of in-hospital mortality, new cerebrovascular accidents, re-exploration for bleeding, deep sternal wound infection and length of stay were found between patients operated on in the 2 groups. Also, there was a comparable risk of late death between the 2 groups (HR 0.88; 95% CI 0.73–1.06; P = 0.27) and this was present regardless of trainees career level and patients surgical risk based on the EuroSCORE. Finally, we showed an increase in patients risk profile in the latest year but, this was not associated with the worst outcomes when trainees performed the operation.

CONCLUSIONS

Surgical aortic valve replacement is a safe and reproducible technique and regardless of the patient’s risk profile, and no differences in the outcomes between trainees and consultant cases were found.

Variation in outcomes across surgeons meeting the Leapfrog volume standard for complex oncologic surgery

BACKGROUND

A large body of evidence supports regionalization of complex oncologic surgery to high-volume surgeons at high-volume hospitals. However, whether there is heterogeneity of outcomes among high-volume surgeons at high-volume hospitals remains unknown.

METHODS

Patients who underwent esophagectomy, lung resection, pancreatectomy, or proctectomy for primary cancer were identified within the Medicare 100% Standard Analytic File (2013-2017). Mixed-effects analyses assessed the association between Leapfrog annual volume standards for surgeons (esophagectomy ≥7, lung resection ≥15, pancreatectomy ≥10, proctectomy ≥6) and hospitals (esophagectomy ≥20, lung resection ≥40, pancreatectomy ≥20, proctectomy ≥16) relative to postoperative complications and 90-day mortality. Additional analyses using New York's all-payer Statewide Planning and Research Cooperative System (2004-2015) were performed.

RESULTS

Among 112,154 Medicare beneficiaries, high-volume surgeons at high-volume hospitals were associated with lower adjusted odds of complications (esophagectomy: odds ratio [OR], 0.73 [95% CI, 0.61-0.86]; lung resection: OR, 0.88 [95% CI, 0.82-0.94]; pancreatectomy: OR, 0.73 [95% CI, 0.66-0.80]; proctectomy: OR, 0.92 [95% CI, 0.85-0.99]) and 90-day mortality (esophagectomy: OR, 0.60 [95% CI, 0.44-0.76]; lung resection: OR, 0.82 [95% CI, 0.73-0.93]; pancreatectomy: OR, 0.66 [95% CI, 0.56-0.76]; proctectomy: OR, 0.74 [95% CI, 0.65-0.85]). For the average patient at the average high-volume hospital, there was a 2-fold difference in the adjusted complication rate between the best-performing and worst-performing high-volume surgeon for all operations (esophagectomy, 28%-55%; lung resection, 7%-21%; pancreatectomy, 16%-35%; proctectomy, 16%-28%). Wide variation was also present in adjusted 90-day mortality for esophagectomy (3.5%-9.3%). Results from New York's all-payer database were similar.

CONCLUSIONS

Even among high-volume surgeons meeting the Leapfrog volume standards, wide variation in postoperative outcomes exists. These findings suggest that volume alone should not be used as a quality indicator, and quality metrics should be continuously evaluated across all surgeons and hospital systems.

LAY SUMMARY

Previous studies have demonstrated a surgical volume-outcome relationship for high-risk operations-that is high-volume surgeons and hospitals that perform a specific surgical procedure more frequently have better outcomes for that operation. Although most high-volume surgeons had better outcomes, this study demonstrated that some high-volume surgeons did not have better outcomes. Therefore, volume is an important factor but should not be the only factor considered when assessing the quality of a surgeon and a hospital for cancer surgery.

Going the Extra Mile? How Provider Network Design Increases Consumer Travel Distance, Particularly for Rural Consumers

CONTEXT

The practical accessibility to medical care facilitated by health insurance plans depends not just on the number of providers within their networks but also on distances consumers must travel to reach the providers. Long travel distances inconvenience almost all consumers and may substantially reduce choice and access to providers for some.

METHODS

The authors assess mean and median travel distances to cardiac surgeons and pediatricians for participants in (1) plans offered through Covered California, (2) comparable commercial plans, and (3) unrestricted open-network plans. The authors repeat the analysis for higher-quality providers.

FINDINGS

The authors find that in all areas, but especially in rural areas, Covered California plan subscribers must travel longer than subscribers in the comparable commercial plan; subscribers to either plan must travel substantially longer than consumers in open networks. Analysis of access to higher-quality providers show somewhat larger travel distances. Differences between ACA and commercial plans are generally substantively small.

CONCLUSIONS

While network design adds travel distance for all consumers, this may be particularly challenging for transportation-disadvantaged populations. As distance is relevant to both health outcomes and the cost of obtaining care, this analysis provides the basis for more appropriate measures of network adequacy than those currently in use.

Trends in Diagnosis Related Groups for Inpatient Admissions and Associated Changes in Payment From 2012 to 2016

IMPORTANCE

Hospitals are reimbursed based on Diagnosis Related Groups (DRGs), which are defined, in part, by patients having 1 or more complications or comorbidities within a given DRG family. Hospitals have made substantial investment in efforts to document these complications and comorbidities.

OBJECTIVE

To examine temporal trends in DRGs with a major complication or comorbidity, compare these findings with 2 alternative measures of disease severity, and estimate associated changes in payment.

DESIGN, SETTING, AND PARTICIPANTS

This retrospective cohort study used data from the all-payer National Inpatient Sample for admissions assigned to 1 of the top 20 reimbursed DRG families at US acute care hospitals from January 1, 2012, to December 31, 2016. Data were analyzed from July 10, 2018, to May 29, 2019.

EXPOSURES

Quarter year of hospitalization.

MAIN OUTCOMES AND MEASURES

The primary outcome was the proportion of DRGs with a major complication or comorbidity. Secondary outcomes were comorbidity scores, risk-adjusted mortality rates, and estimated payment. Changes in assigned DRGs, comorbidity scores, and risk-adjusted mortality rates were analyzed by linear regression. Payment changes were estimated for each DRG by calculating the Centers for Medicare & Medicaid Services weighted payment using 2012 and 2016 case mix and hospitalization counts.

RESULTS

Between 2012 and 2016, there were 62 167 976 hospitalizations for the 20 highest-reimbursed DRG families; the sample was 32.9% male and 66.8% White, with a median age of 57 years (interquartile range, 31-73 years). Within 15 of these DRG families (75%), the proportion of DRGs with a major complication or comorbidity increased significantly over time. Over the same period, comorbidity scores were largely stable, with a decrease in 6 DRG families (30%), no change in 10 (50%), and an increase in 4 (20%). Among 19 DRG families with a calculable mortality rate, the risk-adjusted mortality rate significantly decreased in 8 (42%), did not change in 9 (47%), and increased in 2 (11%). The observed DRG shifts were associated with at least $1.2 billion in increased payment.

CONCLUSIONS AND RELEVANCE

In this cohort study, between 2012 and 2016, the proportion of admissions assigned to a DRG with major complication or comorbidity increased for 15 of the top 20 reimbursed DRG families. This change was not accompanied by commensurate increases in disease severity but was associated with increased payment.

A Knotty Problem: Consumer Access and the Regulation of Provider Networks

In order to increase access to medical services, expanding coverage has long been the preferred solution of policy makers and advocates alike. The calculus appeared straightforward: provide individuals with insurance, and they will be able to see a provider when needed. However, this line of thinking overlooks a crucial intermediary step: provider networks. As provider networks offered by health insurers link available medical services to insurance coverage, their breadth mediates access to health care. Yet the regulation of provider networks is technically, logistically, and normatively complex. What does network regulation currently look like and what should it look like in the future? We take inventory of the ways private and public entities regulate provider networks. Variation across insurance programs and products is truly remarkable, not grounded in empirical justification, and at times inherently absurd. We argue that regulators should be pragmatic and focus on plausible policy levers. These include assuring network accuracy, transparency for consumers, and consumer protections from grievous inadequacies. Ultimately, government regulation provides an important foundation for ensuring minimum levels of access and providing consumers with meaningful information. Yet, information is only truly empowering if consumers can exercise at least some choice in balancing costs, access, and quality.

Effect of Public Reporting on the Utilization of Coronary Angiography After Out-of-Hospital Cardiac Arrest

BACKGROUND

Public reporting of cardiovascular outcomes has been associated with risk aversion for potentially lifesaving procedures and may have spillover effects on nonreported but related procedures.

METHODS AND RESULTS

A cross-sectional analysis of the utilization of coronary angiography among patients presenting with out-of-hospital cardiac arrest between 2005 and 2011 in states with public reporting of percutaneous coronary intervention outcomes (New York and Massachusetts) versus neighboring states without public reporting of percutaneous coronary intervention outcomes (Delaware, Connecticut, Maine, Vermont, Maryland, and Rhode Island) was performed using the Nationwide Inpatient Sample. We analyzed 50 125 admission records with out-of-hospital cardiac arrest between 2005 and 2011. The unadjusted rate of coronary angiography for patients presenting with out-of-hospital cardiac arrest in states with public reporting versus without public reporting was not different (20.8% versus 22.8%, P=0.35). We found no statistically significant difference in the adjusted likelihood of coronary angiography in states with public reporting, though the point estimate suggested decreased utilization (odds ratio, 0.84; 95% CI, 0.66-1.06; P=0.14). There was no difference in the adjusted likelihood of in-hospital mortality for patients presenting with out-of-hospital cardiac arrest in states with public reporting compared to states without public reporting (odds ratio, 0.98; 95% CI, 0.78-1.23; P=0.88).

CONCLUSIONS

Public reporting of percutaneous coronary intervention outcomes was associated with a nonstatistically significant reduction in the utilization of diagnostic coronary angiography, a nonreported but related procedure, for patients with out-of-hospital cardiac arrest.

Mandatory public reporting of cardiac surgery outcomes: The 2003 to 2014 Massachusetts experience

OBJECTIVES

Beginning in 2002, all 14 Massachusetts nonfederal cardiac surgery programs submitted Society of Thoracic Surgeons (STS) National Database data to the Massachusetts Data Analysis Center for mandatory state-based analysis and reporting, and to STS for nationally benchmarked analyses. We sought to determine whether longitudinal prevalences and trends in risk factors and observed and expected mortality differed between Massachusetts and the nation.

METHODS

We analyzed 2003 to 2014 expected (STS predicted risk of operative [in-hospital + 30-day] mortality), observed, and risk-standardized isolated coronary artery bypass graft mortality using Massachusetts STS data (N = 39,400 cases) and national STS data (N = 1,815,234 cases). Analyses included percentage shares of total Massachusetts coronary artery bypass graft volume and expected mortality rates of 2 hospitals before and after outlier designation.

RESULTS

Massachusetts patients had significantly higher odds of diabetes, peripheral vascular disease, low ejection fraction, and age ≥75 years relative to national data and lower odds of shock (odds ratio, 0.66; 99% confidence interval, 0.53-0.83), emergency (odds ratio, 0.57, 99% confidence interval, 0.52-0.61), reoperation, chronic lung disease, dialysis, obesity, and female sex. STS predicted risk of operative [in-hospital + 30-day] mortality for Massachusetts patients was higher than national rates during 2003 to 2007 (P < .001) and no different during 2008 to 2014 (P = .135). Adjusting for STS predicted risk of operative [in-hospital + 30-day] mortality, Massachusetts patients had significantly lower odds (odds ratio, 0.79; 99% confidence interval, 0.66-0.96) of 30-day mortality relative to national data. Outlier programs experienced inconsistent, transient influences on expected mortality and their percentage shares of Massachusetts coronary artery bypass graft cases.

CONCLUSIONS

During 12 years of mandatory public reporting, Massachusetts risk-standardized coronary artery bypass graft mortality was consistently and significantly lower than national rates, expected rates were comparable or higher, and evidence for risk aversion was conflicting and inconclusive.

The effect of consultant outcome publication on surgeon behaviour: a systematic review and narrative synthesis

Introduction Surgeon-specific outcome data, or consultant outcome publication, refers to public access to named surgeon procedural outcomes. Consultant outcome publication originates from cardiothoracic surgery, having been introduced to US and UK surgery in 1991 and 2005, respectively. It has been associated with an improvement in patient outcomes. However, there is concern that it may also have led to changes in surgeon behaviour. This review assesses the literature for evidence of risk-averse behaviour, upgrading of patient risk factors and cessation of low-volume or poorly performing surgeons. Materials and methods A systematic literature review of Embase and Medline databases was performed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines. Original studies including data on consultant outcome publication and its potential effect on surgeon behaviour were included. Results Twenty-five studies were identified from the literature search. Studies suggesting the presence of risk-averse behaviour and upgrading of risk factors tended to be survey based, with studies contrary to these findings using recognised regional and national databases. Discussion and conclusion Our review includes instances of consultant outcome publication leading to risk-averse behaviour, upgrading of risk factors and cessation of low-volume or poorly performing surgeons. As UK data on consultant outcome publication matures, further research is essential to ensure that high-risk patients are not inappropriately turned down for surgery.

Effect of public reporting of surgeons' outcomes on patient selection, "gaming," and mortality in colorectal cancer surgery in England: population based cohort study

OBJECTIVE

To determine the effect of surgeon specific outcome reporting in colorectal cancer surgery on risk averse clinical practice, "gaming" of clinical data, and 90 day postoperative mortality.

DESIGN

National cohort study.

SETTING

English National Health Service hospital trusts.

POPULATION

111 431 patients diagnosed as having colorectal cancer from 1 April 2011 to 31 March 2015 included in the National Bowel Cancer Audit.

INTERVENTION

Public reporting of surgeon specific 90 day mortality in elective colorectal cancer surgery in England introduced in June 2013.

MAIN OUTCOME MEASURES

Proportion of patients with colorectal cancer who had an elective major resection, predicted 90 day mortality based on characteristics of patients and tumours, and observed 90 day mortality adjusted for differences in characteristics of patients and tumours, comparing patients who had surgery between April 2011 and June 2013 and between July 2013 and March 2015.

RESULTS

The proportion of patients with colorectal cancer undergoing major resection did not change after the introduction of surgeon specific public outcome reporting (39 792/62 854 (63.3%) before versus 30 706/48 577 (63.2%) after; P=0.8). The proportion of these major resections categorised as elective or scheduled also did not change (33 638/39 792 (84.5%) before versus 25 905/30 706 (84.4%) after; P=0.5). The predicted 90 day mortality remained the same (2.7% v 2.7%; P=0.3), but the observed 90 day mortality fell (952/33 638 (2.8%) v 552/25 905 (2.1%)). Change point analysis showed that this reduction was over and above the existing downward trend in mortality before the introduction of public outcome reporting (P=0.03).

CONCLUSIONS

This study did not find evidence that the introduction of public reporting of surgeon specific 90 day postoperative mortality in elective colorectal cancer surgery has led to risk averse clinical practice behaviour or "gaming" of data. However, its introduction coincided with a significant reduction in 90 day mortality.

Challenges in Outcome Reporting

Although measuring outcomes is an integral part of medical quality improvement, large-scale outcome reporting efforts face several challenges. Among these are difficulties in establishing consensus definitions for outcome measurement; classifying gray outcomes, such as postoperative respiratory failure; and adequately adjusting for patient comorbidities and severity of illness. Unintended consequences of outcome reporting can also distort care in undesirable ways, and clinician reluctance to care for high-risk patients may occur with reporting programs. Ultimately, clinicians need not compare outcomes to improve and should recognize that even outcomes that cannot be precisely quantitated can still be improved.

Association of Coded Severity With Readmission Reduction After the Hospital Readmissions Reduction Program

This study uses the Medicare Provider Analysis and Review file to examine whether coded severity of illness is associated with reduced rates of readmission after implementation of the Hospital Readmissions Reduction Program.

Impact of Public Reporting of 30-day Mortality on Timing of Death after Coronary Artery Bypass Graft Surgery

BACKGROUND

Recent reports have raised concerns that public reporting of 30-day mortality after cardiac surgery may delay decisions to withdraw life-sustaining therapies for some patients. The authors sought to examine whether timing of mortality after coronary artery bypass graft surgery significantly increases after day 30 in Massachusetts, a state that reports 30-day mortality. The authors used New York as a comparator state, which reports combined 30-day and all in-hospital mortality, irrespective of time since surgery.

METHODS

The authors conducted a retrospective cohort study of patients who underwent coronary artery bypass graft surgery in hospitals in Massachusetts and New York between 2008 and 2013. The authors calculated the empiric daily hazard of in-hospital death without censoring on hospital discharge, and they used joinpoint regression to identify significant changes in the daily hazard over time.

RESULTS

In Massachusetts and New York, 24,864 and 63,323 patients underwent coronary artery bypass graft surgery, respectively. In-hospital mortality was low, with 524 deaths (2.1%) in Massachusetts and 1,398 (2.2%) in New York. Joinpoint regression did not identify a change in the daily hazard of in-hospital death at day 30 or 31 in either state; significant joinpoints were identified on day 10 (95% CI, 7 to 15) for Massachusetts and days 2 (95% CI, 2 to 3) and 12 (95% CI, 8 to 15) for New York.

CONCLUSIONS

In Massachusetts, a state with a long history of publicly reporting cardiac surgery outcomes at day 30, the authors found no evidence of increased mortality occurring immediately after day 30 for patients who underwent coronary artery bypass graft surgery. These findings suggest that delays in withdrawal of life-sustaining therapy do not routinely occur as an unintended consequence of this type of public reporting.

Levosimendan in patients with left ventricular systolic dysfunction undergoing cardiac surgery on cardiopulmonary bypass: Rationale and study design of the Levosimendan in Patients with Left Ventricular Systolic Dysfunction Undergoing Cardiac Surgery Requiring Cardiopulmonary Bypass (LEVO-CTS) trial

BACKGROUND

Low cardiac output syndrome is associated with increased mortality and occurs in 3% to 14% of patients undergoing cardiac surgery on cardiopulmonary bypass (CPB). Levosimendan, a novel calcium sensitizer and K_ATP channel activator with inotropic, vasodilatory, and cardioprotective properties, has shown significant promise in reducing the incidence of low cardiac output syndrome and related adverse outcomes in patients undergoing cardiac surgery on CPB.

METHODS

LEVO-CTS is a phase 3 randomized, controlled, multicenter study evaluating the efficacy, safety, and cost-effectiveness of levosimendan in reducing morbidity and mortality in high-risk patients with reduced left ventricular ejection fraction (≤35%) undergoing cardiac surgery on CPB. Patients will be randomly assigned to receive either intravenous levosimendan (0.2 μg kg^-1 min^-1 for the first hour followed by 0.1 μg/kg for 23hours) or matching placebo initiated within 8hours of surgery. The co-primary end points are (1) the composite of death or renal replacement therapy through day 30 or perioperative myocardial infarction, or mechanical assist device use through day 5 (quad end point tested at α<.01), and (2) the composite of death through postoperative day 30 or mechanical assist device use through day 5 (dual end point tested at α<.04). Safety end points include new atrial fibrillation and death through 90days. In addition, an economic analysis will address the cost-effectiveness of levosimendan compared with placebo in high-risk patients undergoing cardiac surgery on CPB. Approximately 880 patients will be enrolled at approximately 60 sites in the United States and Canada between July 2014 and September 2016, with results anticipated in January 2017.

CONCLUSION

LEVO-CTS, a large randomized multicenter clinical trial, will evaluate the efficacy, safety, and cost-effectiveness of levosimendan in reducing adverse outcomes in high-risk patients undergoing cardiac surgery on CPB.

CLINICAL TRIAL REGISTRATION

ClinicalTrials.gov (NCT02025621).

Do Cardiac Surgery Report Cards Reduce Mortality? Assessing the Evidence

Provider report cards feature prominently in ongoing efforts to improve patient quality. A well-known example is the cardiac surgery report-card program started in New York, which publicly compares hospital and surgeon performance. Public report cards have been associated with decreases in cardiac surgery mortality, but there is substantial disagreement over the source(s) of the improvement. This article develops a conceptual framework to explain how report-card-related responses could result in lower mortality and reviews the evidence. Existing research shows that report cards have not greatly changed referral patterns. How much providers increased their quality of care and altered their selection of patients remains unresolved, and alternative explanations have not been well studied. Future research should expand the number of states and years covered and exploit the variation in institutional features to improve our understanding of the relationship between report cards and outcomes.

Public Disclosure to Improve Physical Education in an Urban School District: Results From a 2-Year Quasi-Experimental Study

BACKGROUND

Many elementary schools have policies requiring a minimum amount of physical education (PE). However, few schools comply with local/state PE policy and little is known about how to improve adherence. We evaluated changes in PE among fifth-grade classes, following participatory action research efforts to improve PE quantity and policy compliance that focused on publically disclosing PE data.

METHODS

Data were collected in 20 San Francisco public elementary schools in spring 2011 and 2013. PE schedules were collected and PE classes were directly observed (2011, N = 30 teachers; 2013, N = 33 teachers). Data on the proportion of schools meeting state PE mandates in 2011 were shared within the school district and disclosed to the general public in 2012.

RESULTS

From 2011 to 2013, PE increased by 11 minutes/week based on teachers' schedules (95% CI: 3.0, 19.6) and by 14 minutes/week (95% CI: 1.9, 26.0) based on observations. The proportion of schools meeting the state PE mandate increased from 20% to 30% (p = .27).

CONCLUSIONS

Positive changes in PE were seen over a 2-year period following the public disclosure of data that highlighted poor PE policy compliance. Public disclosure could be a method for ensuring greater PE policy adherence.

Reliability of surgeon-specific reporting of complications after colectomy

OBJECTIVE

We sought to determine the reliability of surgeon-specific postoperative complication rates after colectomy.

BACKGROUND

Conventional measures of surgeon-specific performance fail to acknowledge variation attributed to statistical noise, risking unreliable assessment of quality.

METHODS

We examined all patients who underwent segmental colectomy with anastomosis from 2008 through 2010 participating in the Michigan Surgical Quality Collaborative Colectomy Project. Surgeon-specific complication rates were risk-adjusted according to patient characteristics with multiple logistic regression. Hierarchical modeling techniques were used to determine the reliability of surgeon-specific risk-adjusted complication rates. We then adjusted these rates for reliability. To evaluate the extent to which surgeon-level variation was reduced, surgeons were placed into quartiles based on performance and complication rates were compared before and after reliability adjustment.

RESULTS

A total of 5033 patients (n = 345 surgeons) undergoing partial colectomy reported a risk-adjusted complication rate of 24.5%. Approximately 86% of the variability of complication rates across surgeons was explained by measurement noise, whereas the remaining 14% represented true signal. Risk-adjusted complication rates varied from 0% to 55.1% across quartiles before adjusting for reliability. Reliability adjustment greatly diminished this variation, generating a 1.2-fold difference (21.4%-25.6%). A caseload of 168 colectomies across 3 years was required to achieve a reliability of more than 0.7, which is considered a proficient level. Only 1 surgeon surpassed this volume threshold.

CONCLUSIONS

The vast majority of surgeons do not perform enough colectomies to generate a reliable surgeon-specific complication rate. Risk-adjusted complication rates should be viewed with caution when evaluating surgeons with low operative volume, as statistical noise is a large determinant in estimating their surgeon-specific complication rates.

The impact of high-risk cases on hospitals' risk-adjusted coronary artery bypass grafting mortality rankings

BACKGROUND

Risk-adjusted mortality (RAM) models are increasingly used to evaluate hospital performance, but the validity of the RAM method has been questioned. Providers are concerned that these methods might not adequately account for the highest levels of risk and that treating high-risk cases will have a negative impact on RAM rankings.

METHODS

Using cases of isolated coronary artery bypass grafting (CABG) performed at 1002 sites in the United States participating in The Society of Thoracic Surgeons (STS) Adult Cardiac Surgery Database from 2008 to 2010 (N = 494,955), the STS CABG RAM model performance in high-risk patients was assessed. The ratios of observed to expected (O/E) perioperative mortality were compared among groups of hospitals with varying expected risks. Finally, RAM rates during the overall study period for each site were compared with its performance in a simulated "nightmare year" in which the site's highest risk cases over a 3-year period were concentrated into a 1-year period of exceptional risk.

RESULTS

The average predicted mortality for center risk groups ranged from 1.46% for the lowest risk quintile to 2.87% for the highest. The O/E ratios for center risk quintiles 1 to 5 during the overall period were 1.01 (95% confidence interval, 0.96% to 1.06%), 1.00 (0.95% to 1.04%), 0.98 (0.94% to 1.03%), 0.97 (0.93% to 1.01%), and 0.80 (0.77% to 0.84%), respectively. The sites' risk-adjusted mortality rates were not increased when the centers' highest risk cases were concentrated into a single "nightmare year."

CONCLUSIONS

Our results show that the current risk-adjusted models accurately estimate CABG mortality and that hospitals accepting more high-risk CABG patients have equal or better outcomes than do those with predominately lower-risk patients.

Ranking and rankability of hospital postoperative mortality rates in colorectal cancer surgery

OBJECTIVES

To examine to what extent random variation and variation in case-mix influence hospital rankings on the basis of mortality rates and to determine the suitability of mortality for ranking hospitals in colorectal surgery.

BACKGROUND

Comparing and ranking postoperative mortality rates between hospitals becomes increasingly popular. Differences in hospital case-mix, and chance variation related to caseload, may influence rankings. The suitability of mortality for rankings remains unclear.

METHODS

Data were derived from the Dutch Surgical Colorectal Audit. Hospital rankings based on fixed- and random-effects logistic regression models, unadjusted and adjusted for case-mix were compared with the percentile based on expected ranks (the chance that a hospital performs better than a random hospital). Rankability, measuring which part of variation between hospitals is not due to chance, was calculated.

RESULTS

Some 25,591 patients undergoing colorectal resections in 92 hospitals were evaluated. Postoperative mortality rates ranged between 0% and 8.8%. Adjustment for case-mix with a fixed-effects model caused large changes in rankings. A smaller additional effect on changes in rankings occurred after adjusting with a random-effects model, with lower volume hospitals moving toward the mean. Percentile based on expected ranks ranged between 10% and 85%. Rankability was 38%, meaning that 62% of hospital variation in mortality was due to chance.

CONCLUSIONS

Hospital ranks changed after case-mix adjustment and random-effects models, compared with unadjusted analysis. A large proportion of hospital variation in mortality was due to chance. Caution should be warranted when interpreting hospital rankings on the basis of postoperative mortality. Percentiles of expected ranks may help identify hospitals with exceptional performance.

Impact of type of procedure and surgeon on EuroSCORE operative risk validation

OBJECTIVE

EuroSCORE has been used in cardiac surgery operative risk assessment, despite important variables were not included. The objective of this study was to validate EuroSCORE on mortality prediction in a Brazilian cardiovascular surgery center, defining the influence of type of procedure and surgical team.

METHODS

Between January 2006 and June 2011, 2320 consecutive adult patients were studied. According to additive EuroSCORE, patients were divided into low risk (score<2), medium risk (3 - 5), high risk (6 - 11) and very high risk (>12). The relation between observed mortality (O) and expected mortality (E) according to logistic EuroSCORE was calculated for each of the groups, types of procedures and surgeons with > 150 operations, and analyzed by logistic regression.

RESULTS

EuroSCORE correlated to the observed mortality (O/E=0.94; P<0.0001; area under the curve 0.78). However, it overestimated the mortality in very high risk patients (O/E=0.74; P=0.001). EuroSCORE tended to overestimate isolated myocardial revascularization mortality (O/E=0.81; P=0.0001) and valve surgery mortality (O/E=0.89; P=0.007) and it tended to underestimate combined procedures mortality (O/E=1.09; P<0.0001). EuroSCORE overestimated surgeon A mortality (O/E=0.46; P<0.0001) and underestimated surgeon B mortality (O/E=1.3; P<0.0001), in every risk category.

CONCLUSION

In the present population, EuroSCORE overestimates mortality in very high risk patients, being influenced by type of procedure and surgical team. The most appropriate surgical team may minimize risks imposed by preoperative profiles.

Dissemination of performance information and continuous improvement: A narrative systematic review

PURPOSE

Developing a performance measure and reporting the results to support decision making at an individual level has yielded poor results in many health systems. The purpose of this paper is to highlight the factors associated with the dissemination of performance information that generate and support continuous improvement in health organizations.

DESIGN/METHODOLOGY/APPROACH

A systematic data collection strategy that includes empirical and theoretical research published from 1980 to 2010, both qualitative and quantitative, was performed on Web of Science, Current Contents, EMBASE and MEDLINE. A narrative synthesis method was used to iteratively detail explicative processes that underlie the intervention. A classification and synthesis framework was developed, drawing on knowledge transfer and exchange (KTE) literature. The sample consisted of 114 articles, including seven systematic or exhaustive reviews.

FINDINGS

Results showed that dissemination in itself is not enough to produce improvement initiatives. Successful dissemination depends on various factors, which influence the way collective actors react to performance information such as the clarity of objectives, the relationships between stakeholders, the system's governance and the available incentives.

RESEARCH LIMITATIONS/IMPLICATIONS

This review was limited to the process of knowledge dissemination in health systems and its utilization by users at the health organization level. Issues related to improvement initiatives deserve more attention.

PRACTICAL IMPLICATIONS

Knowledge dissemination goes beyond better communication and should be considered as carefully as the measurement of performance. Choices pertaining to intervention should be continuously prompted by the concern to support organizational action.

ORIGINALITY/VALUE

While considerable attention was paid to the public reporting of performance information, this review sheds some light on a more promising avenue for changes and improvements, notably in public health systems.

Statistical methods to monitor risk factors in a clinical database: example of a national cardiac surgery registry

BACKGROUND

Comparison of outcomes requires adequate risk adjustment for differences in patient risk and the type of intervention performed. Both unintentional and intentional misclassification (also called gaming) of risk factors might lead to incorrect benchmark results. Therefore, misclassification of risk factors should be detected. We investigated the use of statistical process control techniques to monitor the frequency of risk factors in a clinical database.

METHODS AND RESULTS

A national population-based study was performed using simulation and statistical process control. All patients who underwent cardiac surgery between January 1, 2007, and December 31, 2009, in all 16 cardiothoracic surgery centers in the Netherlands were included. Data on 46 883 consecutive cardiac surgery interventions were extracted. The expected risk factor frequencies were based on 2007 and 2008 data. Monthly frequency rates of 18 risk factors in 2009 were monitored using a Shewhart control chart, exponentially weighted moving average chart, and cumulative sum chart. Upcoding (ie, gaming) in random patients was simulated and detected in 100% of the simulations. Subtle forms of gaming, involving specifically high-risk patients, were more difficult to identify (detection rate of 44%). However, the accompanying rise in mean logistic European system for cardiac operative risk evaluation (EuroSCORE) was detected in all simulations.

CONCLUSIONS

Statistical process control in the form of a Shewhart control chart, exponentially weighted moving average, and cumulative sum charts provide a means to monitor changes in risk factor frequencies in a clinical database. Surveillance of the overall expected risk in addition to the separate risk factors ensures a high sensitivity to detect gaming. The use of statistical process control for risk factor surveillance is recommended.

Analytical perspectives on performance-based management: an outline of theoretical assumptions in the existing literature

Performance-based management (PBM) has become a dominant form of governance in health care and there is a need for careful assessment of its function and effects. This article contains a cross-disciplinary literature synthesis of current studies of PBM. Literature was retrieved by database searches and categorized according to analytical differences and similarities concerning (1) purpose and (2) governance mechanism of PBM. The literature could be grouped into three approaches to the study of PBM, which we termed: the ‘functionalist’, the ‘interpretive’ and the ‘post-modern’ perspective. In the functionalist perspective, PBM is perceived as a management tool aimed at improving health care services by means of market-based mechanisms. In the interpretive perspective, the adoption of PBM is understood as consequence of institutional and individual agents striving for public legitimacy. In the post-modern perspective, PBM is analysed as a form of governance, which has become so ingrained in Western culture that health care professionals internalize and understand their own behaviour and goals according to the values expressed in these governance systems. The recognition of differences in analytical perspectives allows appreciation of otherwise implicit assumptions and potential implications of PBM. Reflections on such differences are important to ensure vigilant appropriation of shifting management tools in health quality governance.

Target setting for elective infra-renal AAA surgery: A single mortality figure?

OBJECTIVES

One of the key standards set by the UK NAAASP is that centres performing elective abdominal aortic aneurysm (AAA) repair have a mortality rate of <6%. In light of this, and the current aim to reduce elective AAA repair mortality to 3.5% by 2013, we sought to investigate the statistical validity of such targets.

METHODS

The National Vascular Database (NVD) was interrogated and the degree of AAA missing data and its geographical variation is described. Utilising published data from 2006 to 2008 a funnel plot was used to illustrate NHS Trust level data for current estimates of mortality rate. A binomial distribution model was applied to calculate variation in observed mortality rates in relation to number of patients treated, based on a "true" mortality rate of 3.5%. Funnel plots were constructed using simulated data-sets for units performing 10, 30, 50, 100 or 150 procedures annually with control-limits calculated using a cumulative probability distribution. Finally the effect of case-mix on mortality was modelled and shown graphically.

RESULTS

The NVD AAA data set shows a range of data missingness across variables (median 22%, IQR 10-64%). High levels of missingness typically coincide with non-required, non-preferred variables however this is subject to geographical variation. Funnel plots of simulated data demonstrate that smaller units have greater variability in 3-year mortality (range 0.0-10.0%) than the largest units performing 150 procedures annually (1.3-5.6%). Around 20% of NVD variables are described as "preferred", these typically relate to clinical measurements and patient medications and would inform any risk model of mortality. Data missingness amongst these variables ranges from 5 to 50%.

CONCLUSIONS

There are many problems with the use of a single mortality figure to assess performance. These include the natural statistical variability and the means by which "case-mix" is taken into consideration. This article calls for further research into mortality target setting and suggests strategies which may help provide solutions nationally and facilitate international comparison.

Public reporting on hospital process improvements is linked to better patient outcomes

The Centers for Medicare and Medicaid Services publicly reports so-called process performance at all U.S. hospitals, such as whether certain recommended treatments are given to specific types of patients. We examined whether hospital performance on key process indicators improved during the three years since this reporting began. We also studied whether or not these changes improved patient outcomes or yielded other quality improvements, such as reduced hospital readmission rates. We found that, from 2004 to 2006, hospital process performance improved and was associated with better patient and quality outcomes. Most notably, for acute myocardial infarction, performance improvements were associated with declines in mortality rates, lengths-of-stay, and readmission rates. Although we cannot conclude that public reporting caused the improvement in processes or outcomes, these results are encouraging, since improving process performance may improve quality more broadly.

Limitations of Ranking Lists Based on Cardiac Surgery Mortality Rates

Background—

Ranking lists are a common way of reporting performance in cardiac surgery; however, rankings have shown to be imprecise, yet the extent of this imprecision is unknown. We aimed to determine the precision of, and fluctuations in, ranking lists in the comparison of cardiac surgery mortality rates.

Methods and Results—

Information on all adult cardiac surgery patients in all 16 cardiothoracic centers in The Netherlands from January 1, 2007, until December 31, 2009, was extracted from the database of the Netherlands Association for Cardio-Thoracic Surgery (n=46883). Ranks were assessed using crude and adjusted mortality rates, using a random effects logistic regression model. Risk adjustment was performed using the logistic EuroSCORE. Statistical precision of ranks was assessed with 95% confidence intervals. Additional analyses were performed for patients with isolated coronary artery bypass grafting. The ranking lists, based on mortality rates in 3 consecutive years, showed considerable reshuffling. When all data were pooled, the mean width of the 95% confidence intervals was 10 ranks using crude and 8 ranks using adjusted mortality rates. The large overlap of the confidence intervals across hospitals indicates that rank statistics were not materially different. Results were similar in the isolated coronary artery bypass grafting subgroup.

Conclusions—

Rankings are an imprecise statistical method to report cardiac surgery mortality rates and prone to (random) fluctuation. Hence, reshuffling of ranks can be expected solely due to chance. Therefore, we strongly discourage the use of ranking lists in the comparison of mortality rates.

Hospital-wide mortality as a quality metric: conceptual and methodological challenges

Hospital-wide mortality rates are used as a measure of overall hospital quality. However, their parsimony and apparent simplicity belie significant conceptual and methodological concerns. For many diagnoses included in hospital-wide mortality, the association between short-term mortality and quality of care is not well established. Furthermore, compared with condition-specific or procedure-specific mortality, hospital-wide mortality rates pose greater methodological challenges (ie, eligibility and exclusion criteria, risk adjustment, statistical techniques for aggregating across diagnoses, usability). Many of these result from substantial interprovider heterogeneity in diagnosis frequency, sample sizes, and patient severity. Hospital-wide mortality is problematic as a quality metric for public reporting, although hospitals may elect to use such measures for other purposes. Potential alternative approaches include multidimensional composite metrics or mortality measurement limited to selected conditions and procedures for which the link between hospital mortality and quality is clear, legitimate exclusions are uncommon, and sample sizes, end points, and risk adjustment are adequate.