Longitudinal profiles of health care providers

Are Case Volume and Facility Complexity Level Associated With Postoperative Complications After Hip Fracture Surgery in the Veterans Affairs Healthcare System?

BACKGROUND

Hospital-related factors associated with mortality and morbidity after hip fracture surgery are not completely understood. The Veterans Health Administration (VHA) is the largest single-payer, networked healthcare system in the country serving a relatively homogenous patient population with facilities that vary in size and resource availability. These characteristics provide some degree of financial and patient-level controls to explore the association, if any, between surgical volume and facility resource availability and hospital performance regarding postoperative complications after hip fracture surgery.

QUESTIONS/PURPOSES

(1) Do VHA facilities with the highest complexity level designation (Level 1a) have a disproportionate number of better-than-expected performance outliers for major postoperative complications compared with lower-complexity level facilities? (2) Do VHA facilities with higher hip fracture surgical volume have a disproportionate number of better-than-expected performance outliers for major postoperative complications compared with lower-volume facilities?

METHODS

We explored the Veterans Affairs Surgical Quality Improvement Project (VASQIP) database from October 2001 to September 2012 for records of hip fracture surgery performed. Data reliability of the VASQIP database has been previously validated. We excluded nine of the 98 VHA facilities for contributing fewer than 30 records. The remaining 89 VHA facilities provided 23,029 records. The VHA designates a complexity level to each facility based on multiple criteria. We labeled facilities with a complexity Level 1a (38 facilities)-the highest achievable VHA designated complexity level-as high complexity; we labeled all other complexity level designations as low complexity (51 facilities). Facility volume was divided into tertiles: high (> 277 hip fracture procedures during the sampling frame), medium (204 to 277 procedures), and low (< 204 procedures). The patient population treated by low-complexity facilities was older, had a higher prevalence of severe chronic obstructive pulmonary disease (26% versus 22%, p < 0.001), and had a higher percentage of patients having surgery within 2 days of hospital admission (83% versus 76%, p < 0.001). High-complexity facilities treated more patients with recent congestive heart failure exacerbation (4% versus 3%, p < 0.001). We defined major postoperative complications as having at least one of the following: death within 30 days of surgery, cardiac arrest requiring cardiopulmonary resuscitation, new q-wave myocardial infarction, deep vein thrombosis and/or pulmonary embolism, ventilator dependence for at least 48 hours after surgery, reintubation for respiratory or cardiac failure, acute renal failure requiring renal replacement therapy, progressive renal insufficiency with a rise in serum creatinine of at least 2 mg/dL from preoperative value, pneumonia, or surgical site infection. We used the observed-to-expected ratio (O/E ratio)-a risk-adjusted metric to classify facility performance-for major postoperative complications to assess the performance of VHA facilities. Outlier facilities with 95% confidence intervals (95% CI) for O/E ratio completely less than 1.0 were labeled "exceed expectation;" those that were completely greater than 1.0 were labeled "below expectation." We compared differences in the distribution of outlier facilities between high and low-complexity facilities, and between high-, medium-, and low-volume facilities using Fisher's exact test.

RESULTS

We observed no association between facility complexity level and the distribution of outlier facilities (high-complexity: 5% exceeded expectation, 5% below expectation; low-complexity: 8% exceeded expectation, 2% below expectation; p = 0.742). Compared with high-complexity facilities, the adjusted odds ratio for major postoperative complications for low-complexity facilities was 0.85 (95% CI, 0.67-1.09; p = 0.108).We observed no association between facility volume and the distribution of outlier facilities: 3% exceeded expectation and 3% below expectation for high-volume; 10% exceeded expectation and 3% below expectation for medium-volume; and 7% exceeded expectation and 3% below expectation for low-volume; p = 0.890). The adjusted odds ratios for major postoperative complications were 0.87 (95% CI, 0.73-1.05) for low- versus high-volume facilities and 0.89 (95% CI, 0.79-1.02] for medium- versus high-volume facilities (p = 0.155).

CONCLUSIONS

These results do not support restricting facilities from treating hip fracture patients based on historical surgical volume or facility resource availability. Identification of consistent performance outliers may help health care organizations with multiple facilities determine allocation of services and identify characteristics and processes that determine outlier status in the interest of continued quality improvement.

LEVEL OF EVIDENCE

Level III, therapeutic study.

Regulator Loss Functions and Hierarchical Modeling for Safety Decision Making

BACKGROUND

Regulators must act to protect the public when evidence indicates safety problems with medical devices. This requires complex tradeoffs among risks and benefits, which conventional safety surveillance methods do not incorporate.

OBJECTIVE

To combine explicit regulator loss functions with statistical evidence on medical device safety signals to improve decision making.

METHODS

In the Hospital Cost and Utilization Project National Inpatient Sample, we select pediatric inpatient admissions and identify adverse medical device events (AMDEs). We fit hierarchical Bayesian models to the annual hospital-level AMDE rates, accounting for patient and hospital characteristics. These models produce expected AMDE rates (a safety target), against which we compare the observed rates in a test year to compute a safety signal. We specify a set of loss functions that quantify the costs and benefits of each action as a function of the safety signal. We integrate the loss functions over the posterior distribution of the safety signal to obtain the posterior (Bayes) risk; the preferred action has the smallest Bayes risk. Using simulation and an analysis of AMDE data, we compare our minimum-risk decisions to a conventional Z score approach for classifying safety signals.

RESULTS

The 2 rules produced different actions for nearly half of hospitals (45%). In the simulation, decisions that minimize Bayes risk outperform Z score-based decisions, even when the loss functions or hierarchical models are misspecified.

LIMITATIONS

Our method is sensitive to the choice of loss functions; eliciting quantitative inputs to the loss functions from regulators is challenging.

CONCLUSIONS

A decision-theoretic approach to acting on safety signals is potentially promising but requires careful specification of loss functions in consultation with subject matter experts.

HOW MUCH FAVORABLE SELECTION IS LEFT IN MEDICARE ADVANTAGE?

The health economics literature contains two models of selection, one with endogenous plan characteristics to attract good risks and one with fixed plan characteristics; neither model contains a regulator. Medicare Advantage, a principal example of selection in the literature, is, however, subject to anti-selection regulations. Because selection causes economic inefficiency and because the historically favorable selection into Medicare Advantage plans increased government cost, the effectiveness of the anti-selection regulations is an important policy question, especially since the Medicare Advantage program has grown to comprise 30 percent of Medicare beneficiaries. Moreover, similar anti-selection regulations are being used in health insurance exchanges for those under 65. Contrary to earlier work, we show that the strengthened anti-selection regulations that Medicare introduced starting in 2004 markedly reduced government overpayment attributable to favorable selection in Medicare Advantage. At least some of the remaining selection is plausibly related to fixed plan characteristics of Traditional Medicare versus Medicare Advantage rather than changed selection strategies by Medicare Advantage plans.

Level-adjusted funnel plots based on predicted marginal expectations: an application to prophylactic antibiotics in gallstone surgery

Funnel plots are widely used to visualize grouped data, for example, in institutional comparison. This paper extends the concept to a multi-level setting, displaying one level at a time, adjusted for the other levels, as well as for covariates at all levels. These level-adjusted funnel plots are based on a Markov chain Monte Carlo fit of a random effects model, translating the estimated model parameters to predicted marginal expectations. Working within the estimation framework, we accommodate outlying institutions using heavy-tailed random effects distributions. We also develop computer-efficient methods to compute predicted probabilities in the case of dichotomous outcome data and various random effect distributions. We apply the method to a data set on prophylactic antibiotics in gallstone surgery.

Evaluating the performance of Australian and New Zealand intensive care units in 2009 and 2010

The Australian and New Zealand Intensive Care Society Adult Patient Database (ANZICS APD) is one of the largest databases of its kind in the world and collects individual admissions' data from intensive care units (ICUs) around Australia and New Zealand. Use of this database for monitoring and comparing the performance of ICUs, quantified by the standardised mortality ratio, poses several theoretical and computational challenges, which are addressed in this paper. In particular, the expected number of deaths must be appropriately estimated, the ICU casemix adjustment must be adequate, statistical variation must be fully accounted for, and appropriate adjustment for multiple comparisons must be made. Typically, one or more of these issues have been neglected in ICU comparison studies. Our approach to the analysis proceeds by fitting a random coefficient hierarchical logistic regression model for the inhospital death of each patient, with patients clustered within ICUs. We anticipate the majority of ICUs will be estimated as performing 'usually' after adjusting for important clinical covariates. We take as a starting point the ideas in Ohlssen et al and estimate an appropriate null model that we expect these ICUs to follow, taking a frequentist rather than a Bayesian approach. This methodology allows us to rigorously account for the aforementioned statistical issues and to determine if there are any ICUs contributing to the Australian and New Zealand Intensive Care Society database that have comparatively unusual performance. In addition to investigating the yearly performance of the ICUs, we also estimate changes in individual ICU performance between 2009 and 2010 by adjusting for regression-to-the-mean.

Profiling hospitals based on emergency readmission: a multilevel transition modelling approach

Emergency readmission is seen as an important part of the United Kingdom government policy to improve the quality of care that patients receive. In this context, patients and the public have the right to know how well different health organizations are performing. Most methods for profiling estimate the expected numbers of adverse outcomes (e.g. readmission, mortality) for each organization. A number of statistical concerns have been raised, such as the differences in hospital sizes and the unavailability of relevant data for risk adjustment. Having recognized these statistical concerns, a new framework known as the multilevel transition model is developed. Hospital specific propensities of the first, second and further readmissions are considered to be measures of performance, where these measures are used to define a new performance index. During the period 1997 and 2004, the national (English) hospital episodes statistics dataset comprise more than 5 million patient readmissions. Implementing a multilevel model using the complete population dataset could possibly take weeks to estimate the parameters. To resolve the problem, we extract 1000 random samples from the original data, where each random sample is likely to lead to differing hospital performance measures. For computational efficiency a Grid implementation of the model is developed. Analysing the output from the full 1000 sample, we noticed that 4 out of the 5 worst performing hospitals treating cancer patients were in London. These hospitals are known to be the leading NHS Trusts in England, providing diverse range of services to complex patients, and therefore it is inevitable to expect higher numbers of emergency readmissions.

What is the best way to estimate hospital quality outcomes? A simulation approach

OBJECTIVE

To test the accuracy of alternative estimators of hospital mortality quality using a Monte Carlo simulation experiment.

DATA SOURCES

Data are simulated to create an admission-level analytic dataset. The simulated data are validated by comparing distributional parameters (e.g., mean and standard deviation of 30-day mortality rate, hospital sample size) with the same parameters observed in Medicare data for acute myocardial infarction (AMI) inpatient admissions.

STUDY DESIGN

We perform a Monte Carlo simulation experiment in which true quality is known to test the accuracy of the Observed-over-Expected estimator, the Risk Standardized Mortality Rate (RSMR), the Dimick and Staiger (DS) estimator, the Hierarchical Poisson estimator, and the Moving Average estimator using hospital 30-day mortality for AMI as the outcome. Estimator accuracy is evaluated for all hospitals and for small, medium, and large hospitals.

DATA EXTRACTION METHODS

Data are simulated.

PRINCIPAL FINDINGS

Significant and substantial variation is observed in the accuracy of the tested outcome estimators. The DS estimator is the most accurate for all hospitals and for small hospitals using both accuracy criteria (root mean squared error and proportion of hospitals correctly classified into quintiles).

CONCLUSIONS

The mortality estimator currently in use by Medicare for public quality reporting, the RSMR, has been shown to be less accurate than the DS estimator, although the magnitude of the difference is not large. Pending testing and validation of our findings using current hospital data, CMS should reconsider the decision to publicly report mortality rates using the RSMR.

Provider-hospital "fit" and patient outcomes: evidence from Massachusetts cardiac surgeons, 2002-2004

OBJECTIVE

To examine whether the "fit" of a surgeon with hospital resources impacts cardiac surgery outcomes, separately from hospital or surgeon effects.

DATA SOURCES

Retrospective secondary data from the Massachusetts Department of Public Health's Data Analysis Center, on all 12,983 adult isolated coronary artery bypass surgical admissions in state-regulated hospitals from 2002 through 2004. Clinically audited chart data was collected using Society of Thoracic Surgeons National Cardiac Surgery Database tools and cross-referenced with administrative discharge data in the Division of Health Care Finance and Policy. Mortality was followed up through 2007 via the state vital statistics registry.

STUDY DESIGN

Analysis was at the patient level for those receiving isolated coronary artery bypass surgery (CABG). Sixteen outcomes included 30-day mortality, major morbidity, indicators of perioperative, and predischarge processes of care. Hierarchical crossed mixed models were used to estimate fixed covariate and random effects at hospital, surgeon, and hospital × surgeon level.

PRINCIPAL FINDINGS

Hospital volume was associated with significantly reduced intraoperative durations and significantly increased probability of aspirin, β-blocker, and lipid-lowering discharge medication use. The proportion of outcome variability due to unobserved hospital × surgeon interaction effects was small but meaningful for intraoperative practices, discharge destination, and medication use. For readmissions and mortality within 30 days or 1 year, unobserved patient and hospital factors drove almost all variability in outcomes.

CONCLUSIONS

Among Massachusetts patients receiving isolated CABG, consistent evidence was found that the hospital × surgeon combination independently impacted patient outcomes, beyond hospital or surgeon effects. Such distinct local interactions between a surgeon and hospital resources may play an important part in moderating quality improvement efforts, although residual patient-level factors generally contributed the most to outcome variability.

Accounting for regression-to-the-mean in tests for recent changes in institutional performance: analysis and power

Recent changes in individual units are often of interest when monitoring and assessing the performance of healthcare providers. We consider three high profile examples: (a) annual teenage pregnancy rates in English local authorities, (b) quarterly rates of the hospital-acquired infection Clostridium difficile in National Health Service (NHS) Trusts and (c) annual mortality rates following heart surgery in New York State hospitals. Increasingly, government targets call for continual improvements, in each individual provider as well as overall.Owing to the well-known statistical phenomenon of regression-to-the-mean, observed changes between just two measurements are potentially misleading. This problem has received much attention in other areas, but there is a need for guidelines within performance monitoring.In this paper we show theoretically and with worked examples that a simple random effects predictive distribution can be used to 'correct' for the potentially undesirable consequences of regression-to-the-mean on a test for individual change. We discuss connections to the literature in other fields, and build upon this, in particular by examining the effect of the correction on the power to detect genuine changes. It is demonstrated that a gain in average power can be expected, but that this gain is only very slight if the providers are very different from one another, for example due to poor risk adjustment. Further, the power of the corrected test depends on the provider's baseline rate and, although large gains can be expected for some providers, this is at the cost of some power to detect real changes in others.

Improving the reliability of physician performance assessment: identifying the "physician effect" on quality and creating composite measures

BACKGROUND

The proliferation of efforts to assess physician performance underscore the need to improve the reliability of physician-level quality measures.

OBJECTIVE

Using diabetes care as a model, to address 2 key issues in creating reliable physician-level quality performance scores: estimating the physician effect on quality and creating composite measures.

DESIGN

Retrospective longitudinal observational study.

SUBJECTS

A national sample of physicians (n = 210) their patients with diabetes (n = 7574) participating in the National Committee on Quality Assurance-American Diabetes Association's Diabetes Provider Recognition Program.

MEASURES

Using 11 diabetes process and intermediate outcome quality measures abstracted from the medical records of participants, we tested each measure for the magnitude of physician-level variation (the physician effect or "thumbprint"). We then combined measures with a substantial physician effect into a composite, physician-level diabetes quality score and tested its reliability.

RESULTS

We identified the lowest target values for each outcome measure for which there was a recognizable "physician thumbprint" (ie, intraclass correlation coefficient > or =0.30) to create a composite performance score. The internal consistency reliability (Cronbach's alpha) of the composite score, created by combining the process and outcome measures with an intraclass correlation coefficient > or =0.30, exceeded 0.80. The standard errors of the composite case-mix adjusted score were sufficiently small to discriminate those physicians scoring in the highest from those scoring in the lowest quartiles of the quality of care distribution with no overlap.

CONCLUSIONS

We conclude that the aggregation of well-tested quality measures that maximize the "physician effect" into a composite measure yields reliable physician-level quality of care scores for patients with diabetes.

A comparison of methods for combining quality and efficiency performance measures: profiling the value of hospital care following acute myocardial infarction

Health plans have begun to combine data on the quality and cost of medical providers in an attempt to identify and reward those that offer the greatest 'value.' The analytical methods used to combine these measures in the context of provider profiling have not been rigorously studied. We propose three methods to measure and compare the value of hospital care following acute myocardial infarction by combining a single measure of quality, in-hospital survival, and the cost of an episode of acute care. To illustrate these methods, we use administrative data for heart attack patients treated at 69 acute care hospitals in Massachusetts in fiscal year 2003. In the first method we reproduce a common approach to value profiling by modeling the two case mix-standardized outcomes independently. In the second approach, survival is regressed on patient risk factors and the average cost of care at each hospital. The third method models survival and cost for each hospital jointly and combines the outcomes on a common scale using a cost-effectiveness framework. For each method we use the resulting parameter estimates or functions of the estimates to compute posterior tail probabilities, representing the probability of being classified in the upper or lower quartile of the statewide distribution. Hospitals estimated to have the highest and lowest value according to each method are compared for consistency, and the advantages and disadvantages of each approach are discussed.

Flexible random-effects models using Bayesian semi-parametric models: applications to institutional comparisons

Random effects models are used in many applications in medical statistics, including meta-analysis, cluster randomized trials and comparisons of health care providers. This paper provides a tutorial on the practical implementation of a flexible random effects model based on methodology developed in Bayesian non-parametrics literature, and implemented in freely available software. The approach is applied to the problem of hospital comparisons using routine performance data, and among other benefits provides a diagnostic to detect clusters of providers with unusual results, thus avoiding problems caused by masking in traditional parametric approaches. By providing code for Winbugs we hope that the model can be used by applied statisticians working in a wide variety of applications.

Understanding Patterns of Change over Time to Improve Mammography Rates

This retrospective cohort study determined trends and patterns of mammography rates during 5 years (1997-2001) among female Medicare beneficiaries ages 50 years and older in Connecticut to better understand changes in rates over time and to plan future interventions. Time series analysis and hierarchical Longitudinal logistic regression were used to assess changes over time. Mammography rates increased significantly during the 5-year period (p < .001). A cyclical pattern was observed for all age groups and counties, with dips and peaks in the spring and fall each year (average increase 8% per year), consistent with concentrated intervention activity at those times.

Bayesian statistics in medicine: a 25 year review

This review examines the state of Bayesian thinking as Statistics in Medicine was launched in 1982, reflecting particularly on its applicability and uses in medical research. It then looks at each subsequent five-year epoch, with a focus on papers appearing in Statistics in Medicine, putting these in the context of major developments in Bayesian thinking and computation with reference to important books, landmark meetings and seminal papers. It charts the growth of Bayesian statistics as it is applied to medicine and makes predictions for the future. From sparse beginnings, where Bayesian statistics was barely mentioned, Bayesian statistics has now permeated all the major areas of medical statistics, including clinical trials, epidemiology, meta-analyses and evidence synthesis, spatial modelling, longitudinal modelling, survival modelling, molecular genetics and decision-making in respect of new technologies.

Longitudinal profiling of health care units based on continuous and discrete patient outcomes

Monitoring health care quality involves combining continuous and discrete outcomes measured on subjects across health care units over time. This article describes a Bayesian approach to jointly modeling multilevel multidimensional continuous and discrete outcomes with serial dependence. The overall goal is to characterize trajectories of traits of each unit. Underlying normal regression models for each outcome are used and dependence among different outcomes is induced through latent variables. Serial dependence is accommodated through modeling the pairwise correlations of the latent variables. Methods are illustrated to assess trends in quality of health care units using continuous and discrete outcomes from a sample of adult veterans discharged from 1 of 22 Veterans Integrated Service Networks with a psychiatric diagnosis between 1993 and 1998.

Is risk-adjustor selection more important than statistical approach for provider profiling? Asthma as an example

OBJECTIVES

To examine how the selections of different risk adjustors and statistical approaches affect the profiles of physician groups on patient satisfaction.

DATA SOURCES

Mailed patient surveys. Patients with asthma were selected randomly from each of 20 California physician groups between July 1998 and February 1999. A total of 2515 patients responded.

RESEARCH DESIGN

A cross-sectional study. Patient satisfaction with asthma care was the performance indicator for physician group profiling. Candidate variables for risk-adjustment model development included sociodemographic, clinical characteristics, and self-reported health status. Statistical strategies were the ratio of observed-to-expected rate (OE), fixed effects (FE), and the random effects (RE) approaches. Model performance was evaluated using indicators of discrimination (C-statistic) and calibration (Hosmer-Lemeshow chi2). Ranking impact of using different risk adjustors and statistical approaches was based on the changes in absolute ranking (AR) and quintile ranking (QR) of physician group performance and the weighted kappa for quintile ranking.

RESULTS

Variables that added significantly to the discriminative power of risk-adjustment models included sociodemographic (age, sex, prescription drug coverage), clinical (asthma severity), and health status (SF-36 PCS and MCS). Based on an acceptable goodness-of-fit (P > 0.1)and higher C-statistics, models adjusting for sociodemographic, clinical, and health status variables (Model S-C-H) using either the FE or RE approach were more favorable. However, the C-statistic (=0.68) was only fair for both models. The influence of risk-adjustor selection on change of performance ranking was more salient than choice of statistical strategy (AR: 50%-80% v. 20%-55%; QR: 10%-30% v. 0%-10%). Compared to the model adjusting for sociodemographic and clinical variables only and using OE approach, the Model S-C-H using RE approach resulted in 70% of groups changing in AR and 25% changing in QR (weighted kappa: 0.88). Compared to the Consumer Assessment of Health Plans model, the Model S-C-H using RE approach resulted in 65% of groups changing in AR and 20% changing in QR (weighted kappa: 0.88).

CONCLUSIONS

In comparing the performance of physician groups on patient satisfaction with asthma care, the use of sociodemographic, clinical, and health status variables maximized risk-adjustment model performance. Selection of risk adjustors had more influence on ranking profiles than choice of statistical strategies. Stakeholders employing provider profiling should pay careful attention to the selection of both variables and statistical approach used in risk-adjustment.

SAPS�II revisited

OBJECTIVE

To construct and validate an update of the Simplified Acute Physiology Score II (SAPS II) for the evaluation of clinical performance of Intensive Care Units (ICU).

DESIGN AND SETTING

Retrospective analysis of prospectively collected multicenter data in 32 ICUs located in the Paris area belonging to the Cub-Rea database and participating in a performance evaluation project.

PATIENTS

33,471 patients treated between 1999 and 2000.

MEASUREMENTS AND RESULTS

Two logistic regression models based on SAPS II were developed to estimate in-hospital mortality among ICU patients. The second model comprised reevaluation of original items of SAPS II and integration of the preadmission location and chronic comorbidity. Internal and external validation were performed. In the two validation samples the most complex model had better calibration than the original SAPS II for in-hospital mortality but its discrimination was not significantly higher (area under ROC curve 0.89 vs. 0.87 for SAPS II). Second-level customization and integration of new items improved uniformity of fit for various categories of patients except for diagnosis-related groups. The rank order of ICUs was modified according to the model used.

CONCLUSIONS

The overall performance of SAPS II derived models was good, even in the context of a community cohort and routinely gathered data. However, one-half the variation of outcome remains unexplained after controlling for admission characteristics, and uniformity of prediction across diagnostic subgroups was not achieved. Differences in case-mix still limit comparisons of quality of care.

The effect of provider-level ascertainment bias on profiling nursing homes

Profiling health care providers for the purpose of public reporting and quality improvement has become commonplace. Recently, the Centers for Medicare and Medicaid Services (CMS) began publishing measures of quality for every Medicare/Medicaid-certified nursing home in the country. The facility-specific quality indicators (QIs) reported by CMS are based on quarterly measures from the minimum data set (MDS). However, some QIs from the MDS are potentially subject to ascertainment bias. Ascertainment bias would occur if there was variation in the way items that make up QIs are measured by nurses from each facility. This is potentially a problem for difficult-to-measure items such as pain and pressure ulcers. To assess the impact of ascertainment bias on profiling, we utilize data from a reliability study of nursing homes from six states. We develop methods for profiling providers in situations where the data consist of a response variable for each subject based on assessments from an internal rater, and, for a subset of subjects in each facility, a response variable based on assessments from an independent (external) rater. The internal assessments are potentially subject to provider-level ascertainment bias, whereas the independent assessments are considered the 'gold standard'. Our methods extend popular Bayesian approaches for profiling by using the paired observations from the subset of subjects with error-prone and error-free assessments to adjust for ascertainment bias. We apply the methods to MDS merged with the reliability data, and compare the bias-corrected profiles with those of standard approaches.

Economic profiling of primary care physicians: consistency among risk-adjusted measures

OBJECTIVE

To investigate whether different risk-adjustment methodologies and economic profiling or "practice efficiency" metrics produce differences in practice efficiency rankings for a set of primary care physicians (PCPs).

DATA SOURCE

Twelve months of claims records (inpatient, outpatient, professional, and pharmacy) for an independent practice association HMO.

STUDY DESIGN

Patient risk scores obtained with six profiling risk-adjustment methodologies were used in conjunction with claims cost tabulations to measure practice efficiency of all primary care physicians who managed 25 or more members of an HMO.

DATA COLLECTION

For each of the risk-adjustment methodologies, two measures of "efficiency" were constructed: the standardized cost difference between total observed (standardized actual) and total expected costs for patients managed by each PCP, and the ratio of the PCP's total observed to total expected costs (O/E ratio). Primary care physicians were ranked from most to least efficient according to each risk-adjusted measure, and level of agreement among measures was tested using weighted kappa. Separate rankings were constructed for pediatricians and for other primary care physicians.

FINDINGS

Moderate to high levels of agreement were observed among the six risk-adjusted measures of practice efficiency. Agreement was greater among pediatrician rankings than among adult primary care physician rankings, and, with the standardized difference measure, greater for identifying the least efficient than the most efficient physicians. The O/E ratio was shown to be a biased measure of physician practice efficiency, disproportionately targeting smaller sized panels as outliers.

CONCLUSIONS

Although we observed moderate consistency among different risk-adjusted PCP rankings, consistency of measures does not prove that practice efficiency rankings are valid, and health plans should be careful in how they use practice efficiency information. Indicators of practice efficiency should be based on the standardized cost difference, which controls for number of patients in a panel, instead of O/E ratio, which does not.

Hospital-level performance improvement: beta-blocker use after acute myocardial infarction

BACKGROUND

National surveys indicate improvement in beta-blocker use after acute myocardial infarction (AMI) over time; however, these data could obscure important variation in improvement at individual hospitals. Our objective was to characterize the hospital-level variation in the improvements in beta-blocker prescription rates after AMI and to identify hospital characteristics that were associated with hospital improvement rates after adjustment for patient demographic and clinical characteristics.

METHODS AND RESULTS

We used data (n = 335,244 patients with AMI discharged from 682 hospitals) from the National Registry of Myocardial Infarction (NRMI) and from the American Hospital Association Annual Survey of Hospitals and hierarchical modeling to examine the associations between hospital characteristics and hospital-level rates of change in beta-blocker use during 1996-1999. On average, hospital rates of beta-blocker use for patients with AMI increased 5.9 percentage points (standard deviation, 9.7 percentage points) from the premidpoint time period (April 1996-February 1998) to the postmidpoint time period (March 1998-September 1999) of the study. The range in hospital-level changes in beta-blocker rates was substantial, from a decline of -50.0 percentage points to an increase of +35.7 percentage points. AMI volume and teaching status, geographic region, and initial beta-blocker use rates were associated with rate of improvement, but the magnitude of these effects was modest.

CONCLUSIONS

The study reveals marked hospital-level variation in improvement in beta-blocker use after AMI. Several hospital characteristics were associated with this improvement, but they are weak predictors of hospital-based improvement in the use of beta-blockers.

Public Health Surveillance of Low-Frequency Populations

Public health agencies often require data that address the needs of special populations, such as minority groups. Sources of surveillance data often contain insufficient numbers of subjects to fully inform health agencies. In this review, we address the problems of and potential approaches for situations with insufficient surveillance data. We use the examples of race and ethnic minority groups throughout our discussion. However, many of the broad issues are applicable to other special groups with low frequency or who are especially hard to reach. Our recommendations are based, in part, on a symposium held in Missouri with the collaboration of state health agency, community, and academic research participants. We review problems in using existing data and collecting new data, especially from nonprobability samples. We also describe fieldwork issues for reaching and collecting information from special populations. Decisions among methods and solutions may require seeking additional resources for surveillance.

The volume-outcome relationship: from Luft to Leapfrog

Numerous reports have documented a volume-outcome relationship for complex medical and surgical care, although many such studies are compromised by the use of discharge abstract data, inadequate risk adjustment, and problematic statistical methodology. Because of the volume-outcome association, and because valid outcome measurements are unavailable for many procedures, volume-based referral strategies have been advocated as an alternative approach to health-care quality improvement. This is most appropriate for procedures with the greatest outcome variability between low-volume and high-volume providers, such as esophagectomy and pancreatectomy, and for particularly high-risk subgroups of patients. Whenever possible, risk-adjusted outcome data should supplement or supplant volume standards, and continuous quality improvement programs should seek to emulate the processes of high-volume, high-quality providers. The Leapfrog Group has established a minimum volume requirement of 500 procedures for coronary artery bypass grafting. In view of the questionable basis for this recommendation, we suggest that it be reevaluated.