Statistical Methods for Profiling Providers of Medical Care: Issues and Applications

A Bayesian hierarchical approach to comparative audit for carotid surgery

OBJECTIVES

the aim of this study was to illustrate how a Bayesian hierarchical modelling approach can aid the reliable comparison of outcome rates between surgeons.

DESIGN

retrospective analysis of prospective and retrospective data.

MATERIALS

binary outcome data (death/stroke within 30 days), together with information on 15 possible risk factors specific for CEA were available on 836 CEAs performed by four vascular surgeons from 1992-99. The median patient age was 68 (range 38-86) years and 60% were men.

METHODS

the model was developed using the WinBUGS software. After adjusting for patient-level risk factors, a cross-validatory approach was adopted to identify "divergent" performance. A ranking exercise was also carried out.

RESULTS

the overall observed 30-day stroke/death rate was 3.9% (33/836). The model found diabetes, stroke and heart disease to be significant risk factors. There was no significant difference between the predicted and observed outcome rates for any surgeon (Bayesian p -value>0.05). Each surgeon had a median rank of 3 with associated 95% CI 1.0-5.0, despite the variability of observed stroke/death rate from 2.9-4.4%. After risk adjustment, there was very little residual between-surgeon variability in outcome rate.

CONCLUSIONS

Bayesian hierarchical models can help to accurately quantify the uncertainty associated with surgeons' performance and rank.

Profiling nursing homes using Bayesian hierarchical modeling

OBJECTIVES

New methods developed to improve the statistical basis of provider profiling may be particularly applicable to nursing homes. We examine the use of Bayesian hierarchical modeling in profiling nursing homes on their rate of pressure ulcer development.

DESIGN

Observational study using Minimum Data Set data from 1997 and 1998.

SETTING

A for-profit nursing home chain.

PARTICIPANTS

Residents of 108 nursing homes who were without a pressure ulcer on an index assessment.

MEASUREMENTS

Nursing homes were compared on their performance on risk-adjusted rates of pressure ulcer development calculated using standard statistical techniques and Bayesian hierarchical modeling.

RESULTS

Bayesian estimates of nursing home performance differed considerably from rates calculated using standard statistical techniques. The range of risk-adjusted rates among nursing homes was 0% to 14.3% using standard methods and 1.0% to 4.8% using Bayesian analysis. Fifteen nursing homes were designated as outliers based on their z scores, and two were outliers using Bayesian modeling. Only one nursing home had greater than a 50% probability of having a true rate of ulcer development exceeding 4%.

CONCLUSIONS

Bayesian hierarchical modeling can be successfully applied to the problem of profiling nursing homes. Results obtained from Bayesian modeling are different from those obtained using standard statistical techniques. The continued evaluation and application of this new methodology in nursing homes may ensure that consumers and providers have the most accurate information regarding performance.

Bayeswatch: an overview of Bayesian statistics

Increasingly, clinical research is evaluated on the quality of its statistical analysis. Traditionally, statistical analyses in clinical research have been carried out from a 'frequentist' perspective. The presence of an alternative paradigm - the Bayesian paradigm - has been relatively unknown in clinical research until recently. There is currently a growing interest in the use of Bayesian statistics in health care research. This is due both to a growing realization of the limitations of frequentist methods and to the ability of Bayesian methods explicitly to incorporate prior expert knowledge and belief into the analyses. This is in contrast to frequentist methods, where prior experience and beliefs tend to be incorporated into the analyses in an ad hoc fashion. This paper outlines the frequentist and Bayesian paradigms. Acute myocardial infarction mortality data are then analysed from both a Bayesian and a frequentist perspective. In some analyses, the two methods are seen to produce comparable results; in others, they produce different results. It is noted that in this example, there are clinically relevant questions that are more easily addressed from a Bayesian perspective. Finally, areas in clinical research where Bayesian ideas are increasingly common are highlighted.

Longitudinal profiles of health care providers

Provider profiling is the activity of collecting, comparing and reporting quality of care measures for individuals, groups, agencies and institutions that provide health care services. Univariate provider profiles, such as hospital-specific mortality rates, have been constructed using cross-sectional data based on posterior summaries or maximum likelihood estimates. As data continue to be collected over time, the construction and interpretation of longitudinal profiles of health care providers will become increasingly important. Longitudinal series can be used to improve the precision of estimates - a feature that is particularly important for providers who treat a small number of patients per year. We extend and apply hierarchical models to examine and classify provider performance over time using two examples, one in the area of cardiology and the other in mental health. Performance is evaluated using the squared Mahalanobis distance and posterior probabilities based on this distance. By comparing providers based on level and temporal trend simultaneously, conservative but comprehensive assessments of performance are possible. Furthermore, the longitudinal profiles developed are easily interpreted and flexible, making them of practical use to policy-makers.

A comparison of Bayesian methods for profiling hospital performance

There is a growing interest in the use of Bayesian methods for profiling institutional performance. In the literature, several studies have compared different frequentist methods for classifying hospitals as performance outliers. The purpose of this study was to compare 4 different Bayesian methods for classifying hospitals as outcomes outliers, using 30-day hospital-level mortality rates for a cohort of acute myocardial infarction patients as a test case. The 1st Bayesian method involved determining the probability that a hospital's mortality rare for an average patient exceeded a specified threshold. The 2nd method involved ranking hospitals according to their mortality rate for an average patient. The 3rd method involved determining the probability that a hospital's standardized mortality ratio exceeded a specified threshold. The 4th method involved ranking hospitals according to their standardized mortality ratio. In most of the scenarios examined, there was only marginal agreement between the different methods. In only 4 of 19 comparisons, was there good agreement between the different methods (0.40 < or = kappa < or = 0.75). Methods based on ranking institutions were relatively insensitive to differences between hospitals. These inconsistencies raise questions about the choice of methods for classifying hospital performance, and they suggest a need for urgent research into which methods are best able to discriminate between institutions and which are most meaningful to decision makers.

Sample size considerations in observational health care quality studies

A common objective in health care quality studies involves measuring and comparing the quality of care delivered to cohorts of patients by different health care providers. The data used for inference involve observations on units grouped within clusters, such as patients treated within hospitals. Unlike cluster randomization trials where often clusters are randomized to interventions to learn about individuals, the target of inference in health quality studies is the cluster. Furthermore, randomization is often not performed and the resulting biases may invalidate standard tests. In this paper, we discuss approaches to sample size determination in the design of observational health quality studies when the outcome is binary. Methods for calculating sample size using marginal models are briefly reviewed, but the focus is on hierarchical binomial models. Sample size in unbalanced clusters and stratified designs are characterized. We draw upon the experiences that have arisen from a study funded by the Agency for Healthcare Research and Quality involving assessment of quality of care for patients with cardiovascular disease. If researchers are interested in comparing clusters, hierarchical models are preferred.

Cardiac surgery report cards: comprehensive review and statistical critique

Public report cards and confidential, collaborative peer education represent distinctly different approaches to cardiac surgery quality assessment and improvement. This review discusses the controversies regarding their methodology and relative effectiveness. Report cards have been the more commonly used approach, typically as a result of state legislation. They are based on the presumption that publication of outcomes effectively motivates providers, and that market forces will reward higher quality. Numerous studies have challenged the validity of these hypotheses. Furthermore, although states with report cards have reported significant decreases in risk-adjusted mortality, it is unclear whether this improvement resulted from public disclosure or, rather, from the development of internal quality programs by hospitals. An additional confounding factor is the nationwide decline in heart surgery mortality, including states without quality monitoring. Finally, report cards may engender negative behaviors such as high-risk case avoidance and "gaming" of the reporting system, especially if individual surgeon results are published. The alternative approach, continuous quality improvement, may provide an opportunity to enhance performance and reduce interprovider variability while avoiding the unintended negative consequences of report cards. This collaborative method, which uses exchange visits between programs and determination of best practice, has been highly effective in northern New England and in the Veterans Affairs Administration. However, despite their potential advantages, quality programs based solely on confidential continuous quality improvement do not address the issue of public accountability. For this reason, some states may continue to mandate report cards. In such instances, it is imperative that appropriate statistical techniques and report formats are used, and that professional organizations simultaneously implement continuous quality improvement programs. The statistical methodology underlying current report cards is flawed, and does not justify the degree of accuracy presented to the public. All existing risk-adjustment methods have substantial inherent imprecision, and this is compounded when the results of such patient-level models are aggregated and used inappropriately to assess provider performance. Specific problems include sample size differences, clustering of observations, multiple comparisons, and failure to account for the random component of interprovider variability. We advocate the use of hierarchical or multilevel statistical models to address these concerns, as well as report formats that emphasize the statistical uncertainty of the results.

An introduction to multilevel regression models

Data in health research are frequently structured hierarchically. For example, data may consist of patients nested within physicians, who in turn may be nested in hospitals or geographic regions. Fitting regression models that ignore the hierarchical structure of the data can lead to false inferences being drawn from the data. Implementing a statistical analysis that takes into account the hierarchical structure of the data requires special methodologies. In this paper, we introduce the concept of hierarchically structured data, and present an introduction to hierarchical regression models. We then compare the performance of a traditional regression model with that of a hierarchical regression model on a dataset relating test utilization at the annual health exam with patient and physician characteristics. In comparing the resultant models, we see that false inferences can be drawn by ignoring the structure of the data.

Impact of patient socioeconomic status on physician profiles: a comparison of census-derived and individual measures

BACKGROUND

Patient education has been shown to affect physician performance profiles. It is not known whether census-derived measures of patient socioeconomic status (SES) show comparable effects.

OBJECTIVE

The objective of this study was to compare the effects on physician profiles for patient satisfaction and physical and mental health of adjustment for patient SES derived from patient addresses geocoded to the census block group level, zip codes, and patient education.

DESIGN

This was a cross-sectional survey of patients in physician practices.

SETTING

Subjects came from adult primary care practices in western New York.

PARTICIPANTS

A random sample of 100 primary care physicians and 50 consecutive patients seen by each physician participated in the study.

MEASUREMENTS

Independent variables were census-derived (block group and zip code) patient SES and patient-reported education. The outcomes were physician ranks for patient satisfaction (Patient Satisfaction Questionnaire) and physical and mental health status (SF-12). RESULTS. In empirical Bayes models that adjusted for patient age, age squared, gender, insurance, and case mix, both the census-derived measures (block group and zip code) of SES and education had similar effects on each of the physician profiles. CONCLUSIONS. The results suggest that SES derived from either patient addresses geocoded to the census block group level or zip codes may offer a convenient alternative to individually collected SES when adjusting physician profiles for the socioeconomic characteristics of physicians' practices. The relative ease of using zip codes compared with geocoded addresses and loss of information associated with incomplete matching during geocoding suggest that zip code-derived SES may be preferable.

Influence of patient education on profiles of physician practices

BACKGROUND

Few data are available about the effect of patient socioeconomic status on profiles of physician practices.

OBJECTIVE

To determine the ways in which adjustment for patients' level of education (as a measure of socioeconomic status) changes profiles of physician practices.

DESIGN

Cross-sectional survey of patients in physician practices.

SETTING

Managed care organization in western New York State.

PARTICIPANTS

A random sample of 100 primary care physicians and 50 consecutive patients seen by each physician.

MEASUREMENTS

Ranks of physicians for patient physical and mental health (Short Form 12-Item Health Survey) and satisfaction (Patient Satisfaction Questionnaire), adjusted for patient age, sex, morbidity, and education.

RESULTS

Physicians whose patients had a lower mean level of education had significantly better ranks for patient physical and mental health status after adjustment for patients' level of education level than they did before adjustment (P < 0.001); this result was not seen for patient satisfaction. After adjustment for patients' level of education, each 1-year decrease in mean educational level was associated with a rank that improved by 8.1 (95% CI, 6.6 to 9.6) for patient physical health status and by 4.9 (CI, 3.9 to 5.9) for patient mental health status. Adjustment for education had similar effects for practices with more educated patients and those with less educated patients.

CONCLUSIONS

Profiles of physician practices that base ratings of physician performance on patients' physical and mental health status are substantially affected by patients' level of education. However, these results do not suggest that physicians who care for less educated patients provide worse care. Physician profiling should account for differences in patients' level of education.

League tables and acute myocardial infarction

BACKGROUND

Comparison of the outcomes of care provided by hospitals is a growing trend. Outcomes need to be distinguished into those attributable to the practice of hospitals and those that arise from differences in the characteristics of patients and the underlying morbidity of the populations for whom hospitals provide care. We explored these issues for deaths in hospital or within 30 days of discharge after acute myocardial infarction in Scotland, UK.

METHODS

We used records from December, 1992, to November, 1993, for 14,359 episodes of acute myocardial infarction, the death records of those who died, and 9391 death records for individuals who died after acute myocardial infarction but who had not been in hospital in the 30 days before death. Hospital discharge records were taken from the Scottish Morbidity Records. The outcomes we investigated were all-cause mortality within 30 days of discharge from hospital, and death from acute myocardial infarction at any time during the study period. We estimated separately effects attributable to patients' characteristics, hospitals, and areas of residence with multilevel modelling.

FINDINGS

We found significant differences between hospitals by age, sex, and medical history. The odds ratios for death ranged from 0.62 (95% CI 0.50-0.80) to 1.28 (1.07-1.59), relative to the average performance for Scotland as a whole. Analysis including area of residence, deaths occurring out of hospital, and more detailed information about patients showed no significant differences between hospitals for patients aged 70 years. By postcode area, there was a strong association between out-of-hospital deaths and deaths in hospital or shortly after discharge.

INTERPRETATION

Hospital outcomes may vary from one subgroup of patients to another and should be assessed independently of patients' areas of residence. Measures of performance that do not provide valid comparisons could diminish public confidence in hospital services.