Performance reports on quality--prototypes, problems, and prospects

Health care report cards: implications for vulnerable patient groups and the organizations providing them care

Standardized public reporting on the quality of health care (report cards) offers an opportunity to empower purchasers and consumers so that they can make choices that can result in better health care for less money. However, not all population subgroups are equally well served by the publication of such data. In particular, vulnerable patient groups such as the poor, the less educated, the chronically sick, and members of ethnic or linguistic minorities may find issues of importance to them largely neglected. In addition, the way that report card data are collected, analyzed, and presented may further marginalize the experiences of these groups who in any case are already underserved by the health system. This observation also has important implications for health care providers who serve primarily large numbers of vulnerable patients. The differential impacts of report card data on vulnerable patient groups (and their providers) need to be addressed by researchers and policy makers if access issues are not to be damaged further by the providers' pursuit of quality and value.

Health care factors related to stage at diagnosis and survival among Medicare patients with colorectal cancer

BACKGROUND

With the growth in enrollment of Medicare patients in HMOs the effectiveness of care received by Medicare/HMO patients continues to be of concern. By considering the relationship of insurance to stage at diagnosis, this study inquires whether HMOs emphasize early diagnosis of colorectal cancer to a greater extent than FFS plans, if particular HMO types (group/nongroup models) are more successful in doing so, and how this pertains to survival.

METHODS

Data for 1329 Medicare patients with colorectal cancer, diagnosed 1987 to 1993, and residing in northern California, were acquired from a population-based cancer registry. Insurance included two types of Medicare HMOs (group and nongroup model) and three fee-for-service (FFS) categories: Medicare with private supplement, Medicare/Medicaid, and Medicare only. The relationships of insurance to AJCC stage at diagnosis and of insurance to survival following diagnosis were examined, respectively, with logistic regression models and survival analysis (controlling for age, ethnicity, tumor location, educational level, sex, and hospital type).

RESULTS

Likelihood of early stage colorectal cancer was greater for Medicare patients in nongroup model HMOs or having private FFS supplements than for those in group model HMOs, Medicare/Medicaid, or Medicare alone. All-cause and colorectal cancer mortality did not differ significantly among Medicare patients with group model HMO, nongroup model HMO and private FFS supplements. Medicare/Medicaid patients experienced significantly greater all-cause mortality than private FFS patients.

CONCLUSIONS

Differences within this study population in early stage diagnosis of colorectal cancer and breast cancer, respectively, by type of Medicare supplemental insurance may be attributable to which preventive screening measures are included in health plan report cards.

Is health plan employer data and information set performance associated with withdrawal from medicare managed care?

BACKGROUND

Withdrawals of health plans from Medicare have affected more than 1.6 million beneficiaries. Some plans claim that providing higher quality care raises costs, lowers profits, and spurs withdrawal because plans cannot sustain high quality care under current payment levels.

OBJECTIVE

To assess whether higher performance by Medicare health plans on quality indicators was associated with withdrawal.

DESIGN

Retrospective cohort study.

SUBJECTS

Taking each county where a contract was active as a unit of analysis, Medicare managed care plans active in 2310 contract-county combinations in 1997 were studied and followed for 3 years.

MEASURES

Independent variables were scores on six indicators from the Health Plan Employer Data and Information Set (HEDIS) for each contract, collapsed into two summary measures: clinical and ambulatory care access. Separate Cox proportional hazards regressions were used for each indicator, and each summary measure, to assess the association of HEDIS performance with our outcome measure, time-to-withdrawal from Medicare. Multiple potential confounders were adjusted for.

RESULTS

Of 2310 managed care contract-county combinations, 877 (38%) withdrew. The proportion of contract-counties with high scores on the summary clinical quality measure that withdrew was one-fifth that for low scorers (4.2% vs. 20.5%). For summary ambulatory care access performance, the corresponding ratio was two-fifths (12.8% vs. 32.0%). Lower payments were associated with higher withdrawal risk, but also higher clinical and ambulatory care access quality performance. In separate multivariable analyses controlling for confounders, both high clinical performance (HR, 0.18; 95% CI, 0.08-0.42) and high ambulatory care access performance (HR, 0.53; 95% CI, 0.27-1.07) were independently associated with lower withdrawal risk.

CONCLUSIONS

Health plans continuing to provide care to Medicare beneficiaries have higher average performance on HEDIS clinical and ambulatory care access measures than plans that withdrew.

Inconsistent report cards: assessing the comparability of various measures of the quality of ambulatory care

BACKGROUND

Report cards based on various performance measures have become increasingly common for rating hospitals and health care plans. However, little has been done to create report cards at the ambulatory clinic level, nor has there been much comparison of the potential components of report cards.

OBJECTIVES

To create a report card for ambulatory clinics based on different data collection methods and to assess the correlations of clinic scores across various domains of quality.

RESEARCH DESIGN

Cross-sectional chart review (n = 3614), patient (n = 2180), and physician surveys (n = 169).

SUBJECTS

Sample of outpatients ages 20 to 75 and their primary care providers in 11 ambulatory clinic sites in the Boston-area from May 1996 to June 1997.

MEASURES

Performance on various quality indicators for each site.

RESULTS

Report card scores for five quality domains (performance on HEDIS-like measures, clinic function, patient satisfaction, diabetes guideline compliance, asthma guideline compliance) were created for each site. None of the five domain scores were significantly correlated with any of the other domains. In addition, there was substantial intraclinic variation in domain scores when compared with the corresponding mean domain score across all clinics. Additional clinic domain scores were created by limiting measures to those found on chart review or survey alone. The chart review and survey domain scores for each clinic were also not significantly correlated.

CONCLUSIONS

Report cards that emphasize only one domain of quality or use limited data collection methods may provide incomplete or inconsistent information to health care consumers about the overall quality of an outpatient clinic.

Measuring Physicians’ Performances and Marketing the Results: A Breast Cancer Model

The recent transition to managed care has intensified the public's concerns about the quality of medical care. In response, payers, who until recently seemed preoccupied with costs, are demonstrating a renewed interest in value, which in turn could lead to an expansion of negotiations with physicians to include the tracking of such issues as physician performance, patient satisfaction, and patient outcomes. As a response to public concern and demand for accountability, the medical establishment must develop methods to assist payers in estimating relative value of competing medical services. In anticipation of a values transition in their specialty, breast care, the authors established a performance-oriented database which facilitated assessments of their performance in relation to community standards, and enhanced efforts to identify and correct performance deficiencies. Year-end results were summarized in a report-card format that improved marketability. The author's experiences should be of interest to physicians who are attempting to respond to changes in the rapidly evolving medical marketplace.

Patient protection and risk selection: do primary care physicians encourage their patients to join or avoid capitated health plans according to the patient's health status?

BACKGROUND

Individual physicians who are paid prospectively, as in capitated health plans, might tend to encourage patients to avoid or to join these plans according to the patient's health status. Though insurance risk selection has been well documented among organizations paid on a prospective basis, such physician-level risk selection has not been studied.

OBJECTIVE

To assess physician reports of risk selection in capitated health plans and explore potentially related factors.

DESIGN AND PARTICIPANTS

National mailed survey of primary care physicians in 1997-1998, oversampling physicians in areas with more capitated health plans.

RESULTS

The response rate was 63% (787 of 1,252 eligible recipients). Overall, 44% of physicians reported encouraging patients either to join or to avoid capitated health plans according to the patients' health status: 40% encouraged more complex and ill patients to avoid capitated plans and 23% encouraged healthier patients to join capitated plans. In multivariable models, physicians with negative perceptions of capitated plan quality, with more negative experiences in capitated plans, and those who knew at each patient encounter how they were being compensated had higher odds of encouraging sicker patients to avoid capitated plans (odds ratios, 2.0, 2.2, and 2.0; all confidence intervals >1).

CONCLUSIONS

Many primary care physicians report encouraging patients to join or avoid capitated plans according to the patient's health status. Although these physicians' recommendations might be associated primarily with concerns about quality, they can have the effect of insulating certain health plans from covering sicker and more expensive patients.

Overview of Evidence-based Medicine: Challenges for Evidence-based Laboratory Medicine

Evidence-based medicine (EBM) has been driven by the need to cope with information overload, by cost-control, and by a public impatient for the best in diagnostics and treatment. Clinical guidelines, care maps, and outcome measures are quality improvement tools for the appropriateness, efficiency, and effectiveness of health services. Although they are imperfect, their value increases with the quality of the evidence they incorporate. Laboratory professionals must direct more effort to demonstrating the impact of laboratory tests on a greater variety of clinical outcomes. Laboratory and clinical practitioners must be familiar with many of the accessible electronic and paper tools for searching for evidence. Detailed statistical and epidemiologic knowledge is not essential, but critical appraisal skills and a competent understanding of the strengths and weaknesses of systematic review and metaanalysis are necessary. Overemphasis on complexity and failure to recognize time limitations are major barriers to translating EBM into everyday practice. Emphasizing and practicing the role of the laboratory professional as a skilled clinical consultant strongly grounded in evidence as well, in addition to better integration of laboratory and clinical information and improved laboratory reports will overcome most barriers. There is a poverty of good, primary studies of test evaluations. Institution of more consistent standards for the design and reporting of studies on diagnostic accuracy should improve the situation. If nothing else, systematic reviews have demonstrated the need for more good-quality primary research in laboratory medicine.

Risk adjustment for measuring health outcomes: an application in VA long-term care

An empirically derived risk adjustment model is useful in distinguishing among facilities in their quality of care. We used Veterans Affairs (VA) administrative databases to develop and validate a risk adjustment model to predict decline in functional status, an important outcome measure in long-term care, among patients residing in VA long-term care facilities. This model was used to compare facilities on adjusted and unadjusted rates of decline. Predictors of decline included age, time between assessments, baseline functional status, terminal illness, pressure ulcers, pulmonary disease, cancer, arthritis, congestive heart failure, substance-related disorders, and various neurologic disorders. The model performed well in the development and validation databases (c statistics, 0.70 and 0.68, respectively). Risk-adjusted rates and rankings of facilities differed from unadjusted ratings. We conclude that judgments of facility performance depend on whether risk-adjusted or unadjusted decline rates are used. Valid risk adjustment models are therefore necessary when comparing facilities on outcomes.

Publicly disclosed information about the quality of health care: response of the US public

Public disclosure of information about the quality of health plans, hospitals, and doctors continues to be controversial. The US experience of the past decade suggests that sophisticated quality measures and reporting systems that disclose information on quality have improved the process and outcomes of care in limited ways in some settings, but these efforts have not led to the "consumer choice" market envisaged. Important reasons for this failure include limited salience of objective measures to consumers, the complexity of the task of interpretation, and insufficient use of quality results by organised purchasers and insurers to inform contracting and pricing decisions. Nevertheless, public disclosure may motivate quality managers and providers to undertake changes that improve the delivery of care. Efforts to measure and report information about quality should remain public, but may be most effective if they are targeted to the needs of institutional and individual providers of care.

Comparison of clinical indicators for performance measurement of health care quality: a cautionary note

The use of clinical performance data is increasing rapidly. Yet, substantial variation exists across indicators designed to measure the same clinical event. We compared indicators from several indicator measurement systems to determine the consistency of results. Five measurement systems with well-defined indicators were selected. They were applied to 24 hospitals. Indicators for mortality from coronary artery bypass graft surgery and mortality in the perioperative period were chosen from these measurement systems. Analyses results and concludes that it is faulty to assume that clinical indicators derived from different measurement systems will give the same rank order. Widespread demand for external release of outcome data from hospitals must be balanced by an educational effort about the factors that influence and potentially confound reported rates.

Access, satisfaction, and utilization in two forms of Medicaid managed care

We determined access and satisfaction of 2,598 recipients of Virginia's Medicaid program, comparing its health maintenance organizations (HMOs) to its primary care case management (PCCM) program. Positive responses were summed as sub-domains either of access, satisfaction, or of utilization, and adjusted odds ratios were calculated for HMO (vs. PCCM) sub-domain scores. The response rate was 47 per cent. We found few significant differences in perceived access, satisfaction, and utilization. Both HMO adults and children more often perceived good geographic access (adults, OR, [CI] = 1.50, [1.04-2.16]; children, OR, [CI] = 1.773 [1.158, 2.716]). But HMO patients less often reported good after-hours access (adults, OR, [CI] = 0.527 [0.335, 0.830]; children, OR, [CI] = 0.583 [0.380, 0.894]). Among all patients reporting poorer function, HMO patients more often reported good general and preventive care (OR, [CI] = 2.735 [1.138, 6.575]). We found some differences between Medicaid HMO versus PCCM recipients' reported access, satisfaction, and utilization, but were unable to validate concerns about access and quality under more restrictive forms of Medicaid managed care.

The Oregon Health Plan: predictors of office-based diabetic quality of care

OBJECTIVE

In 1994, the Oregon Health Plan (OHP) expanded basic Medicaid insurance to residents under the federal poverty limit, adopted a prioritized limited benefits package, and converted to managed care. The quality of care in predominantly Medicaid populations with diabetes has not been previously described. In OHP enrollees, we examined predictors of diabetes care based on American Diabetes Association guidelines and described OHP diabetes care compared with national benchmarks.

RESEARCH DESIGN AND METHODS

Chart abstraction and Medicaid data for 1995-1996 yielded 996 nonpregnant diabetic patients who were 18-64 years of age. Using HbA1c, lipid panel, and urine protein/microalbumin documentation ordered during the study year, we constructed a standard care (SC) index: SC for all three tests, mixed care (MC) for one to two tests, or no tests documented (NTD).

RESULTS

Our sample was predominantly white, 48 +/- 11 years of age, 63% women, with 8 +/- 5 provider visits. Providers ordered HbA1c (70%), urine microalbumin/protein (57%), and lipid panel (41%) tests. Patients distributed into SC (22%), MC (62%), or NTD (16%). Thirteen variables predicted SC. Patients had a higher likelihood of SC if they were 18-24 years of age, had more clinic visits, were on insulin daily, were in several comorbid groups, were enrolled in salaried or capitated health plans, or lived in counties with more hospital beds. Four studies were used as comparable national benchmarks.

CONCLUSIONS

Care provided to OHP patients with diabetes compares favorably with national benchmarks. Yet, most OHP patients with diabetes are still not achieving optimal care. Examining predictors of SC may play an important role in further policy development.

The brave new world of medical standards of care

There have always been medical standards of care in the American health-care sector. However, never before have they been so deeply incorporated in the delivery of health care as they are today. With the increased delivery of care through integrated delivery systems, as well as the development of the computerized patient record, medical standards of care are now used in innovative ways by providers and health plans in delivering health care to individual patients. There is great potential for even more innovative uses of medical standards of care in the future.

This article first presents a taxonomy of the medical standards of care that are involved in health-care delivery today Next, the article traces the historical evolution of medical standards of care since the early 1980s. Included in this discussion are the origins of the standard-setting movement as well as the developments that led to the way standards of care are currently used by large institutional providers and managed care plans to improve the quality of their health-care services. The article concludes with a brief analysis of the key legal issues that affect how standards of care can be used to improve the health care of patients.

Impact of sociodemographic case mix on the HEDIS measures of health plan quality

BACKGROUND

The widely used Health Plan Employer Data and Information Set (HEDIS) measures may be affected by differences among plans in sociodemographic characteristics of members.

OBJECTIVE

The objective of this study was to estimate effects of geographically linked patient sociodemographic characteristics on differential performance within and among plans on HEDIS measures.

RESEARCH DESIGN

Using logistic regression, we modeled associations between age, sex, and residential area characteristics of health plan members and results on HEDIS measures. We then calculated the impact of adjusting for these associations on plan-level measures.

SUBJECTS

This study included 92,232 commercially insured members with individual-level HEDIS data and an additional 20,615 members whose geographic distribution was provided.

MEASURES

This study used 7 measures of screening and preventive services.

RESULTS

Performance was negatively associated with percent receiving public assistance in the local area (6 of 7 measures), percent black (5 measures), and percent Hispanic (2 measures) and positively associated with percent college educated (6 measures), percent urban (2 measures), and percent Asian (1 measure) after controlling for plan and product type. These effects were generally consistent across plans. When measures were adjusted for these characteristics, rates for most plans changed by less than 5 percentage points. The largest change in the difference between plans ranged from 1.5% for retinal exams for people with diabetes to 20.2% for immunization of adolescents.

CONCLUSIONS

Performance on quality indicators for individual members is associated with sociodemographic context. Adjustment has little impact on the measured performance of most plans but a substantial impact on a few. Further study with more plans is required to determine the appropriateness and feasibility of adjustment.

Quality in health care and ethical principles

Quality in health care and ethical principles The last three decades have seen rapid changes in the way United States of America (USA) health care has been delivered, financed and regulated. Four major stakeholders have emerged in the health care debate: patients, providers, payers and public regulatory agencies. These groups do not agree on a definition of quality health care. This paper suggests five ethical principles - autonomy, justice, beneficence, non-maleficence, and prudence - be included in the framework of quality health care. A framework that outlines possible relationships among these ethical attributes and four major stakeholders is presented.

Does clinical evidence support ICD-9-CM diagnosis coding of complications?

BACKGROUND

Hospital discharge diagnoses, coded by use of the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM), increasingly determine reimbursement and support quality monitoring. Prior studies of coding validity have investigated whether coding guidelines were met, not whether the clinical condition was actually present.

OBJECTIVE

To determine whether clinical evidence in medical records confirms selected ICD-9-CM discharge diagnoses coded by hospitals.

RESEARCH DESIGN AND SUBJECTS

Retrospective record review of 485 randomly sampled 1994 hospitalizations of elderly Medicare beneficiaries in Califomia and Connecticut.

MAIN OUTCOME MEASURE

Proportion of patients with specified ICD-9-CM codes representing potential complications who had clinical evidence confirming the coded condition.

RESULTS

Clinical evidence supported most postoperative acute myocardial infarction diagnoses, but fewer than 60% of other diagnoses had confirmatory clinical evidence by explicit clinical criteria; 30% of medical and 19% of surgical patients lacked objective confirmatory evidence in the medical record. Across 11 surgical and 2 medical complications, objective clinical criteria or physicians' notes supported the coded diagnosis in >90% of patients for 2 complications, 80% to 90% of patients for 4 complications, 70% to <80% of patients for 5 complications, and <70% for 2 complications. For some complications (postoperative pneumonia, aspiration pneumonia, and hemorrhage or hematoma), a large fraction of patients had only a physician's note reporting the complication.

CONCLUSIONS

Our findings raise questions about whether the clinical conditions represented by ICD-9-CM codes used by the Complications Screening Program were in fact always present. These findings highlight concerns about the clinical validity of using ICD-9-CM codes for quality monitoring.

Results of report cards for patients with congestive heart failure depend on the method used to adjust for severity

BACKGROUND

The validity of outcome report cards may depend on the ways in which they are adjusted for risk.

OBJECTIVES

To compare the predictive ability of generic and disease-specific survival prediction models appropriate for use in patients with heart failure, to simulate outcome report cards by comparing survival across hospitals and adjusting for severity of illness using these models, and to assess the ways in which the results of these comparisons depend on the adjustment method.

DESIGN

Analysis of data from a prospective cohort study.

SETTING

A university hospital, a Veterans Affairs (VA) medical center, and a community hospital.

PATIENTS

Sequential patients presenting in the emergency department with acute congestive heart failure.

MEASUREMENTS

Unadjusted 30-day and 1-year mortality across hospitals and 30-day and 1-year mortality adjusted by using disease-specific survival prediction models (two sickness-at-admission models, the Cleveland Health Quality Choice model, the Congestive Heart Failure Mortality Time-Independent Predictive Instrument) and generic models (Acute Physiology and Chronic Health Evaluation [APACHE] II, APACHE III, the mortality prediction model, and the Chadson comorbidity index).

RESULTS

The community hospital's unadjusted 30-day survival rate (85.0%) and the VA medical center's unadjusted 1-year survival rate (60.9%) were significantly lower than corresponding rates at the university hospital (92.7% and 67.5%, respectively). No severity model had excellent ability to discriminate patients by survival rates (all areas under the receiver-operating characteristic curve < 0.73). Whether the VA medical center, the community hospital, both, or neither had worse survival rates on simulated report cards than the university hospital depended on the prediction model used for adjustment.

CONCLUSIONS

Results of simulated outcome report cards for survival in patients with congestive heart failure depend on the method used to adjust for severity.

Clinical status. Charting for outcomes in behavioral health

This article summarizes the criteria that clinical outcome data must meet to be useful in the quest for empirical effectiveness data about mental health services. Although demand is high for such data, its potential usefulness is just beginning to be tapped. The mental health field presents unique challenges for implementing outcome tracking systems, for analyzing and reporting results, and for using results to improve the processes of care. Measuring outcomes will not automatically improve care or ensure quality. It may, however, provide information that will be useful in guiding efforts to improve the quality of mental health services.

Antagonism and accommodation: interpreting the relationship between public health and medicine in the United States during the 20th century

Throughout the course of the 20th century, many observers have noted important tensions and antipathies between public health and medicine. At the same time, reformers have often called for better engagement and collaboration between the 2 fields. This article examines the history of the relationship between medicine and public health to examine how they developed as separate and often conflicting professions. The historical character of this relationship can be understood only in the context of institutional developments in professional education, the rise of the biomedical model of disease, and the epidemiologic transition from infectious disease to the predominance of systemic chronic diseases. Many problems in the contemporary burden of disease pose opportunities for effective collaborations between population-based and clinical interventions. A stronger alliance between public health and medicine through accommodation to a reductionist biomedicine, however, threatens to subvert public health's historical commitment to understanding and addressing the social roots of disease.

What's happening in quality improvement at the local hospital: a state-wide study from the Cooperative Cardiovascular Project

The objective of this study was to investigate what happened to improve the quality of care for acute myocardial infarction (AMI) at all 32 nonfederal hospitals in Connecticut and to assess the impact of the Cooperative Cardiovascular Project (CCP) on quality improvement (QI) activities for AMI. We performed a questionnaire study with secondary analyses using the CCP database. On-site interviews were conducted with QI directors at all 32 Connecticut nonfederal hospitals that participated in the Health Care Financing Administration's Cooperative Cardiovascular Project (CCP) in 1992-93 and 1995. The interviews sought information about the makeup of QI departments, specific approaches used to improve the care of patients with AMI, and the perceived value of the CCP to each individual hospital. Results showed that the number of full-time equivalents (FTEs) and FTEs per beds employed in QI departments ranged from 1 to 30 and from 0.4 to 7.9, respectively, with a registered nurse most often serving as the department head (27/32). Over half of the departments (17/32) had additional responsibilities. The majority (25/32) used some combination of physician champions, multidisciplinary QI teams, standing orders, or critical pathways to effect change in AMI care. Finally, 26 of the 32 hospitals believed the CCP was valuable because it provided credible benchmark data, a catalyst for change, or a specific focus on processes of care needing improvement in AMI. Despite great variability in institutional resources, all 32 hospitals used a similar combination of QI approaches to effect change in AMI care. However, there is variable scientific evidence supporting these approaches. Externally sponsored projects such as the CCP appear to play a useful role for individual hospitals. Defining the optimal methods of QI is difficult given that hospitals are using complex combinations of nonstandardized improvement interventions.

Consumer responses to health plan report cards in two markets

BACKGROUND

Health plans can compete on quality when consumers have helpful information. Report cards strive to meet this need, but consumer responses have not been measured.

OBJECTIVES

The objectives of this study were (1) to compare consumer responses to report cards in 2 markets, (2) to determine how personal characteristics relate to exposure, and (3) to assess the perceived helpfulness of the report cards.

RESEARCH DESIGN

A postenrollment survey was used.

SUBJECTS

The study included 784 employees of Monsanto (St Louis, 1996) and 670 employees of a health care purchasing cooperative (Denver, 1997).

DEPENDENT MEASURES

The dependent measures were (1) exposure, specifically remembering the report card, and intensity of reading it and (2) perceived helpfulness in learning about plan quality and in deciding to stay or switch.

RESULTS

Except for remembering seeing the report card (Denver, 47%; St Louis, 55%), the 2 groups did not differ. Forty percent read most or all of the report card; 82% found the report helpful in learning about quality; and 66% found it helpful in deciding to stay or switch. Employees who used patient survey information in their plan decision were more likely to remember seeing the report card (odds ratio [OR], 4.85), to read it intensely (OR, 2.84), and to find it helpful in learning about plan quality (OR, 3.04) and deciding whether to stay or switch plans (OR, 2.64).

CONCLUSIONS

Although the 2 samples differed markedly, their responses to report cards were similar. Exposure and helpfulness were related more to employee preferences for the type of information than to their health care decision needs.

Initial two years of experience with the AAO National Eyecare Outcomes Network (NEON) cataract surgery database

PURPOSE

To report the results of the first 2 years of experience with an American Academy of Ophthalmology (AAO) cataract surgery registry; to compare patient characteristics, operative procedures, and patient outcomes observed in the registry to those observed in the Cataract PORT study; and to discuss the current shortcomings and potential benefits of a national cataract surgery registry.

DESIGN

Observational study of episodes of cataract surgery reported by a self-selected sample of 249 ophthalmologists.

PARTICIPANTS

Seven thousand six hundred twenty-six patients undergoing cataract surgery during 1996 and 1997.

METHODS

Beginning in January 1996, participation in the AAO cataract surgery registry was offered to all ophthalmologists. Participants could use software or paper data collection forms to submit a common set of data regarding patients' demographics, preoperative ophthalmologic history, physical examination and test results, functional status and symptoms, intraoperative procedures and events, and postoperative outcomes for all patients undergoing first or second eye cataract surgery. Data were transmitted to a central database, where they were aggregated and analyzed. Findings were compared with those observed in the Cataract PORT study, which was conducted in 1991 and 1992.

RESULTS

Between January 1, 1996, and February 28, 1998, 249 ophthalmologists submitted data on at least one patient who underwent cataract surgery. A total of 7626 patients undergoing first or second eye surgery were enrolled, with all preoperative, intraoperative, and postoperative data forms submitted for 3342 patients (44%). The preoperative characteristics of patients reported to National Eyecare Outcomes Network (NEON) were similar to those of patients enrolled in the Cataract PORT study except for a higher reported prevalence of ocular comorbidity in NEON patients. Use of retrobulbar anesthesia was reported far less commonly, and use of topical anesthesia, phacoemulsification, and foldable intraocular lenses was reported far more often for NEON than for Cataract PORT study patients. Patient outcomes reported to NEON were similar to those observed in the Cataract PORT study.

MAIN OUTCOME MEASURES

Visual acuity, VF-14, Cataract Symptom Score, surgical complications.

CONCLUSIONS

During the first 2 years of NEON, ophthalmologist participation in the NEON cataract surgery database was low and consisted of a self-selected and likely nonrepresentative sample of ophthalmologists. The representativeness of patients for whom data were reported is unknown. In addition, complete data were submitted on only a minority of patients who were enrolled. Even so, the preoperative characteristics of patients on whom data were submitted to NEON were similar to those of patients enrolled in the Cataract PORT study. The initial experience with NEON demonstrates that it is technically possible to collect clinical data from, and report aggregated results to, practicing clinicians' offices. In addition, at least some practicing clinicians are willing to spend the time required to participate in the NEON registry. The NEON cataract surgery database thus has the potential to provide a practical means for tracking practice patterns and patient outcomes in real time. If a representative sample of physicians was willing to contribute data systematically and accurately over time, initiatives such as NEON could provide a means for professional societies and physicians to play a leadership role in defining and monitoring quality of care.

Implementation of the ANA report card

A major challenge in healthcare today is measuring the quality of care. To explore nursing's contribution to patients in acute care settings, the American Nurses Association commissioned the development of the "Nursing Report Card." This study explored whether these report card indicators capture quality care. The convenience sample comprised 1,500 patients and 300 nurses from 16 units at an academic medical center. Using regression analysis, the most consistent predictor of outcome indicators was the percentage of RNs of the total staff.

Barriers to compliance with screening guidelines for diabetic retinopathy

OBJECTIVE

To identify barriers to compliance with guidelines for diabetic retinopathy screening.

METHODS

The population studied included 4410 adults, aged 31 to 64, enrolled in an Independent Practice Association (IPA) plan in Upstate New York, who were diagnosed with diabetes, and their Primary Care Physicians (408 PCPs). Claims data were used to calculate variables characterizing patients and their PCPs. Logistic regression models were estimated to identify factors associated with higher probability of screening.

RESULTS

34% of patients were screened in 1993. The probability of screening was significantly higher for older patients, for women, for patients who visit their PCPs more often and for those living in areas of higher average education and lower percentage of blacks. However, only 16% of diabetic patients received an annual screen in two consecutive years (1992 and 1993). The probability of consecutive annual screening was significantly associated only with gender and patient expenditures per month.

CONCLUSION

The very low rate of diabetic retinopathy screening has implications for quality of life of patients with diabetes, long term costs of caring for them and social costs due to lost productivity. Interventions to increase screening rates are needed and should target both patients and their Primary Care Physicians.

Implementing outcome systems: lessons from a test of the BASIS-32 and the SF-36

With increasing pressure from third-party payers to assess client outcomes, clinical programs want to know how to implement outcome systems. This article focuses on practical and logistic questions involved in implementing an outcome assessment system in ambulatory behavioral healthcare settings. Study questions addressed outcome systems in general and the use of the Behavior and Symptom Identification Scale (BASIS-32) and the Short Form Health Status Profile (SF-36) in particular. General questions focused on obtaining provider buy-in, client consent and confidentiality, data collection methods, sampling, time points, maximizing client participation, clinical utility of outcome data, and resources needed for outcome assessment. Measure-specific questions focused on client acceptability of the instruments and applicability of measures to diverse populations. The article suggests several strategies for enhancing outcome assessment efforts and concludes that there remains a need for further understanding of ways to maximize the utility and value of outcome measurement.

Applying Bayesian ideas to the development of medical guidelines

Measurements of the quality of health care, in particular the underuse and overuse of medical therapies and diagnostic tests, often involve employment of medical practice guidelines to assess the appropriateness of treatments. This paper presents a case study of a Bayesian analysis for the development of medical guidelines based on expert opinion, using ordinal categorical rater data. We develop guidelines for the use of coronary angiography following an acute myocardial infarction (AMI) for 890 clinical indications using statistical models fit to appropriateness ratings obtained from a nine-member expert panel. The main foci of our analyses were on the estimation of an appropriateness score for each of the clinical indications, an associated measure of precision, and functions of the underlying score. We considered two classes of models that assume the ratings are either in the form of grouped normal data or are ungrouped variables arising from a normal distribution, while permitting rater effects and indication heterogeneity in both. We estimated models using Markov chain Monte Carlo methods and constructed indices quantifying appropriateness based on posterior probabilities of selected model parameters. We compared our model-based approach to the standard approach currently employed in medical guideline development and found that the standard approach correctly identified 99 per cent of the appropriate indications while overestimating appropriateness 18 per cent of the time compared to our model-based approach.

Health status in VA patients: results from the Veterans Health Study

Recently, the Veterans Administration (VA) Under Secretary for Health has designated functional status as one of the domains of value for the system, given its increasing importance for clinical care. The Veterans Health Study (VHS) was designed to assist the VA in monitoring outcomes and measuring the case mix of patients who use the VA. The Veterans SF-36 (short form functional status assessment for veterans) was administered to 2425 veterans receiving ambulatory care. Measures of the Veterans SF-36 were strongly correlated with sociodemographics and morbidities of the veterans. Young veterans had poorer mental health status than older veterans. Veterans who used ambulatory care in the VHS reported lower levels of health status, reflecting more disease than a non-VA civilian population. These measures of health are important indicators of the disease burden or case mix of the patients and are pertinent to health systems such as the VA for resource allocation decisions and as outcomes of care.

Purchasing population health: aligning financial incentives to improve health outcomes

OBJECTIVE

To review the concept of population health, including its definition, measurement, and determinants, and to suggest an approach for aligning financial incentives toward this goal. DATA SOURCE, STUDY DESIGN, DATA EXTRACTION: Literature review, policy analysis

PRINCIPAL FINDINGS

The article presents the argument that a major reason for our slow progress toward health outcome improvement is that there is no operational definition of population health and that financial incentives are not aligned to this goal. Current attempts at process measures as indicators of quality or outcome are not adequate for the task. It is suggested that some measure of health-adjusted life expectancy be adopted for this purpose, and that integrated delivery systems and other agents responsible for nonmedical determinants be rewarded for improvement in this measure. This will require the development of an investment portfolio across the determinants of health based on relative marginal return to health, with horizontal integration strategies across sectoral boundaries. A 20-year three-phase development strategy is proposed, including components of research and acceptance, integrated health system implementation, and cross-sectoral integration.

CONCLUSIONS

The U.S. health care system is a $1 trillion industry without a definition of its product. Until population outcome measures are developed and rewarded for, we will not solve the twenty-first century challenge of maximizing health outcome improvement for the resources available.

Performance management using health outcomes: in search of instrumentality

The past decade has seen the growing use of performance data in the hope of bringing about quality improvements in health care. Most recently, an emphasis on health outcomes (and especially mortality rates) has led to much activity around collecting and publishing such data. Two major problems intervene. What meanings can be ascribed to reported health outcomes? And what impacts are they likely to have on clinical performance? Much empirical work supports the assertion that reported outcomes may be poor indicators of service quality. In addition, the impact of these data may be small or even detrimental unless great care is made to connect the reporting with explicit quality-improving actions.

Trust in performance indicators?

The 1980s and 90s have seen the proliferation of all forms of performance indicators as part of attempts to command and control health services. The latest area to receive attention is health outcomes. Published league tables of mortality and other health outcomes have been available in the United States for some time and in Scotland since the early 1990s; they have now been developed for England and Wales. Publication of these data has proceeded despite warnings as to their limited meaningfulness and usefulness. The time has come to ask whether the remedy is worse than the malady: are published health outcomes contributing to quality efforts or subverting more constructive approaches? This paper argues that attempts to force improvements through publishing health outcomes can be counterproductive, and outlines an alternative approach which involves fostering greater trust in professionalism as a basis for quality enhancements.

Protecting patient welfare in managed care: six safeguards

The public is very suspicious and fearful that managed care threatens their health because of its interest in reducing costs. Because physicians' decisions control 75 percent of all health care spending, managed care organizations are focusing their cost-cutting strategies on influencing physician decision making through financial incentives and guidelines. These two techniques have had some important contributions, especially in enhancing efficiency and standardizing care to a high level. Nevertheless, they pose a threat--and are perceived by the public to pose a threat--to patients' health and well-being. How can we mitigate the threats to patient welfare posed by financial incentives and guidelines? We propose and analyze six safeguards. These safeguards are not an attempt to revive the fee-for-service system, but an effort to make managed care ethical and to focus it on improving patient welfare. They are designed to work together to ensure that patient welfare remains the primary focus of managed care organizations; they try to create institutional structures that emphasize quality over mere cost reductions.

Patient and hospital characteristics associated with patient assessments of hospital obstetrical care

OBJECTIVES

The goals of this study were to examine the relationship of patient assessments of hospital care with patient and hospital characteristics. In addition, the authors sought to assess relationships between patient assessments and other patient-derived measures of care (eg, how much they were helped by the hospitalization and amount of pain experienced).

METHODS

The authors surveyed 16,051 women (response rate, 58%) discharged after labor and delivery from 18 hospitals during the study period of 1992 to 1994. Patient assessments were obtained using a previously validated survey instrument, Patient Judgment of Hospital Quality, that includes eight scales assessing different aspects of the process of care (eg, physician care, discharge procedures) and other single item assessments (eg, overall quality). For this study, we utilized five of the scales (physician care, nursing care, information, discharge preparation, global assessments [willingness to brag, recommend or return to the hospital]). For analysis, items were rated on a five-point ordinal scale from poor to excellent. For scoring purposes, responses were transformed to linear ratings, ranging from 0 to 100 (eg, 0 = poor care, 100 = excellent care).

RESULTS

In multivariable analyses, the authors found that patients who were older, white, not married, uninsured or had commercial insurance, and in better health status were significantly more likely to give higher assessments (P < 0.01), although very little of the variance in assessment scores was explained by these characteristics (2%-3%). In bivariate analyses, patient assessments were higher in nonteaching hospitals and those with fewer beds, fewer deliveries, lower cesarean-section (C-section) rates, fewer patients with Medicaid, and higher rates of vaginal births after C-section deliveries. When these variables were utilized as independent predictors in multivariable analyses using adjusted nested linear regression (to account for clustering of patients), few of the hospital characteristics reached a level of statistical significance. Finally, correlations between the five scales and other patient assessments of quality, such as how much they were helped by the hospitalization, were statistically significant (P < 0.01) and high in magnitude, ranging from 0.47 to 0.61.

CONCLUSIONS

Although hospital scores differed according to several patient and hospital characteristics, the magnitude of the associations was relatively small. The findings suggest that, with respect to obstetric care, patient assessments may represent a robust measure that can be applied to diverse hospitals and patient casemix.

Medicare and managed care

Medicare offers nearly universal, but limited, coverage for the elderly. The vast majority of beneficiaries therefore obtain supplemental coverage, or they enroll in HMOs to gain extra benefits at substantially lower or zero cost. This is possible because of reduced utilization and costs, as well as favorable selection of lower-risk enrollees into HMOs. Competition from HMOs may lower local fee-for-service costs as well. Quality and satisfaction measures are quite balanced, with some results showing better HMO performance and some worse. The absence of adequate risk-adjusted payments to HMOs, however, gives them little incentive to develop high-quality programs for the sickest enrollees.

Problems in assessing diabetes control in an ambulatory setting

Problems in using medical records to assess outcomes of diabetes care have not been well defined. We reviewed the medical records of 288 patients with diabetes receiving ambulatory care over a 2-year period. We determined the availability of different tests of glycemic control and described site performance as the percentage of patients with a blood glucose exceeding either 180 or 240 mg/dl. Glycosylated hemoglobin determinations were performed in only 26.7% of patients. A blood glucose was available in 208 patients (72.2%) during a 6-month outcome period. For almost 50% of the sample, the glucose was greater than 180 mg/dl, whereas in 20% it exceeded 240 mg/dl. Judgments of whether sites differed in performance depended on how control was defined. Using a single glucose determination and a threshold of 180 mg/dl, similar fractions of patients were poorly controlled at each site (51.2 versus 45.0 versus 47.0%) (P = 0.75). At 240 mg/dl, although, one site performed much worse than the other two (14.6 versus 16.7 versus 31.8%) (P = 0.02). These results highlight difficulties in defining the outcome measure when using medical records to evaluate quality of care.

Reducing random variation in reported rates of pressure ulcer development

OBJECTIVES

The authors evaluated methods of reporting on rates of pressure ulcer development in long-term care to identify approaches that lead to more stable estimates of actual performance.

METHODS

Performance measures for facilities that adequately adjust for both random variation and casemix should be relatively stable from one time period to the next. The authors calculated facility rates of pressure ulcer development over eight consecutive time periods and correlated measures over time using different reporting methods including z-scores, combining rates from several time periods, and limiting analyses to large facilities. Results were compared with a Monte Carlo simulation.

RESULTS

Observed facility rates of pressure ulcer development varied considerably over time. The average correlation coefficient across seven time comparisons for observed rates was 0.17. Reporting performance as a z-score or limiting the analyses to large facilities increased the correlation. Combining two time periods was effective only when used with one of these other approaches. The correlation coefficient based on a simulation using only large facilities was 0.51.

CONCLUSIONS

Random variation affects reported rates of pressure ulcer development. Using only large facilities and combining two time periods limits the effects of random variation and results in more stable estimates of performance. When describing performance, management must consider tradeoffs between having more accurate data, the frequency with which data are provided, and whether it is given to all providers.

Profiling outcomes of ambulatory care: casemix affects perceived performance

OBJECTIVES

The authors explored the role of casemix adjustment when profiling outcomes of ambulatory care.

METHODS

The authors reviewed the medical records of 656 patients with hypertension, diabetes, or chronic obstructive pulmonary disease (COPD) receiving care at one of three Department of Veterans Affairs medical centers. Outcomes included measures of physiological control for hypertension and diabetes, and of exacerbations for COPD. Predictors of poor outcomes, including physical examination findings, symptoms, and comorbidities, were identified and entered into regression models. Observed minus expected performance was described for each site, both before and after casemix adjustment.

RESULTS

Risk-adjustment models were developed that were clinically plausible and had good performance properties. Differences existed among the three sites in the severity of the patients being cared for. For example, the percentage of patients expected to have poor blood pressure control were 35% at site 1, 37% at site 2, and 44% at site 3 (P < 0.01). Casemix-adjusted measures of performance were different from unadjusted measures. Sites that were outliers (P < 0.05) with one approach had observed performance no different from expected with another approach.

CONCLUSIONS

Casemix adjustment models can be developed for outpatient medical conditions. Sites differ in the severity of patients they treat, and adjusting for these differences can alter judgments of site performance. Casemix adjustment is necessary when profiling outpatient medical conditions.

Rates of disagreement in imaging interpretation in a group of community hospitals

RATIONALE AND OBJECTIVES

Prospective studies of radiologists' interpretations of selected radiographs reported 20-40 years ago indicated error rates of 30% and higher. The authors retrospectively evaluated the interpretations of groups of radiologists and determined a range of rates of disagreement in interpretation. Quality assessment or recredentialing may add to the importance of such studies in the future.

MATERIALS AND METHODS

Over a 7-year period, a team of radiologists reviewed imaging interpretations in the radiology departments of six community hospitals. Each review, which lasted about 3 days, included evaluation of the interpretations of a 3%-4% sample of the images read by the radiologists at these hospitals. Reading errors were quantitated and evaluated qualitatively.

RESULTS

In a review of over 11,000 images read by 35 radiologists, the authors found a 4.4% mean rate of interpretation disagreement; only one radiologist had a mean rate above 8%. Qualitative analysis of the interpretation errors revealed a mean rate of 3.0% of errors that were considered to be below an acceptable standard of care. Radiologists whose errors included a relatively high proportion of false-positive findings tended to make relatively fewer total errors.

CONCLUSION

Rates of disagreement for a broad range of studies that radiologists interpret in a community hospital setting appear to be far lower than earlier studies on selective radiographs indicated.

League tables and acute myocardial infarction

BACKGROUND

Comparison of the outcomes of care provided by hospitals is a growing trend. Outcomes need to be distinguished into those attributable to the practice of hospitals and those that arise from differences in the characteristics of patients and the underlying morbidity of the populations for whom hospitals provide care. We explored these issues for deaths in hospital or within 30 days of discharge after acute myocardial infarction in Scotland, UK.

METHODS

We used records from December, 1992, to November, 1993, for 14,359 episodes of acute myocardial infarction, the death records of those who died, and 9391 death records for individuals who died after acute myocardial infarction but who had not been in hospital in the 30 days before death. Hospital discharge records were taken from the Scottish Morbidity Records. The outcomes we investigated were all-cause mortality within 30 days of discharge from hospital, and death from acute myocardial infarction at any time during the study period. We estimated separately effects attributable to patients' characteristics, hospitals, and areas of residence with multilevel modelling.

FINDINGS

We found significant differences between hospitals by age, sex, and medical history. The odds ratios for death ranged from 0.62 (95% CI 0.50-0.80) to 1.28 (1.07-1.59), relative to the average performance for Scotland as a whole. Analysis including area of residence, deaths occurring out of hospital, and more detailed information about patients showed no significant differences between hospitals for patients aged 70 years. By postcode area, there was a strong association between out-of-hospital deaths and deaths in hospital or shortly after discharge.

INTERPRETATION

Hospital outcomes may vary from one subgroup of patients to another and should be assessed independently of patients' areas of residence. Measures of performance that do not provide valid comparisons could diminish public confidence in hospital services.

A mental health program report card: a multidimensional approach to performance monitoring in public sector programs

This report presents a comprehensive, multi-dimensional mental health program performance monitoring system that has recently been implemented in the Department of Veterans Affairs. Principles underlying the development of the system are reviewed and 68 specific monitors are described addressing four major performance domains: access, inpatient care, outpatient care, and economic performance. Simple methods are presented for identifying outliers, for generating summary performance scores across series' of related monitors, and for adjusting results for differences in patient characteristics across locales. Although still technically imperfect, and therefore requiring continuous improvement, monitoring systems such as the one presented can be useful tools guiding and improving service delivery and mental health system performance, and providing a medium of accountability to consumers and other stakeholders.

History-taking and preventive medicine skills among primary care physicians: an assessment using standardized patients

BACKGROUND

The ability of primary care physicians to obtain important clinical information in initial encounters with new patients is a core competency that has received little attention in previous studies. This paper describes the history-taking and preventive screening skills of practicing primary care physicians in initial interactions with ambulatory patients, as determined by a large panel of standardized patients.

METHODS

Standardized patient cases with diverse presentations were developed and used to assess the clinical skills of 134 primary care physicians from five Northwest states. Scoring categories for each case identified the percentage and content of essential history items and preventive screening items performed. Physicians' scores were compared by training and practice characteristics.

RESULTS

Physicians asked 59% of essential history items. They frequently obtained appropriate information about presenting symptoms and medications, but they often missed important information about related symptoms and medical history. Physicians frequently screened for smoking and alcohol use, but rarely asked about recreational drug use. Although board-certified general internists performed more comprehensive histories than board-certified family practitioners in the same amount of time, both groups of providers missed a large number of items that should have been influential in developing diagnostic and treatment plans.

CONCLUSIONS

Primary care physicians may miss important patient information in their initial interactions with patients. Medical intake questionnaires or other approaches should be considered to ensure that more complete and accurate information is available to guide diagnostic and treatment plans.