Performance reports on quality--prototypes, problems, and prospects

A Multifaceted Organizational Physician Assessment Program: Validity Evidence and Implications for the Use of Performance Data

Objective

To provide validity evidence for a multifaceted organizational program for assessing physician performance and evaluate the practical and psychometric consequences of 2 approaches to scoring (mean vs top box scores).

Participants and Methods

Participants included physicians with a predominantly outpatient practice in general internal medicine (n=95), neurology (n=99), and psychiatry (n=39) at Mayo Clinic from January 1, 2013, through December 31, 2014. Study measures included hire year, patient complaint and compliment rates, note-signing timeliness, cost per episode of care, and Likert-scaled surveys from patients, learners, and colleagues (scored using mean ratings and top box percentages).

Results

Physicians had a mean ± SD of 0.32±1.78 complaints and 0.12±0.76 compliments per 100 outpatient visits. Most notes were signed on time (mean ± SD, 96%±6.6%). Mean ± SD cost was 0.56±0.59 SDs above the institutional average. Mean ± SD scores were 3.77±0.25 on 4-point and 4.06±0.31 to 4.94±0.08 on 5-point Likert-scaled surveys. Mean ± SD top box scores ranged from 18.6%±16.8% to 90.7%±10.5%. Learner survey scores were positively associated with patient survey scores (r=0.26; P=.003) and negatively associated with years in practice (r=−0.20; P=.02).

Conclusion

This study provides validity evidence for 7 assessments commonly used by medical centers to measure physician performance and reports that top box scores amplify differences among high-performing physicians. These findings inform the most appropriate uses of physician performance data and provide practical guidance to organizations seeking to implement similar assessment programs or use existing performance data in more meaningful ways.

Conceptual and Technical Issues Regarding the Use of HEDIS and HEDIS-Like Measures in Preferred Provider Organizations

Preferred provider organizations (PPOs) represent a diverse and complex set of arrangements among insurance entities, networks of physicians, network organizers, and purchasers. Opinions differ regarding the degree to which PPOs have responsibility to manage care and to measure and report key aspects of their performance to customers and the public. Technical and operational challenges to performance measurement currently limit public reporting, even when agreement exists that it is appropriate for PPOs to do so. The Health Plan and Employer Data and Information Set (HEDIS) is a health maintenance organization performance measure that could provide standards for PPO reporting. This article explores conceptual and methodological considerations regarding HEDIS and other performance measurement in PPOs and identifies failures of the current marketplace. While using some measures may be premature or inappropriate, there are significant opportunities to apply other measures now and, by doing so, to create a functional health care marketplace.

Conceptual and Technical Issues Regarding the Use of HEDIS and HEDIS-Like Measures in Preferred Provider Organizations

Preferred provider organizations (PPOs) represent a diverse and complex set of arrangements among insurance entities, networks of physicians, network organizers, and purchasers. Opinions differ regarding the degree to which PPOs have responsibility to manage care and to measure and report key aspects of their performance to customers and the public. Technical and operational challenges to performance measurement currently limit public reporting, even when agreement exists that it is appropriate for PPOs to do so. The Health Plan and Employer Data and Information Set (HEDIS) is a health maintenance organization performance measure that could provide standards for PPO reporting. This article explores conceptual and methodological considerations regarding HEDIS and other performance measurement in PPOs and identifies failures of the current marketplace. While using some measures may be premature or inappropriate, there are significant opportunities to apply other measures now and, by doing so, to create a functional health care marketplace.

Decentralization and health system performance - a focused review of dimensions, difficulties, and derivatives in India

Introduction

One of the principal goals of any health care system is to improve health through the provision of clinical and public health services. Decentralization as a reform measure aims to improve inputs, management processes and health outcomes, and has political, administrative and financial connotations. It is argued that the robustness of a health system in achieving desirable outcomes is contingent upon the width and depth of ‘decision space’ at the local level. Studies have used different approaches to examine one or more facets of decentralization and its effect on health system functioning; however, lack of consensus on an acceptable framework is a critical gap in determining its quantum and quality. Theorists have resorted to concepts of ‘trust’, ‘convenience’ and ‘mutual benefits’ to explain, define and measure components of governance in health. In the emerging ‘continuum of health services’ model, the challenge lies in identifying variables of performance (fiscal allocation, autonomy at local level, perception of key stakeholders, service delivery outputs, etc.) through the prism of decentralization in the first place, and in establishing directed relationships among them.

Methods

This focused review paper conducted extensive web-based literature search, using PubMed and Google Scholar search engines. After screening of key words and study objectives, we retrieved 180 articles for next round of screening. One hundred and four full articles (three working papers and 101 published papers) were reviewed in totality. We attempted to summarize existing literature on decentralization and health systems performance, explain key concepts and essential variables, and develop a framework for further scientific scrutiny. Themes are presented in three separate segments of dimensions, difficulties and derivatives.

Results

Evaluation of local decision making and its effect on health system performance has been studied in a compartmentalized manner. There is sparse evidence about innovations attributable to decentralization. We observed that in India, there is very scant evaluative study on the subject. We didn’t come across a single study examining the perception and experiences of local decision makers about the opportunities and challenges they faced. The existing body of evidences may be inadequate to feed into sound policy making. The principles of management hinge on measurement of inputs, processes and outputs. In the conceptual framework we propose three levels of functions (health systems functions, management functions and measurement functions) being intricately related to inputs, processes and outputs. Each level of function encompasses essential elements derived from the synthesis of information gathered through literature review and non-participant observation. We observed that it is difficult to quantify characteristics of governance at institutional, system and individual levels except through proxy means.

Conclusion

There is an urgent need to sensitize governments and academia about how best more objective evaluation of ‘shared governance’ can be undertaken to benefit policy making. The future direction of enquiry should focus on context-specific evidence of its effect on the entire spectrum of health system, with special emphasis on efficiency, community participation, human resource management and quality of services.

Keys to successful implementation of a French national quality indicator in health care organizations: a qualitative study

Background

Several countries have launched public reporting systems based on quality indicators (QIs) to increase transparency and improve quality in health care organizations (HCOs). However, a prerequisite to quality improvement is successful local QI implementation. The aim of this study was to explore the pathway through which a mandatory QI of the French national public reporting system, namely the quality of the anesthesia file (QAF), was put into practice.

Method

Seven ethnographic case studies in French HCOs combining in situ observations and 37 semi-structured interviews.

Results

A significant proportion of potential QAF users, such as anesthetists or other health professionals were often unaware of quality data. They were, however, involved in improvement actions to meet the QAF criteria. In fact, three intertwined factors influenced QAF appropriation by anesthesia teams and impacted practice. The first factor was the action of clinical managers (chief anesthetists and head of department) who helped translate public policy into local practice largely by providing legitimacy by highlighting the scientific evidence underlying QAF, achieving consensus among team members, and pointing out the value of QAF as a means of work recognition. The two other factors related to the socio-material context, namely the coherence of information systems and the quality of interpersonal ties within the department.

Conclusions

Public policy tends to focus on the metrological validity of QIs and on ranking methods and overlooks QI implementation. However, effective QI implementation depends on local managerial activity that is often invisible, in interaction with socio-material factors. When developing national quality improvement programs, health authorities might do well to specifically target these clinical managers who act as invaluable mediators. Their key role should be acknowledged and they ought to be provided with adequate resources.

Recognizing Differences in Hospital Quality Performance for Pediatric Inpatient Care

BACKGROUND

Hospital quality-of-care measures are publicly reported to inform consumer choice and stimulate quality improvement. The number of hospitals and states with enough pediatric hospital discharges to detect worse-than-average inpatient care remains unknown.

METHODS

This study was a retrospective analysis of hospital discharges for children aged 0 to 17 years from 3974 hospitals in 44 states in the 2009 Kids' Inpatient Database. For 11 measures of all-condition or condition-specific quality, we assessed the number of hospitals and states that met a "power standard" of 80% power for a 5% level significance test to detect when care is 20% worse than average over a 3-year period. For this assessment, we approximated volume as 3 times actual 2009 admission volumes.

RESULTS

For all-condition quality, 1380 hospitals (87% of all pediatric discharges) and all states met the power standard for the family experience-of-care measure; 1958 hospitals (95% of discharges) and all states met the standard for adverse drug events. For condition-specific quality measures of asthma, birth, and mental health, 203 to 482 hospitals (52%-90% of condition-specific discharges) met the power standard and 40 to 44 states met the standard. One hospital and 16 states met the standard for sickle cell disease. No hospital and ≤27 states met the standard for the remaining measures studied (appendectomy, cerebrospinal fluid shunt surgery, gastroenteritis, heart surgery, and seizure).

CONCLUSIONS

Most children are admitted to hospitals in which all-condition measures of quality have adequate power to show modest differences in performance from average, but most condition-specific measures do not. Policies regarding incentives for pediatric inpatient quality should take these findings into account.

Comparative effectiveness issues in lung cancer

Lung cancer accounts for more cancer deaths than breast, prostate, colorectal and pancreatic cancer combined. With an aging population, greater intensity of cancer care, and the need for care of the growing number of cancer survivors, comparative effectiveness research opportunities will continue to emerge for this disease. In this chapter, we focus on CER opportunities in lung cancer surgery from the vantage point of those factors directly influenced by the surgeon, patient and the healthcare system.

Diabetic retinopathy care--an international quality comparison

PURPOSE

The purpose of this paper is to study the quality indicator appropriateness and use it for international quality comparison on diabetic retinopathy (DR) patient care process in one American and one Dutch eye hospital.

DESIGN/METHODOLOGY/APPROACH

A 17-item DR quality indicator set was composed based on a literature review and systematically applied in two hospitals. Qualitative analysis entailed document study and 12 semi-structured face-to-face interviews with ophthalmologists, managers, and board members of the two hospitals.

FINDINGS

While the medical-clinical approach to DR treatment in both hospitals was similar, differences were found in quality of care perception and operationalization. Neither hospital systematically used outcome indicators for DR care. On the process level, the authors found larger differences. Similarities and differences were found in the structure of both hospitals. The hospitals' particular contexts influenced the interpretation and use of quality indicators.

PRACTICAL IMPLICATIONS

Although quality indicators and quality comparison between hospitals are increasingly used in international settings, important local differences influence their application. Context should be taken into account. Since that context is locally bound and directly linked to hospital setting, caution should be used interpreting the results of quality comparison studies.

ORIGINALITY/VALUE

International quality comparison is increasingly suggested as a useful way to improve healthcare. Little is known, however, about the appropriateness and use of quality indicators in local hospital care practices.

Dissemination of performance information and continuous improvement: A narrative systematic review

PURPOSE

Developing a performance measure and reporting the results to support decision making at an individual level has yielded poor results in many health systems. The purpose of this paper is to highlight the factors associated with the dissemination of performance information that generate and support continuous improvement in health organizations.

DESIGN/METHODOLOGY/APPROACH

A systematic data collection strategy that includes empirical and theoretical research published from 1980 to 2010, both qualitative and quantitative, was performed on Web of Science, Current Contents, EMBASE and MEDLINE. A narrative synthesis method was used to iteratively detail explicative processes that underlie the intervention. A classification and synthesis framework was developed, drawing on knowledge transfer and exchange (KTE) literature. The sample consisted of 114 articles, including seven systematic or exhaustive reviews.

FINDINGS

Results showed that dissemination in itself is not enough to produce improvement initiatives. Successful dissemination depends on various factors, which influence the way collective actors react to performance information such as the clarity of objectives, the relationships between stakeholders, the system's governance and the available incentives.

RESEARCH LIMITATIONS/IMPLICATIONS

This review was limited to the process of knowledge dissemination in health systems and its utilization by users at the health organization level. Issues related to improvement initiatives deserve more attention.

PRACTICAL IMPLICATIONS

Knowledge dissemination goes beyond better communication and should be considered as carefully as the measurement of performance. Choices pertaining to intervention should be continuously prompted by the concern to support organizational action.

ORIGINALITY/VALUE

While considerable attention was paid to the public reporting of performance information, this review sheds some light on a more promising avenue for changes and improvements, notably in public health systems.

Analytical perspectives on performance-based management: an outline of theoretical assumptions in the existing literature

Performance-based management (PBM) has become a dominant form of governance in health care and there is a need for careful assessment of its function and effects. This article contains a cross-disciplinary literature synthesis of current studies of PBM. Literature was retrieved by database searches and categorized according to analytical differences and similarities concerning (1) purpose and (2) governance mechanism of PBM. The literature could be grouped into three approaches to the study of PBM, which we termed: the ‘functionalist’, the ‘interpretive’ and the ‘post-modern’ perspective. In the functionalist perspective, PBM is perceived as a management tool aimed at improving health care services by means of market-based mechanisms. In the interpretive perspective, the adoption of PBM is understood as consequence of institutional and individual agents striving for public legitimacy. In the post-modern perspective, PBM is analysed as a form of governance, which has become so ingrained in Western culture that health care professionals internalize and understand their own behaviour and goals according to the values expressed in these governance systems. The recognition of differences in analytical perspectives allows appreciation of otherwise implicit assumptions and potential implications of PBM. Reflections on such differences are important to ensure vigilant appropriation of shifting management tools in health quality governance.

Profiling hospitals based on emergency readmission: a multilevel transition modelling approach

Emergency readmission is seen as an important part of the United Kingdom government policy to improve the quality of care that patients receive. In this context, patients and the public have the right to know how well different health organizations are performing. Most methods for profiling estimate the expected numbers of adverse outcomes (e.g. readmission, mortality) for each organization. A number of statistical concerns have been raised, such as the differences in hospital sizes and the unavailability of relevant data for risk adjustment. Having recognized these statistical concerns, a new framework known as the multilevel transition model is developed. Hospital specific propensities of the first, second and further readmissions are considered to be measures of performance, where these measures are used to define a new performance index. During the period 1997 and 2004, the national (English) hospital episodes statistics dataset comprise more than 5 million patient readmissions. Implementing a multilevel model using the complete population dataset could possibly take weeks to estimate the parameters. To resolve the problem, we extract 1000 random samples from the original data, where each random sample is likely to lead to differing hospital performance measures. For computational efficiency a Grid implementation of the model is developed. Analysing the output from the full 1000 sample, we noticed that 4 out of the 5 worst performing hospitals treating cancer patients were in London. These hospitals are known to be the leading NHS Trusts in England, providing diverse range of services to complex patients, and therefore it is inevitable to expect higher numbers of emergency readmissions.

The impact of emerging standards adoption on automated quality reporting

Current quality measurement processes are labor-intensive, involving manual chart reviews and use of paper-based quality measures that vary in format and definitions from measure to measure. Automated quality reporting is considered by many to be an important tool that will help close the gaps in the quality of US health by increasing the timeliness, effectiveness, and use of quality assessment. In 2007, the US Department of Health and Human Services Office of the National Coordinator for Health Information Technology (ONC) funded three Nationwide Health Information Network (NHIN) health information exchanges (HIE) to demonstrate the feasibility of automated quality reporting by using existing or emerging standards to aggregate information from multiple providers, transmit patient-level quality data in standardized formats, perform an automated quality assessment, and generate a quality report document for electronic transmission. Long Beach Network for Health (LBNH), a NHIN Cooperative HIE, developed a web-based, real-time quality assessment service that calculates quality of care measure using clinical data aggregated through a HIE. LBNH used a set of draft standards to demonstrate automated quality reporting, but noted three important recommendations for future work. First, greater coordination is needed around initiatives that address the gaps in electronic quality measurement standards and processes, including strong Federal involvement and guidance. Second, a harmonized, evergreen quality use case is needed to provide stakeholders with a common understanding on the constantly evolving approaches towards automated quality measurement and reporting. Finally, there needs to be substantial investment in building on existing work and developing a comprehensive set of data and messaging standards to preserve semantic interoperability of quality measure data.

Benchmarking strategies for measuring the quality of healthcare: problems and prospects

Over the last few years, increasing attention has been directed toward the problems inherent to measuring the quality of healthcare and implementing benchmarking strategies. Besides offering accreditation and certification processes, recent approaches measure the performance of healthcare institutions in order to evaluate their effectiveness, defined as the capacity to provide treatment that modifies and improves the patient's state of health. This paper, dealing with hospital effectiveness, focuses on research methods for effectiveness analyses within a strategy comparing different healthcare institutions. The paper, after having introduced readers to the principle debates on benchmarking strategies, which depend on the perspective and type of indicators used, focuses on the methodological problems related to performing consistent benchmarking analyses. Particularly, statistical methods suitable for controlling case-mix, analyzing aggregate data, rare events, and continuous outcomes measured with error are examined. Specific challenges of benchmarking strategies, such as the risk of risk adjustment (case-mix fallacy, underreporting, risk of comparing noncomparable hospitals), selection bias, and possible strategies for the development of consistent benchmarking analyses, are discussed. Finally, to demonstrate the feasibility of the illustrated benchmarking strategies, an application focused on determining regional benchmarks for patient satisfaction (using 2009 Lombardy Region Patient Satisfaction Questionnaire) is proposed.

Profiling hospitals by survival of patients with colorectal cancer

OBJECTIVE

To profile hospitals by survival rates of colorectal cancer patients in multiple periods after initial treatment.

DATA SOURCES

California Cancer Registry data from 50,544 patients receiving primary surgery with curative intent for stage I-III colorectal cancer in 1994-1998, supplemented with hospital discharge abstracts.

STUDY DESIGN

We estimated a single Bayesian hierarchical model to quantify associations of survival to 30 days, 30 days to 1 year, and 1-5 years by hospital, adjusted for patient age, sex, race, stage, tumor site, and comorbidities. We compared two profiling methods for 30-day survival and four longer-term profiling methods by the fractions of hospitals with demonstrably superior survival profiles and of hospital pairs whose relative standings could be established confidently.

PRINCIPAL FINDINGS

Interperiod correlation coefficients of the random effects are (95 percent credible interval 0.27, 0.85), (0.20, 0.76), and (0.19, 0.82). The three-period model ranks 5.4 percent of pairwise comparisons by 30-day survival with at least 95 percent confidence, versus 3.3 percent of pairs using a single-period model, and 15-20 percent by weighted multiperiod methods.

CONCLUSIONS

The quality of care for colorectal cancer provided by a hospital system is somewhat consistent across the immediate postoperative and long-term follow-up periods. Combining mortality profiles across longer periods may improve the statistical reliability of outcome comparisons.

A two-stage estimation of hospital quality using mortality outcome measures: an application using hospital administrative data

This paper proposes a method of deriving a quality indicator for hospitals using mortality outcome measures. The method aggregates any number of mortality outcomes into a single indicator via a two-stage procedure. In the first stage, mortality outcomes are risk-adjusted using a system of seemingly unrelated regression equations. These risk-adjusted mortality rates are then aggregated into a single quality indicator in the second stage via weighted least squares. This method addresses the dimensionality problem in measuring hospital quality, which is multifaceted in nature. In addition, our method also facilitates further analyses of determinants of hospital quality by allowing the resulting quality estimates be associated with hospital characteristics. The method is applied to a sample of heart-disease episodes extracted from hospital administrative data from the state of Victoria, Australia. Using the quality estimates, we show that teaching hospitals and large regional hospitals provide higher quality of care than other hospitals and this superior performance is related to hospital case-load volume.

Quality initiatives: Key performance indicators for measuring and improving radiology department performance

Key performance indicators (KPIs) are financial and nonfinancial measures that are used to define and evaluate the success of an organization. KPIs differ, depending on the nature of the organization and the organizational strategy; they are devised to help evaluate the progress of an organization toward achieving its long-term goals and fulfilling its vision. In healthcare organizations, performance assessment is especially critical for the development of best practices that can lead to improved outcomes in patient care, and KPIs have been incorporated into many healthcare management systems. In the future, radiology-specific KPIs such as those in use at the authors' institution may help provide a framework for measuring performance in radiology practice.

Relationship of primary care physicians' patient caseload with measurement of quality and cost performance

CONTEXT

Sufficient numbers of patients are necessary to generate statistically reliable measurements of physicians' quality and cost performance.

OBJECTIVE

To determine whether primary care physicians in the same physician practice collectively see enough Medicare patients annually to detect meaningful differences between practices in ambulatory quality and cost measures.

DESIGN, SETTING, AND PATIENTS

Primary care physicians in the United States were linked to their physician practices using the Healthcare Organization Services database maintained by IMS Health. Patients who visited primary care physicians in the 2005 Medicare Part B 20% sample were used to estimate Medicare caseloads per practice. Caseloads necessary to detect 10% relative differences in costs and quality were calculated using national mean ambulatory Medicare spending, rates of mammography for women 66 to 69 years, and hemoglobin A(1c) testing for 66- to 75-year-olds with diabetes, preventable hospitalization rate, and 30-day readmission rate after discharge for congestive heart failure (CHF).

MAIN OUTCOME MEASURES

Percentage of primary care physician practices with a sufficient number of eligible patients to detect a 10% relative difference in each performance measure.

RESULTS

Primary care physician practices had annual median caseloads of 260 Medicare patients (interquartile range [IQR], 135-500), 25 women eligible for mammography (IQR, 10-50), 30 patients with diabetes eligible for hemoglobin A(1c) testing (IQR, 15-55), and 0 patients hospitalized for CHF. For ambulatory costs, mammography rate, and hemoglobin A(1c) testing rate, the percentage of primary care physician practices with sufficient caseloads to detect 10% relative differences in performance ranged from less than 10% of practices with fewer than 11 primary care physicians to 100% of practices with more than 50 primary care physicians. None of the primary care physician practices had sufficient caseloads to detect 10% relative differences in preventable hospitalization or 30-day readmission after discharge for CHF.

CONCLUSION

Relatively few primary care physician practices are large enough to reliably measure 10% relative differences in common measures of quality and cost performance among fee-for-service Medicare patients.

Role and quality of nurse practitioner practice: a policy issue

Nurse practitioners (NPs) are at a crucial period in the development of their profession. The increasing demand for primary care practitioners is changing the environment in which they practice. As they face both increased opportunity and continuing opposition to independent practice, NPs must define their role in healthcare delivery and establish their capabilities. The debate around what is and is not an appropriate role for the NP often focuses on quality of care; however, the real issues may be turf and economic defensiveness in an increasingly competitive market. This article discusses the challenges NPs face in establishing the quality of care they provide, and it reviews the literature on the subject, identifying its strengths and weaknesses and recommending policy changes.

Reliability of a 25-item low-stakes multiple-choice assessment of bronchoscopic knowledge

BACKGROUND

A need for improved patient safety, quality of care, and accountability has prompted the development of competency-based educational processes. Assessment tools related to bronchoscopy training, however, have not yet been developed or validated.

PURPOSES

To determine whether 25 multiple-choice questions (MCQs) extracted from the free, Web-based Essential Bronchoscopist (EB) learning guide qualify in their original form as a preliminary pool of questions for a low-stakes assessment of bronchoscopic knowledge.

MATERIALS AND METHODS

Twenty-five randomly selected MCQs from among the top 70 question-answer sets of the EB were administered to 40 self-declared novice bronchoscopists (n = 13), experienced bronchoscopists (n = 21), and expert bronchoscopists (n = 6). A difficulty index and a discrimination index (DI) were calculated for each item. Internal consistency reliability was calculated using item-total correlation and Cronbach alpha. Content validity was determined by five independent experts. Ideal test items based on a difficulty index and item-total correlation were administered to a different group of 24 bronchoscopists to prospectively reassess internal consistency reliability.

RESULTS

The mean (+/- SD) score for the 40 participants was 16.47 +/- 3.72 (median score, 17; score range, 7 to 22). The mean difficulty index was 0.65 +/- 0.22, and the mean DI was 0.52 +/- 0.28. Item total-correlations ranged from - 0.01 to + 0.71. Test content was unanimously validated. The Cronbach alpha was 0.69. There was no significant correlation between scores and the number of bronchoscopies performed or self-declared expertise. Eleven ideal test MCQs were identified. The internal consistency of these items remained satisfactory (Cronbach alpha = 0.75) when assessed prospectively in a different cohort.

CONCLUSION

Reliable and valid MCQs were identified to initiate a preliminary pool of questions for a low-stakes assessment of bronchoscopic knowledge.

Population-based standardization (PBS) of institutional cesarean delivery rates

BACKGROUND

Raw cesarean section rates are often compared among institutions with different patient populations. The purpose of this study was to develop an easily reproducible mathematical model that allows comparisons of cesarean delivery rates across different patient populations and institutions.

STUDY DESIGN

We first calculated three institution's cesarean delivery rates for each of ten groups of patients based on age and parity. The population based adjusted total cesarean delivery rate was then calculated for each institution based on the distribution of patients in the 2004 national birth data (n=4,097,029) but using the three institutions' individual cesarean delivery rates.

RESULTS

The adjusted for age and parity cesarean delivery rate was significantly lower from raw cesarean delivery rates in two of the three institutions (A: 28.2% adjusted vs. 36.5% raw; P=0; B: 28.2% adjusted vs. 30.4% raw, P=0.0411; C: 28.7% adjusted vs. 29.7% raw, NS) reflecting the older and more nulliparous patients in these two institutions.

CONCLUSIONS

Our study confirms that raw cesarean delivery rate should not be used to compare quality of care within and among different institutions unless they are adjusted for different patient characteristics. We believe that using unadjusted cesarean delivery rates without appropriate adjustments in quality assurance and when comparing data with other institutions and the national rate is erroneous and misleading.

Fostering knowledge exchange between researchers and decision-makers: exploring the effectiveness of a mixed-methods approach

OBJECTIVES

Knowledge exchange is thought to enhance research utilization by decision-makers but there is little guidance on appropriate methods. This study evaluated the effectiveness of a research planning exercise utilizing technical (ranking, rating) and interpretive strategies (interdisciplinary workshop deliberation).

METHODS

Participants were surveyed to establish research priorities and professional roles. Observation was used to examine actual contribution and outcomes. Semi-structured interviews with participants elicited perceived outcomes, commitment, contribution and learning. Survey data was reported with summary statistics. Transcripts were analyzed qualitatively.

RESULTS

Stakeholders were satisfied with the overall process, gaps in research were prioritized, and research questions were proposed, but anticipated intermediate or lateral outcomes were not achieved. Identifying differing perspectives and establishing relationships were unanticipated outcomes. Barriers included group dynamics, lack of clarity on objectives and processes, and minimal experience or interest in interpretive activities.

CONCLUSIONS

A conceptual framework for evaluating factors influencing knowledge exchange outcomes had not been previously investigated. Strategies for overcoming identified barriers include better facilitation, involving a critical volume of non-clinicians, in-person sharing of background information, and incentives for decision-makers. Further research is required to examine the effectiveness of different forms of knowledge exchange, and the degree to which they are currently being practiced.

Accuracy of hospital report cards based on administrative data

CONTEXT

Many of the publicly available health quality report cards are based on administrative data. ICD-9-CM codes in administrative data are not date stamped to distinguish between medical conditions present at the time of hospital admission and complications, which occur after hospital admission. Treating complications as preexisting conditions gives poor-performing hospitals "credit" for their complications and may cause some hospitals that are delivering low-quality care to be misclassified as average- or high-performing hospitals.

OBJECTIVE

To determine whether hospital quality assessment based on administrative data is impacted by the inclusion of condition present at admission (CPAA) modifiers in administrative data as a date stamp indicator.

DESIGN, SETTING, AND PATIENTS

Retrospective cohort study based on 648,866 inpatient admissions between 1998 and 2000 for coronary artery bypass graft (CABG) surgery, coronary angioplasty (PTCA), carotid endarterectomy (CEA), abdominal aortic aneurysm (AAA) repair, total hip replacement (THR), acute MI (AMI), and stroke using the California State Inpatient Database which includes CPAA modifiers. Hierarchical logistic regression was used to create separate condition-specific risk adjustment models. For each study population, one model was constructed using only secondary diagnoses present at admission based on the CPAA modifier: "date stamp" model. The second model was constructed using all secondary diagnoses, ignoring the information present in the CPAA modifier: the "no date stamp model." Hospital quality was assessed separately using the "date stamp" and the "no date stamp" risk-adjustment models.

RESULTS

Forty percent of the CABG hospitals, 33 percent of the PTCA hospitals, 40 percent of the THR hospitals, and 33 percent of the AMI hospitals identified as low-performance hospitals by the "date stamp" models were not classified as low-performance hospitals by the "no date stamp" models. Fifty percent of the CABG hospitals, 33 percent of the PTCA hospitals, 50 percent of the CEA hospitals, and 36 percent of the AMI hospitals identified as low-performance hospitals by the "no date stamp" models were not identified as low-performance hospitals by the "date stamp" models. The inclusion of the CPAA modifier had a minor impact on hospital quality assessment for AAA repair, stroke, and CEA.

CONCLUSION

This study supports the hypothesis that the use of routine administrative data without date stamp information to construct hospital quality report cards may result in the mis-identification of hospital quality outliers. However, the CPAA modifier will need to be further validated before date stamped administrative data can be used as the basis for health quality report cards.

The Predictive Accuracy Of The New York State Coronary Artery Bypass Surgery Report-Card System

We examined the impact of New York State's public reporting system for coronary artery bypass surgery fifteen years after its launch. We found that users who picked a top-performing hospital or surgeon from the latest available report had approximately half the chance of dying as did those who picked a hospital or surgeon from the bottom quartile. Nevertheless, performance was not associated with a subsequent change in market share. Surgeons with the highest mortality rates were much more likely than other surgeons to retire or leave practice after the release of each report card.

Profiling providers on use of adjuvant chemotherapy by combining cancer registry and medical record data

PURPOSE

Treatment information collected by cancer registries can be used to monitor the provision of guideline-recommended chemotherapy to colorectal cancer patients. Incomplete information may bias comparisons of these rates. We developed statistical methods that combine data from a registry and physicians' records to assess hospital quality.

DATA

From California Cancer Registry data, we selected all patients (n=12,594) newly diagnosed with stage III colon cancer or stage II or III rectal cancer from 428 hospitals during the years 1994 to 1998. To assess rates and predictors of underreporting of chemotherapy, we surveyed physicians treating 1449 of these patients from 98 hospitals during the years 1996 to 1997.

METHODS

Using Bayesian statistical models, we imputed unobserved treatments. We studied the impact of underreporting on provider profiling by comparing rankings, estimates, and credible intervals based only on registry data to those incorporating physician survey data.

RESULTS

Analyses that account for incompleteness of reporting yielded wider credible intervals for provider profiles than those that ignored such incompleteness. Among the 109 (25%) hospitals in the highest quartile of chemotherapy rates according to the registry data, 16 were not so classified when incomplete reporting was taken into account. With the more comprehensive model, 12 hospitals could be identified that ranked in the top quartile with probability>0.90.

CONCLUSION

Estimates of adjusted hospital chemotherapy rates based solely on cancer registry data overstate the precision of assessments of hospital quality. Using additional information from a physician survey and applying rigorous statistical models, better inferences can be drawn about provider quality.

Quality assurance and dental hygiene

Dental hygiene in Canada has experienced significant growth. It has shifted from an emerging occupation to a regulated health profession in several jurisdictions. Many achievements may be attributed to this growth, including self-regulation and a national code of ethics. However, the majority of Canadian dental hygienists are relying on traditional, outdated and ineffective quality assurance mechanisms, such as mandatory continuing education requirements. In the interests of public protection, dental hygiene needs to ensure that the quality assurance activities required from its members are effective, valid and reliable. Quality assurance in health care continues to undergo modifications that better reflect the public's need for competent, ethical, safe and appropriate health care. Dental hygienists and dental hygiene regulatory bodies are challenged to find valid, reliable and effective methods of quality assurance. This paper discusses some of the developments in quality assurance in health care as well as some of the key and significant achievements of dental hygiene in Canada. The use of quality assurance mechanisms currently used in dental hygiene in Canada is also discussed. The paper concludes with a discussion on the potential barriers and issues that the profession may face when attempting to incorporate suitable quality assurance activities into daily dental hygiene practice.

Quality assurance in health care: past, present and future

Quality of health care delivery is a growing concern globally given current budget restraints and increasing demands on health care providers. The variety of quality assurance and quality management activities equals the numerous ways health care practitioners of all genres provide health care. Dental hygienists around the world must be knowledgeable about quality assurance and management in health care as it is a significant factor in the evolution of the dental hygiene profession and the quality of oral health care provided by dental hygienists. The objective of this research was to conduct a literature review on quality assurance and quality management. A MEDLINE search from 1966 to 2002 was conducted. The search resulted in approximately 145 articles. Additional references from works generated by the search were also obtained. The literature revealed information on the background and history of quality assurance and quality management. Much of the literature was devoted to discussions of the validity, reliability and effectiveness of most prominent quality management activities being utilised in health care today. The investigation revealed numerous issues and barriers surrounding quality management. This article concludes with suggestions for future directions of quality assurance and quality management.

Quality-based purchasing in health care

Quality-based purchasing is a growing trend that seeks to improve healthcare quality through the purchaser-provider relationship. This article provides a unifying conceptual framework, presents examples of the purchaser-provider relationship in countries at different income levels, and identifies important supporting mechanisms for quality-based purchasing. As countries become wealthier, a higher proportion of healthcare spending is channeled through pooled arrangements, allowing for greater involvement of purchasers in promoting the quality of service provision. Global and line item budgets are the most common type of provider payment system in low and middle-income countries. In these countries, improving public hospital performance through contracting and incentives is a key issue. In middle and high-income countries, there are several documented examples of governments contracting to private or non-governmental health care providers, resulting in higher perceived quality of care and lower delivery costs. Encouraging quality through employer purchasing arrangements has been promoted in several countries, particularly the United States. Community-based financing schemes are an increasingly common form of health financing in parts of sub-Saharan Africa and Asia, but these schemes still cover less than 10% of national populations in countries in which they are active. To date, there is little evidence of their impact on healthcare quality. The availability of information--concerning healthcare service provision and outcomes--determines the options for establishing and monitoring contract provisions and promoting quality. Regardless of the context, quality-based purchasing depends critically on informa-tion--reporting, monitoring, and providing useful information to healthcare consumers. In many low and middle-income countries, the lack of availability of information is the principal constraint on measuring performance, a critical component of quality-based purchasing.

Quality care and practice variation: the roles of practice guidelines and public profiles

Goals of the quality-of-care initiative are to improve the structure, process, and outcome of health care. The effectiveness of methods to improve quality have been largely unverified. Most methods are costly to implement and time-consuming to perform; some threaten professional autonomy. The characteristic feature of modern medicine that fuels the debate over quality is the variation in the delivery of health care. This review examines the "variation phenomenon" in medicine and the roles that practice guidelines and physician profiling have in improving health care, in general, and for adult cataract, in particular.

Patient-reported measures of health: The Veterans Health Study

The goal of the Veterans Health Study (VHS) was to extend the work of the Medical Outcomes Study (MOS) into the VA, by developing methodology for monitoring patient-based outcomes of care for use in ambulatory outpatient care. The principal objective of the VHS was developing valid and reliable measures to assess general health-related quality of life (HRQoL) and identifying the presence of selected health conditions, their severity, and their impact on HRQoL. In this article, we provide an overview of the historical context, framework, objectives, and applications of the VHS for the purpose of assessing the health outcomes of veteran patients. The VHS is a prospective observational study that has followed 2425 VA patients for up to 2 years. The patients were sampled from users of the Veterans Affairs (VA) ambulatory care system in the Boston area. The health conditions selected were hypertension, diabetes, chronic lung disease, osteoarthritis of the knee, chronic low-back pain, and alcohol-related problems. These conditions were chosen because they are both prevalent in the VA and have measurable impacts on HRQoL. One of the cornerstones of the VHS was the development of the Veterans SF-36, modified from the MOS SF-36 for use in veteran ambulatory populations. Other key accomplishments included the development of patient-based disease-specific measures of health and the establishment of methods and logistics for comprehensive health outcomes research in large health care systems such as the VA, using these patient-based measures. Selected measures developed in the VHS, eg, the Veterans SF-36, have been integrated into the VA outcomes measurement system. The scope of the VHS is unique; it resulted in the development of a broad range of patient-focused process and outcome measures, as well as methodologies for assessing large numbers of patients, that have been widely used in the VA outpatient health care system for monitoring health outcomes across the nation.

Measuring the quality of diabetes care using administrative data: is there bias?

OBJECTIVES

Health care organizations often measure processes of care using only administrative data. We assessed whether measuring processes of diabetes care using administrative data without medical record data is likely to underdetect compliance with accepted standards for certain groups of patients.

DATA SOURCES/STUDY SETTING

Assessment of quality indicators during 1998 using administrative and medical records data for a cohort of 1,335 diabetic patients enrolled in three Minnesota health plans.

STUDY DESIGN

Cross-sectional retrospective study assessing hemoglobin A1c testing, LDL cholesterol testing, and retinopathy screening from the two data sources. Analyses examined whether patient or clinic characteristics were associated with underdetection of quality indicators when administrative data were not supplemented with medical record data.

DATA COLLECTION/EXTRACTION METHODS

The health plans provided administrative data, and trained abstractors collected medical records data.

PRINCIPAL FINDINGS

Quality indicators that would be identified if administrative data were supplemented with medical records data are often not identified using administrative data alone. In adjusted analyses, older patients were more likely to have hemoglobin A1c testing underdetected in administrative data (compared to patients <45 years, OR 2.95, 95 percent CI 1.09 to 7.96 for patients 65 to 74 years, and OR 4.20, 95 percent CI 1.81 to 9.77 for patients 75 years and older). Black patients were more likely than white patients to have retinopathy screening underdetected using administrative data (2.57, 95 percent CI 1.16 to 5.70). Patients in different health plans also differed in the likelihood of having quality indicators underdetected.

CONCLUSIONS

Diabetes quality indicators may be underdetected more frequently for elderly and black patients and the physicians, clinics, and plans who care for such patients when quality measurement is based on administrative data alone. This suggests that providers who care for such patients may be disproportionately affected by public release of such data or by its use in determining the magnitude of financial incentives.

Interventions for patients, providers, and health care organizations

Clinicians and the organizations within which they practice play a major role in enabling patient participation in cancer screening and ensuring quality services. Guided by an ecologic framework, the authors summarize previous literature reviews and exemplary studies of breast, cervical, and colorectal cancer screening intervention studies conducted in health care settings. Lessons learned regarding interventions to maximize the potential of cancer screening are distilled. Four broad lessons learned emphasize that multiple levels of factors-public policy, organizational systems and practice settings, clinicians, and patients-influence cancer screening; that a diverse set of intervention strategies targeted at each of these levels can improve cancer screening rates; that the synergistic effects of multiple strategies often are most effective; and that targeting all components of the screening continuum is important. Recommendations are made for future research and practice, including priorities for intervention research specific to health care settings, the need to take research phases into consideration, the need for studies of health services delivery trends, and methods and measurement issues.

The effect of publicly reporting hospital performance on market share and risk-adjusted mortality at high-mortality hospitals

BACKGROUND

It is unclear whether publicly reporting hospitals' risk-adjusted mortality affects market share and mortality at outlier hospitals.

OBJECTIVES

To examine hospitals' market share and risk-adjusted mortality from 1991 to 1997 at hospitals participating in Cleveland Health Quality Choice (CHQC).

RESEARCH DESIGN

Time series.

SUBJECTS

Changes in market share were examined for all patients hospitalized with acute myocardial infarction, heart failure, gastrointestinal hemorrhage, obstructive pulmonary disease, pneumonia, or stroke at all 30 nonfederal hospitals in Northeast Ohio. Patients insured by Medicare were used to examine changes in mortality.

MEASURES

Trends in market share (proportion of patients with the target conditions discharged from a given hospital) and risk-adjusted 30-day mortality.

RESULTS

CHQC identified several hospitals with consistently higher than expected mortality. The five hospitals with the highest mortality tended to lose market share (mean change -0.6%, 95% CI -1.9-0.6), but this was not significant. The only outlier hospital with a large decline in market share had declining volume for 2 years before being declared an outlier. Risk-adjusted mortality declined only slightly at hospitals classified by us as "below average" (-0.8%; 95% CI, 2.9-1.8%) or "worst" (-0.4%; 95% CI -2.3-1.7). However, risk-adjusted mortality at one hospital changed from consistently above expected to consistently below expected shortly after first being declared an outlier.

CONCLUSION

Despite CHQC's strengths, identifying hospitals with higher than expected mortality did not adversely affect their market share or, with one exception, lead to improved outcomes. This failure may have resulted from consumer disinterest or difficulty interpreting CHQC reports, unwillingness of businesses to create incentives targeted to hospitals' performance, and hospitals' inability to develop effective quality improvement programs.

Public reporting on quality in the United States and the United Kingdom

The public reporting of comparative information about health care quality is becoming an accepted way of improving accountability and quality. Quality report cards have been prominent in the United States for more than a decade and are a central feature of British health system reform. In this paper we examine the common challenges and differences in implementation of the policy in the two countries. We use this information to explore some key questions relating to the content, target audience, and use of published information. We end by making specific recommendations for maximizing the effectiveness of public reporting.

"Quality profiling" for complementary medicine: the example of a hospital for traditional Chinese medicine

OBJECTIVE

The goal of the methodological approach of "quality profiling" for complementary and alternative medicine (CAM) is to offer an empirical database that would enable different participants in the health care system to evaluate the quality of a medical provider.

METHODS

Quality profiling is a structured way of describing quality on the levels of infra-structure, patients, medical interventions, outcomes, and quality assurance related to one specific provider. As part of a program called "quality management and research," this type of profiling constitutes one basic step for generating knowledge in terms of evidence-based medicine as well as confidence-based medicine. Quality profiling is exemplified by a hospital for Traditional Chinese Medicine in Germany. Within 1 year all in-patients were included in the database using questionnaires for physicians and patients at the time of admission, discharge from the hospital, and follow-up inquiries at intervals up to 1 year after discharge. The frequency of diagnostic and therapeutic interventions was recorded daily.

RESULTS

Data for 1036 patients (mean age 53 years old, 73% female) were analyzed. The most frequent diagnostic categories were musculoskeletal disorders (30%) and neurologic disorders (26%). Therapeutic effects were shown in various outcome measures such as reduced intensity of complaints, improved quality of life, increased satisfaction in lifestyle areas, and fewer days off work. In 6.5% of the subjects, adverse events (mostly of minor severity) were recorded.

CONCLUSIONS

Quality profiles can serve as a basic tool for evaluating provider quality when the results are compared with either a predefined standard or with profiles of other providers who are offering similar medical services.

Report cards: relevance for nursing and patient care safety

The emergence of health care report cards in the North American environment is outlined. While it is evident that substantial activity has emerged, the majority of these initiatives excluded nursing, or use a broad indicator for nursing that may not provide meaningful representations of the quality of nursing care provided in the system and the relevance of this care to patient care safety. Given that nurses are the primary care provider in health care settings, this represents a significant gap in health care report cards. The pioneering work of the American Nurses Association (ANA) Nursing Report Card in the development and validation of report card indicators for nursing is discussed. Challenges related to data availability and data quality are identified. Potential opportunities for linking nursing practice outcomes to patient care quality and patient safety through a report card process are outlined.

Mortality trends during a program that publicly reported hospital performance

BACKGROUND

It is unclear whether publicly reporting hospitals' risk-adjusted mortality leads to improvements in outcomes.

OBJECTIVES

To examine mortality trends during a period (1991-1997) when the Cleveland Health Quality Choice program was operational.

RESEARCH DESIGN

Time series.

SUBJECTS

Medicare patients hospitalized with acute myocardial infarction (AMI; n = 10,439), congestive heart failure (CHF; n = 23,505), gastrointestinal hemorrhage (GIH; n = 11,088), chronic obstructive pulmonary disease (COPD; n = 8495), pneumonia (n = 23,719), or stroke (n = 14,293).

MEASURES

Risk-adjusted in-hospital mortality, early postdischarge mortality (between discharge and 30 days after admission), and 30-day mortality.

RESULTS

Risk-adjusted in-hospital mortality declined significantly for all conditions except stroke and GIH, with absolute declines ranging from -2.1% for COPD to -4.8% for pneumonia. However, the mortality rate in the early postdischarge period rose significantly for all conditions except COPD, with increases ranging from 1.4% for GIH to 3.8% for stroke. As a consequence, the 30-day mortality declined significantly only for CHF (absolute decline 1.4%, 95% CI, -2.5 to -0.1%) and COPD (absolute decline 1.6%, 95% CI, -2.8-0.0%). For stroke, risk-adjusted 30-day mortality actually increased by 4.3% (95% CI, 1.8-7.1%).

CONCLUSION

During Cleveland's experiment with hospital report cards, deaths shifted from in hospital to the period immediately after discharge with little or no net reduction in 30-day mortality for most conditions. Hospital profiling remains an unproven strategy for improving outcomes of care for medical conditions. Using in-hospital mortality rates to monitor trends in outcomes for hospitalized patients may lead to spurious conclusions.