Consistency in performance among primary care practitioners

Improving the reliability of physician "report cards"

BACKGROUND

Performance measures are widely used to profile primary care physicians (PCPs) but their reliability is often limited by small sample sizes. We evaluated the reliability of individual PCP profiles and whether they can be improved by combining measures into composites or by profiling practice groups.

METHODS

We performed a cross-sectional analysis of electronic health record data for patients with diabetes (DM), congestive heart failure (CHF), ischemic vascular disease (IVD), or eligible for preventive care services seen by a PCP within a large, integrated health care system between April 2009 and May 2010. We evaluated performance on 14 measures of DM care, 9 of CHF, 7 of IVD, and 4 of preventive care.

RESULTS

There were 51,771 patients observed by 163 physicians in 17 clinics. Few PCPs (0%-60%) could be profiled with 80% reliability using single process or intermediate-outcome measures. Combining measures into single-disease composites improved reliability for DM and preventive care with 74.5% and 76.7% of PCPs having sufficient panel sizes, but composites remained unreliable for CHF and IVD. A total of 85.3% of PCPs could be reliably profiled using a single overall composite. Aggregating PCPs into practice groups (3 to 21 PCPs per group) did not improve reliability in most cases because of little between-group practice variation.

CONCLUSIONS

Single measures rarely differentiate between individual PCPs or groups of PCPs reliably. Combining measures into single-disease or multidisease composites can improve reliability for some common conditions, but not all. Assessing PCP practice groups within a single health care system, rather than individual PCPs, did not substantially improve reliability.

Correlation of neonatal intensive care unit performance across multiple measures of quality of care

OBJECTIVES

To examine whether high performance on one measure of quality is associated with high performance on others and to develop a data-driven explanatory model of neonatal intensive care unit (NICU) performance.

DESIGN

We conducted a cross-sectional data analysis of a statewide perinatal care database. Risk-adjusted NICU ranks were computed for each of 8 measures of quality selected based on expert input. Correlations across measures were tested using the Pearson correlation coefficient. Exploratory factor analysis was used to determine whether underlying factors were driving the correlations.

SETTING

Twenty-two regional NICUs in California.

PATIENTS

In total, 5445 very low-birth-weight infants cared for between January 1, 2004, and December 31, 2007.

MAIN OUTCOMES MEASURES

Pneumothorax, growth velocity, health care-associated infection, antenatal corticosteroid use, hypothermia during the first hour of life, chronic lung disease, mortality in the NICU, and discharge on any human breast milk.

RESULTS

The NICUs varied substantially in their clinical performance across measures of quality. Of 28 unit-level correlations, 6 were significant (ρ < .05). Correlations between pairs of measures of quality of care were strong (ρ ≥ .5) for 1 pair, moderate (range, ρ ≥ .3 to ρ < .5) for 8 pairs, weak (range, ρ ≥ .1 to ρ < .3) for 5 pairs, and negligible (ρ < .1) for 14 pairs. Exploratory factor analysis revealed 4 underlying factors of quality in this sample. Pneumothorax, mortality in the NICU, and antenatal corticosteroid use loaded on factor 1; growth velocity and health care-associated infection loaded on factor 2; chronic lung disease loaded on factor 3; and discharge on any human breast milk loaded on factor 4.

CONCLUSION

In this sample, the ability of individual measures of quality to explain overall quality of neonatal intensive care was modest.

The comprehensive care project: measuring physician performance in ambulatory practice

OBJECTIVE

To investigate the feasibility, reliability, and validity of comprehensively assessing physician-level performance in ambulatory practice.

DATA SOURCES/STUDY SETTING

Ambulatory-based general internists in 13 states participated in the assessment.

STUDY DESIGN

We assessed physician-level performance, adjusted for patient factors, on 46 individual measures, an overall composite measure, and composite measures for chronic, acute, and preventive care. Between- versus within-physician variation was quantified by intraclass correlation coefficients (ICC). External validity was assessed by correlating performance on a certification exam.

DATA COLLECTION/EXTRACTION METHODS

Medical records for 236 physicians were audited for seven chronic and four acute care conditions, and six age- and gender-appropriate preventive services.

PRINCIPAL FINDINGS

Performance on the individual and composite measures varied substantially within (range 5-86 percent compliance on 46 measures) and between physicians (ICC range 0.12-0.88). Reliabilities for the composite measures were robust: 0.88 for chronic care and 0.87 for preventive services. Higher certification exam scores were associated with better performance on the overall (r = 0.19; p<.01), chronic care (r = 0.14, p = .04), and preventive services composites (r = 0.17, p = .01).

CONCLUSIONS

Our results suggest that reliable and valid comprehensive assessment of the quality of chronic and preventive care can be achieved by creating composite measures and by sampling feasible numbers of patients for each condition.

Relationship of primary care physicians' patient caseload with measurement of quality and cost performance

CONTEXT

Sufficient numbers of patients are necessary to generate statistically reliable measurements of physicians' quality and cost performance.

OBJECTIVE

To determine whether primary care physicians in the same physician practice collectively see enough Medicare patients annually to detect meaningful differences between practices in ambulatory quality and cost measures.

DESIGN, SETTING, AND PATIENTS

Primary care physicians in the United States were linked to their physician practices using the Healthcare Organization Services database maintained by IMS Health. Patients who visited primary care physicians in the 2005 Medicare Part B 20% sample were used to estimate Medicare caseloads per practice. Caseloads necessary to detect 10% relative differences in costs and quality were calculated using national mean ambulatory Medicare spending, rates of mammography for women 66 to 69 years, and hemoglobin A(1c) testing for 66- to 75-year-olds with diabetes, preventable hospitalization rate, and 30-day readmission rate after discharge for congestive heart failure (CHF).

MAIN OUTCOME MEASURES

Percentage of primary care physician practices with a sufficient number of eligible patients to detect a 10% relative difference in each performance measure.

RESULTS

Primary care physician practices had annual median caseloads of 260 Medicare patients (interquartile range [IQR], 135-500), 25 women eligible for mammography (IQR, 10-50), 30 patients with diabetes eligible for hemoglobin A(1c) testing (IQR, 15-55), and 0 patients hospitalized for CHF. For ambulatory costs, mammography rate, and hemoglobin A(1c) testing rate, the percentage of primary care physician practices with sufficient caseloads to detect 10% relative differences in performance ranged from less than 10% of practices with fewer than 11 primary care physicians to 100% of practices with more than 50 primary care physicians. None of the primary care physician practices had sufficient caseloads to detect 10% relative differences in preventable hospitalization or 30-day readmission after discharge for CHF.

CONCLUSION

Relatively few primary care physician practices are large enough to reliably measure 10% relative differences in common measures of quality and cost performance among fee-for-service Medicare patients.

Correlations among measures of quality in HIV care in the United States: cross sectional study

OBJECTIVE

To determine whether a selected set of indicators can represent a single overall quality construct.

DESIGN

Cross sectional study of data abstracted during an evaluation of an initiative to improve quality of care for people with HIV.

SETTING

69 sites in 30 states.

DATA SOURCES

Medical records of 9020 patients.

MAIN OUTCOME MEASURES

Adjusted performance rates at site level for eight measures of quality of care specific to HIV and a site level summary performance score (the number of measures for which the site was in the top quarter of the distribution).

RESULTS

Of 28 site level correlations between measures, two were greater than 0.40, two were between 0.30 and 0.39, four were between 0.20 and 0.29, and the 20 remaining were all less than 0.20. One site was in the top quarter for seven measures, but no sites were in the top quarter for six or eight of the measures. Across the eight quality measures, sites were in the top quarter no more often than predicted by a chance (binomial) distribution.

CONCLUSIONS

The quality suggested by one measured indicator cannot necessarily be generalised to unmeasured indicators, even if this might be expected for clinical or other reasons.

Organizational characteristics and preventive service delivery in private practices: a peek inside the "black box" of private practices caring for children

OBJECTIVE

Although privately owned practices provide the majority of primary care for children, little is known about the organizational characteristics of these practices or how these characteristics affect the quality of care for children. The purpose of this study was to describe selected organizational characteristics and preventive service delivery features that might affect the quality of primary care for children in private practices.

METHODS

A cross-sectional study of 44 private pediatric and family medicine practices in 2 regions of North Carolina was performed. Preventive service performance was assessed through chart abstraction for 60 randomly selected children between 24 and 30 months of age, for evaluation of immunizations and anemia, tuberculosis, and lead screening delivery by 2 years of age. Organizational characteristics were determined through surveys of all physicians and staff members. We used descriptive statistics and scatter plots to describe variations in organizational characteristics and preventive services.

RESULTS

Overall, practices demonstrated low levels of preventive service performance, with substantial variation among practices. Only 39% of children received 3 of the 4 recommended preventive services measured (practice range: 2-88%). Few practices demonstrated evidence of a systematic approach to prevention. For example, only 12 (27%) of the 44 practices used >1 of 5 recommended preventive service delivery strategies. Furthermore, practices varied greatly with respect to many of the measured organizational characteristics, which were consistent with organizational stress in some cases. For example, turnover of clinicians and staff members was remarkably high, with practices losing an average of 27% of their clinicians every 4 years (range: 0-170%) and 39% of their office staff members every 2 years (range: 0-170%).

CONCLUSIONS

Private practices caring for children in North Carolina demonstrated low overall performance for the 4 recommended preventive services examined, with large variations among practices. Few practices had evidence of comprehensive systems for prevention. There was also evidence of substantial variation in many organizational characteristics. Some organizational characteristics were at levels that might impede delivery of high-quality primary care for children. These findings suggest a growing need for research that examines the impact of organizational characteristics on the quality of care in private practices.

Assessment of patient management skills and clinical skills of practising doctors using computer-based case simulations and standardised patients

CONTEXT

Standardised assessments of practising doctors are receiving growing support, but theoretical and logistical issues pose serious obstacles.

OBJECTIVES

To obtain reference performance levels from experienced doctors on computer-based case simulation (CCS) and standardised patient-based (SP) methods, and to evaluate the utility of these methods in diagnostic assessment.

SETTING AND PARTICIPANTS

The study was carried out at a military tertiary care facility and involved 54 residents and credentialed staff from the emergency medicine, general surgery and internal medicine departments.

MAIN OUTCOME MEASURES

Doctors completed 8 CCS and 8 SP cases targeted at doctors entering the profession. Standardised patient performances were compared to archived Year 4 medical student data.

RESULTS

While staff doctors and residents performed well on both CCS and SP cases, a wide range of scores was exhibited on all cases. There were no significant differences between the scores of participants from differing specialties or of varying experience. Among participants who completed both CCS and SP testing (n = 44), a moderate positive correlation between CCS and SP checklist scores was observed. There was a negative correlation between doctor experience and SP checklist scores. Whereas the time students spent with SPs varied little with clinical task, doctors appeared to spend more time on communication/counselling cases than on cases involving acute/chronic medical problems.

CONCLUSION

Computer-based case simulations and standardised patient-based assessments may be useful as part of a multimodal programme to evaluate practising doctors. Additional study is needed on SP standard setting and scoring methods. Establishing empirical likelihoods for a range of performances on assessments of this character should receive priority.

Quantifying the physician contribution to managed care pharmacy expenses: a random effects approach

BACKGROUND

Despite the availability of more sophisticated techniques, few alternatives to ordinary least squares (OLS) regression have been utilized to profile physician prescribing in managed care. It is not known to what extent the modest R values derived from OLS models reflect incomplete risk adjustment or widely varying physician prescribing patterns.

OBJECTIVES

To quantify the role of interphysician variability relative to overall variability in managed care pharmacy expenses, and to examine the extent to which different statistical approaches generate meaningful differences in profile results.

RESEARCH DESIGN

Comparison of three basic statistical modeling approaches: OLS, fixed effects regression, and random effects (ie, hierarchical) regression models.

SETTING

Two managed care populations that differed more than 2-fold in per member pharmacy expenditures in 1999, one from the Midwestern United States, the other from three Western States.

MAIN OUTCOME MEASURES

The intraclass correlation coefficient (ICC, the proportion of variability in expenses attributable to differences among physicians) and the range of projected expenses attributed to each physician's prescribing style.

RESULTS

The ICCs were small for aggregated pharmacy expenditures, 0.04 or less in both populations. As determined by OLS, the most costly physician contributed 94,399 U.S. dollars in excess expenses to the organization whereas the most parsimonious saved 89,940 U.S. dollars. When derived from random effects models, the range in performance was 63% of that derived from OLS.

CONCLUSIONS

In the populations studied, systematic prescribing differences among physicians were small relative to the overall variability in pharmacy expenses, suggesting other factors were more likely driving these costs. Random effects models generated smaller estimates of the individual physicians' contribution to costs, sometimes considerably, relative to those derived from OLS and fixed effects approaches.

Developing a reliable, valid, and feasible plan for quality-of-care measurement for cancer: how should we measure?

BACKGROUND

Recent changes in the US health care delivery system have raised expectations that the medical marketplace will compete on quality and cost of care. This effort will require a systematic evaluation of the measurement of quality of care as it applies to cancer and other critical conditions.

OBJECTIVES

To articulate the components of the design of quality-of-care measurement systems that must be considered and optimally manipulated to generate feasible, reliable, and valid data pertinent to patients with cancer.

RESEARCH DESIGN

A synthesis of information obtained from literature reviews and experience.

MEASURES

Four key areas of design that influence quality-of-care measurement scores are discussed: case identification, data source, data-collection strategies, and the quality of the care-measurement model.

RESULTS

Challenges associated with these design and measurement strategies are defined and discussed.

CONCLUSIONS

Policy analyses vary as a function of measurement domains. The design of a quality-of-care measurement system should consider trade-offs between validity and burden by considering the intricate relations between domains of measurement.

Inconsistent report cards: assessing the comparability of various measures of the quality of ambulatory care

BACKGROUND

Report cards based on various performance measures have become increasingly common for rating hospitals and health care plans. However, little has been done to create report cards at the ambulatory clinic level, nor has there been much comparison of the potential components of report cards.

OBJECTIVES

To create a report card for ambulatory clinics based on different data collection methods and to assess the correlations of clinic scores across various domains of quality.

RESEARCH DESIGN

Cross-sectional chart review (n = 3614), patient (n = 2180), and physician surveys (n = 169).

SUBJECTS

Sample of outpatients ages 20 to 75 and their primary care providers in 11 ambulatory clinic sites in the Boston-area from May 1996 to June 1997.

MEASURES

Performance on various quality indicators for each site.

RESULTS

Report card scores for five quality domains (performance on HEDIS-like measures, clinic function, patient satisfaction, diabetes guideline compliance, asthma guideline compliance) were created for each site. None of the five domain scores were significantly correlated with any of the other domains. In addition, there was substantial intraclinic variation in domain scores when compared with the corresponding mean domain score across all clinics. Additional clinic domain scores were created by limiting measures to those found on chart review or survey alone. The chart review and survey domain scores for each clinic were also not significantly correlated.

CONCLUSIONS

Report cards that emphasize only one domain of quality or use limited data collection methods may provide incomplete or inconsistent information to health care consumers about the overall quality of an outpatient clinic.

Relationship between asthma medication and antibiotic use

STUDY OBJECTIVES

Increasing morbidity due to asthma and antimicrobial resistance among human pathogens are both major public-health concerns. Numerous studies describe the overuse of antibiotics in general populations and underuse of anti-inflammatory medications by asthmatic patients. However, little is known about the relationship between asthma medication and antibiotic use in asthmatics. Specifically, we tested the hypothesis that higher use of bronchodilator and anti-inflammatory medication by asthmatics, as a marker of problematic asthma, is associated with greater antibiotic use. We also test the hypothesis that physicians who are low prescribers of anti-inflammatory medications are high prescribers of antibiotics.

DESIGN

We conducted a retrospective cohort study evaluating asthma medication and antibiotic use by children and adults with asthma and the prescribing of these medications by primary-care physicians.

SETTING/PATIENTS

Subjects were continuously enrolled asthma patients aged 6 to 55 years receiving care in an urban, group-model, health maintenance organization.

INTERVENTIONS

None.

MEASUREMENT AND RESULTS

Main outcome measures were (1) antibiotic use by asthmatics stratified by low, moderate, and high bronchodilator use; (2) antibiotic use by asthmatics stratified by no, intermittent, and long-term anti-inflammatory use; and (3) correlation between physician-level anti-inflammatory agent to bronchodilator ratio (AIF:BD) and their rate of antibiotic prescribing. We found that (1) high bronchodilator users received 1.72 antibiotics per person-year (95% confidence interval [CI], 1.62 to 1.83), whereas low bronchodilator users received 1.23 antibiotics per person-year (95% CI, 1.19 to 1.27; p < 0.0001); (2) long-term users of anti-inflammatory agents received 1.85 antibiotics per person-year (95% CI, 1.76 to 1.95), whereas those not receiving an anti-inflammatory agent received 0.95 antibiotics per person-year (95% CI, 0.90 to 1.00; p < 0.0001); and (3) despite variations in physician AIF:BDs and antibiotic prescribing, respectively, these measures were not correlated.

CONCLUSIONS

Antibiotic use and asthma medication use are positively associated in a cohort of asthma patients. Greater effort is needed to define the appropriate role of antibiotics in asthma management.

Methodologic challenges in developing and implementing measures of quality for child health care

OBJECTIVE

To review the major building blocks in measurement of quality for child health care, with recommendations for future research.

METHODS

We describe a framework of building blocks for quality measurement and discuss how an investigator's choices for each component are constrained by the special features of child health care.

RESULTS

Methodologic challenges for children's health care include developmental change and dependency on others, fragmentary care and inadequate health care data, unusual care settings, potential for long-term consequences, proxy reporting of outcomes and patient experience, small sample sizes, and lack of evidence that links processes and outcomes of care and of methods for risk adjustment. We cite examples of child-specific measures of quality that illustrate solutions to these challenges.

CONCLUSIONS

Children are different from adults, and measures of health care quality for children must differ from those for adults. We suggest future research on measures of quality directed toward overcoming the methodologic problems specific to child health care.

Some opinions on the future of evidence-based medicine

Clinicians who introduced evidence-based medicine (EBM) argued for an approach to medical practice and teaching based on knowledge of the evidence upon which practice is founded and on the strength of the evidence itself. This article reviews the impetus for the design and growth of EBM, defines and examines the paradigm of EBM, and offers some opinions on the future of EBM in critical care.

Quality improvement among primary care practitioners: an overall appraisal of results of the Ambulatory Care Medical Audit Demonstration Project

OBJECTIVES

The authors appraised the overall conclusions of a randomized, controlled trial of quality assurance in 16 primary care group practices, addressing the relevance of findings to health maintenance organizations in the 1990s.

METHODS

The framework was the analogy of opening the "black box" of quality assurance interventions to examine circumstances in which interventions worked.

RESULTS

External pressures for quality improvement were weak during the study and knowledge of continuous quality improvement principles lacking. Correspondingly, within study practices, pre-existing mechanisms lacked the rigorous data-driven approach and system focus of the quality assurance cycles conducted. Additional barriers to demonstrating an effect of quality assurance included pre-existing good performance, high variability in performance measurements, and lack of time within the study for radical re-design of systems. Improvement in performance for one guideline was impeded by change of practice recommendations during the study. Nevertheless, clinically and statistically significant improvements in quality were obtained in five of the seven remaining guidelines, with effects peaking after feedback of performance results. A sixth guideline showed improvement in practices in which the physician leader influenced colleagues to improve. The seventh guideline showed improvement that did not reach statistical significance, in part because of lack of statistical power.

CONCLUSIONS

This study demonstrated the effectiveness of cycles of quality measurement and improvement. The findings provide guidance for health-care practitioners and managers of the 1990s, for whom quality measurement and improvement has become a priority.

Issues of variability and bias affecting multisite measurement of quality of care

OBJECTIVES

Using data from a randomized trial to improve the quality of ambulatory care, the authors quantify the various sources of variability and bias that affect measures of quality of care and suggest experimental designs and analyses that reduce both bias and variability.

METHODS

There is a growing desire among health care researchers and government agencies to profile and compare practitioner performance. Such efforts are complicated by extreme inherent variability in most measures of quality of care, as well as potential biases introduced by "experiments," where patients cannot act as the unit of randomization. When the authors measured practitioner performance for eight patient-care guidelines, they found little association of level of performance across guidelines. Thus, the authors considered performance for each guideline separately, also taking into account variability between patients, practitioners, and practice conditions.

RESULTS

Randomization can reduce bias in large studies but should be supplemented by multivariate models. A preintervention and postintervention design can reduce variability, but much of the variability that remains is because of unmeasured patient/error variance.

CONCLUSIONS

Incorporation of these concepts into future studies using quality measurements will help researchers design smaller and more sensitive trials to draw more accurate and precise conclusions.