Focus on quality: profiling physicians' practice patterns

Detecting and Visualizing Outliers in Provider Profiling Using Funnel Plots and Mixed Effects Models-An Example from Prescription Claims Data

When prescribing a drug for a patient, a physician also has to consider economic aspects. We were interested in the feasibility and validity of profiling based on funnel plots and mixed effect models for the surveillance of German ambulatory care physicians’ prescribing. We analyzed prescriptions issued to patients with a health insurance card attending neurologists’ and psychiatrists’ ambulatory practices in the German federal state of Saarland. The German National Association of Statutory Health Insurance Physicians developed a prescribing assessment scheme (PAS) which contains a systematic appraisal of the benefit of drugs for so far 12 different indications. The drugs have been classified on the basis of their clinical evidence as “standard”, “reserve” or “third level” medication. We had 152.583 prescriptions in 56 practices available for analysis. A total of 38.796 patients received these prescriptions. The funnel plot approach with additive correction for overdispersion was almost equivalent to a mixed effects model which directly took the multilevel structure of the data into account. In the first case three practices were labeled as outliers, the mixed effects model resulted in two outliers. We suggest that both techniques should be routinely applied within a surveillance system of prescription claims data.

Administrative Databases in Orthopaedic Research: Pearls and Pitfalls of Big Data

The drive for evidence-based decision-making has highlighted the shortcomings of traditional orthopaedic literature. Although high-quality, prospective, randomized studies in surgery are the benchmark in orthopaedic literature, they are often limited by size, scope, cost, time, and ethical concerns and may not be generalizable to larger populations. Given these restrictions, there is a growing trend toward the use of large administrative databases to investigate orthopaedic outcomes. These datasets afford the opportunity to identify a large numbers of patients across a broad spectrum of comorbidities, providing information regarding disparities in care and outcomes, preoperative risk stratification parameters for perioperative morbidity and mortality, and national epidemiologic rates and trends. Although there is power in these databases in terms of their impact, potential problems include administrative data that are at risk of clerical inaccuracies, recording bias secondary to financial incentives, temporal changes in billing codes, a lack of numerous clinically relevant variables and orthopaedic-specific outcomes, and the absolute requirement of an experienced epidemiologist and/or statistician when evaluating results and controlling for confounders. Despite these drawbacks, administrative database studies are fundamental and powerful tools in assessing outcomes on a national scale and will likely be of substantial assistance in the future of orthopaedic research.

If you did not document it, it did not happen: rates of documentation of discussion of infertility risk in adolescent and young adult oncology patients' medical records

PURPOSE

The adolescent and young adult (AYA) population is underserved because of unique late-effect issues, particularly future fertility. This study sought to establish rates of documentation of discussion of risk of infertility, fertility preservation (FP) options, and referrals to fertility specialists in AYA patients' medical records at four cancer centers.

METHODS

All centers reviewed randomized records within the top four AYA disease sites (breast, leukemia/lymphoma, sarcoma, and testicular). Eligible records included those of patients: diagnosed in 2011, with no prior receipt of gonadotoxic therapy; age 18 to 45 years; with no multiple primary cancers; and for whom record was not second opinion. Quality Oncology Practice Initiative methods were used to evaluate documentation of discussion of risk of infertility, discussion of FP options, and referral to a fertility specialist.

RESULTS

Of 231 records, 26% documented infertility risk discussion, 24% documented FP option discussion, and 13% documented referral to a fertility specialist. Records were less likely to contain evidence of infertility risk and FP option discussions for female patients (P = .030 and .004, respectively) and those with breast cancer (P = .021 and < .001, respectively). Records for Hispanic/Latino patients were less likely to contain evidence of infertility risk discussion (P = .037). Records were less likely to document infertility risk discussion, FP option discussion, and fertility specialist referral for patients age ≥ 40 years (P < .001, < .001, and .002, respectively) and those who already had children (all P < .001).

CONCLUSION

The overall rate of documentation of discussion of FP is low, and results show disparities among specific groups. Although greater numbers of discussions may be occurring, there is a need to create interventions to improve documentation.

Physician practice behavior and practice guidelines: using unannounced standardized patients to gather data

BACKGROUND

Measuring actual practice behaviors of physicians, particularly as they relate to established clinical guidelines, is challenging. Standardized patients provide one method of collecting such data.

OBJECTIVE

To demonstrate the use of unannounced standardized patients in gathering data that may address adherence to guidelines in an office setting.

DESIGN

Unannounced standardized patients (SPs) simulating an initial type 2 diabetic visit presented to community offices of 32 internists as "real" patients to record physicians' evaluation and management.

PARTICIPANTS

Unannounced SPs presented to the office of 32 internists as "real" patients.

MEASUREMENTS

Unannounced SPs, simulating type 2 diabetics, completed a standardized assessment sheet, based on ADA guidelines to record physicians' evaluation and management following an initial visit. Patient charts were also reviewed to determine if evaluation adhered to the guidelines.

RESULTS

Unannounced SPs recorded 56 visits with 32 community internists; all SPs remained undetected. All physicians asked SPs about medications. At least 50% of physicians asked about home blood sugar monitoring, last eye exam, smoking, edema, and told patients to stop smoking. Less than 50% of physicians asked about parasthesias, performed fundoscopy, examined feet, referred the patient to a diabetic educator or ophthalmologist, or gave patients suggestions regarding glucose monitoring or exercise. HbA1c was ordered in 78%, metabolic profiles in 86%, and urinalysis/microalbumin in 41% of patients.

CONCLUSIONS

Unannounced standardized patients can successfully collect important data regarding physician practices in community settings. This method may be helpful in assessing physician adherence to established clinical practice guidelines.

Critical look at the clinical practice guidelines for allergic rhinitis

Allergic rhinitis (AR) is a major health concern and numerous guidelines have been developed to standardize and to improve the management of this disease. As in many other areas of medicine, the methodology of the AR guidelines has evolved from opinion-based to evidence-based medicine. Although evidence-based medicine has many benefits, it also has limitations and cannot cancel the value of the individual clinical expertise. More important than the methodology of guideline development is the efficacy of guidelines to change patient and physician behaviour and to improve clinical outcomes. At present, however, studies on the effectiveness of guidelines are few. The International Consensus on Rhinitis from 1994 is the only guideline for AR that has been assessed for its effects on health outcomes. Furthermore, there is a lack of valid and reliable instruments to assess physician's and patient's attitude towards and compliance with guideline recommendations. There is no single effective way to ensure the use of guidelines into practice, but a carefully developed and multifaceted dissemination and implementation strategy and targeting and adapting guideline recommendations to the local and individual level are key elements. The final and most important step of putting guidelines into practice occurs at the level of the patient. Patients should be considered as effective partners in health care. Education of the patient and efforts to change patient's behaviour can maximize compliance, increase satisfaction and optimize health outcomes.

Validation of diagnostic codes within medical services claims

OBJECTIVES

Few studies have attempted to validate the diagnostic information contained within medical service claims data, and only a small proportion of these have attempted to do so using the medical chart as a gold standard. The goal of this study is to determine the sensitivity and specificity of medical services claims diagnoses for surveillance of 14 drug disease contraindications used in drug utilization review, the Charlson comorbidity index and the Johns Hopkins Adjusted Care Group Case-Mix profile (ADGs).

STUDY DESIGN AND SETTING

Diagnoses were abstracted from the medical charts of 14,980 patients, and were used as the "gold standard," against which diagnoses obtained from the administrative database for the same patients were compared.

RESULTS

Conditions associated with drug disease contraindications with the exception of hypertension and chronic obstructive pulmonary disease (COPD) showed a specificity of 90% or higher. Sensitivity of claims data was substantially lower, with glaucoma, hypertension, and diabetes being the most sensitive conditions at 76, 69, and 64%, respectively. Each of the 18 disease conditions contained in the Charlson comorbidity index showed high specificity, but sensitivity was more variable among conditions as well as by coding definitions. Although ADG specificity was also high, the vast majority of ADGs had sensitivities of less than 60%.

CONCLUSION

The administrative data was found to have diagnoses and conditions that were highly specific but that vary greatly by condition in terms of sensitivity. To appropriately obtain diagnostic profiles, it is recommended that data pertaining to all physician billings be used.

Physician performance assessment: nonequivalence of primary care measures

BACKGROUND

Assessment of the performance of primary care physicians requires multiple, reliable measures. This article explores the appropriateness of selected Health Plan Employer Data and Information Set (HEDIS) measures, developed to assess health plans, to assess individual physician performance.

OBJECTIVES

To determine the consistency and reliability of 4 measures of primary care physician performance measures: cancer screening, diabetic management, patient satisfaction, and ambulatory costs.

METHODS

The study population consisted of all 194 family practitioners and general internists providing ambulatory services in 1998 to a defined patient panel of 320,000 adult health maintenance organization members. Administrative data on physician practice and performance were assessed with multiple regression and analysis of variance.

RESULTS

Each performance measure was significantly related to 1 or 2 of the other measures: high cancer screening rates with good diabetic management and high patient satisfaction, good diabetic management with high cancer screening rates, high patient satisfaction with high cancer screening rates and high ambulatory costs, or high ambulatory costs with higher patient satisfaction. Although 76% of the physicians ranked in the highest third for at least 1 measure, 81% of these high performers ranked in the lower third for at least 1 other measure. Three percent of physicians ranked exclusively in the top or bottom third on all measures.

CONCLUSIONS

Care should be taken in assessing physicians based on narrow performance measures. Assessments of individual physicians with current performance measures might identify areas in which improvement is needed and to provide feedback to improve performance quality and efficiency. However, assumptions should not be made from one measure of performance to another.

Benchmarking and quality in residential and nursing homes: lessons from the US

BACKGROUND

Performance measurement and benchmarking are common concerns in the delivery of long term care. It is common to measure the performance of providers and to publicly report these data. This paper examines selected technical challenges facing those who design, implement and disseminate health care quality performance measures.

METHOD

Review of the application of measures of performance in the US nursing home sector.

RESULTS

Using examples drawn from the skilled nursing home arena, problems ranging from data reliability and validity, the multi-dimensional nature of quality measures and selection bias as well as differential measurement abilities are discussed.

CONCLUSIONS

Benchmarking of performance is an inherently complex issue. However, to ensure that such comparisons are both fair and valid requires measures to be more technically sophisticated and sensitive to real changes attributable to changes in care.

Whom should we profile? Examining diabetes care practice variation among primary care providers, provider groups, and health care facilities

OBJECTIVE

To evaluate the amount of variation in diabetes practice patterns at the primary care provider (PCP), provider group, and facility level, and to examine the reliability of diabetes care profiles constructed using electronic databases.

DATA SOURCES/STUDY SETTING

Clinical and administrative data obtained from the electronic information systems at all facilities in a Department of Veterans Affairs' (VA) integrated service network for a study period of October 1997 through September 1998.

STUDY DESIGN

This is a cohort study. The key variables of interest are different types of diabetes quality indicators, including measures of technical process, intermediate outcomes, and resource use.

DATA COLLECTION/EXTRACTION METHODS

A coordinated registry of patients with diabetes was constructed by integrating laboratory, pharmacy, utilization, and primary care provider data extracted from the local clinical information system used at all VA medical centers. The study sample consisted of 12,110 patients with diabetes, 258 PCPs, 42 provider groups, and 13 facilities.

PRINCIPAL FINDINGS

There were large differences in the amount of practice variation across levels of care and for different types of diabetes care indicators. The greatest amount of variance tended to be attributable to the facility level. For process measures, such as whether a hemoglobin A1c was measured, the facility and PCP effects were generally comparable. However, for three resource use measures the facility effect was at least six times the size of the PCP effect, and for inter-mediate outcome indicators, such as hyperlipidemia, facility effects ranged from two to sixty times the size of the PCP level effect. A somewhat larger PCP effect was found (5 percent of the variation) when we examined a "linked" process-outcome measure linking hyperlipidemia and treatment with statins). When the PCP effect is small (i.e., 2 percent), a panel of two hundred diabetes patients is needed to construct profiles with 80 percent reliability.

CONCLUSIONS

little of the variation in many currently measured diabetes care practices is attributable to PCPs and, unless panel sizes are large, PCP profiling will be inaccurate. If profiling is to improve quality, it may be best to focus on examining facility-level performance variations and on developing indicators that promote specific, high-priority clinical actions.

Identifying patient preoperative risk factors and postoperative adverse events in administrative databases: results from the Department of Veterans Affairs National Surgical Quality Improvement Program

BACKGROUND

The Department of Veterans Affairs (DVA) National Surgical Quality Improvement Program (NSQIP) employs trained nurse data collectors to prospectively gather preoperative patient characteristics and 30-day postoperative outcomes for most major operations in 123 DVA hospitals to provide risk-adjusted outcomes to centers as quality indicators. It has been suggested that routine hospital discharge abstracts contain the same information and would provide accurate and complete data at much lower cost.

STUDY DESIGN

With preoperative risks and 30-day outcomes recorded by trained data collectors as criteria standards, ICD-9-CM hospital discharge diagnosis codes in the Patient Treatment File (PTF) were tested for sensitivity and positive predictive value. ICD-9-CM codes for 61 preoperative patient characteristics and 21 postoperative adverse events were identified.

RESULTS

Moderately good ICD-9-CM matches of descriptions were found for 37 NSQIP preoperative patient characteristics (61%); good data were available from other automated sources for another 15 (25%). ICD-9-CM coding was available for only 13 (45%) of the top 29 predictor variables. In only three (23%) was sensitivity and in only four (31%) was positive predictive value greater than 0.500. There were ICD-9-CM matches for all 21 NSQIP postoperative adverse events; multiple matches were appropriate for most. Postoperative occurrence was implied in only 41%; same breadth of clinical description in only 23%. In only four (7%) was sensitivity and only two (4%) was positive predictive value greater than 0.500.

CONCLUSION

Sensitivity and positive predictive value of administrative data in comparison to NSQIP data were poor. We cannot recommend substitution of administrative data for NSQIP data methods.

Measuring compliance with preventive care guidelines: standardized patients, clinical vignettes, and the medical record

OBJECTIVE

To determine how accurately preventive care reported in the medical record reflects actual physician practice or competence.

DESIGN

Scoring criteria based on national guidelines were developed for 7 separate items of preventive care. The preventive care provided by randomly selected physicians was measured prospectively for each of the 7 items. Three measurement methods were used for comparison: (1) the abstracted medical record from a standardized patient (SP) visit; (2) explicit reports of physician practice during those visits from the SPs, who were actors trained to present undetected as patients; and (3) physician responses to written case scenarios (vignettes) identical to the SP presentations.

SETTING

The general medicine primary care clinics of two university-afflliated VA medical centers.

PARTICIPANTS

Twenty randomly selected physicians (10 at each site) from among eligible second- and third-year general internal medicine residents and attending physicians.

MEASUREMENTS AND MAIN RESULTS

Physicians saw 160 SPs (8 cases x 20 physicians). We calculated the percentage of visits in which each prevention item was recorded in the chart, determined the marginal percentage improvement of SP checklists and vignettes over chart abstraction alone, and compared the three methods using an analysis-of-variance model. We found that chart abstraction underestimated overall prevention compliance by 16% (P < .01) compared with SP checklists. Chart abstraction scores were lower than SP checklists for all seven items and lower than vignettes for four items. The marginal percentage improvement of SP checklists and vignettes to performance as measured by chart abstraction was significant for all seven prevention items and raised the overall prevention scores from 46% to 72% (P < .0001).

CONCLUSIONS

These data indicate that physicians perform more preventive care than they report in the medical record. Thus, benchmarks of preventive care by individual physicians and institutions that rely solely on the medical record may be misleading, at best.

Are clinical practice guidelines valid for primary care?

Development of evidence-based clinical practice guidelines has emphasized treatment efficacy under methodologically rigorous conditions; internal validity (i.e., efficacy) has been assessed well. Despite their experimental foundations, guidelines originally were considered physician education tools relevant to singular, idealized encounters. Now, in rushing to generalize, quantify, and regulate quality, guidelines are being applied to whole populations as quality review criteria. Clinical guidelines do not make operational how, or even make clear whether they should be so used. We studied the external validity, the generalizability and measurability for primary care, of a guideline for treatment of systolic heart failure. Patient and clinical factors reducing generalizability were examined. Imprecise definition and reliance on single measures of recommendations were studied as limits to measurability. Patient, physician, organizational, and system factors influenced guideline validity. Attention to myriad invalidity sources undermining implementation and evaluation of efforts to improve quality and outcomes of primary care is critical.

Sources of influence on medical practice

OBJECTIVES

To explore the opinion of general practitioners on the importance and legitimacy of sources of influence on medical practice.

METHODS

General practitioners (n=723) assigned to Primary Care Teams (PCTs) in two Spanish regions were randomly selected to participate in this study. A self administered questionnaire was sent by mail and collected by hand. The dependent variable collected the opinion on different sources that exert influence on medical practice. Importance was measured with a 9 item scale while legitimacy was evaluated with 16 items measured with a 1 to 7 point Likert scale.

RESULTS

The most important and legitimate sources of influence according to general practitioners were: training courses and scientific articles, designing self developed protocols and discussing with colleagues. The worst evaluated were: financial incentives and the role played by the pharmaceutical industry.

CONCLUSIONS

The development of medical practice is determined by many factors, grouped around three big areas: organisational setting, professional system and social setting. The medical professional system is the one considered as being the most important and legitimate by general practitioners. Other strategies of influence, considered to be very important by the predominant management culture (financial incentives), are not considered to be so by general practitioners. These results, however, are not completely reliable as regards the real network of influences existing in medical practice, which reflect instead different "value systems".

How well does chart abstraction measure quality? A prospective comparison of standardized patients with the medical record

PURPOSE

Despite widespread reliance on chart abstraction for quality measurement, concerns persist about its reliability and validity. We prospectively evaluated the validity of chart abstraction by directly comparing it with the gold standard of reports by standardized patients.

SUBJECTS AND METHODS

Twenty randomly selected general internal medicine residents and attending faculty physicians at the primary care clinics of two Veterans Affairs Medical Centers blindly evaluated and treated actor-patients (standardized patients) who had one of four common diseases: diabetes, chronic obstructive pulmonary disease, coronary artery disease, or low back pain. Charts from the visits were abstracted using explicit quality criteria; standardized patients completed a checklist containing the same criteria. For each physician, quality was measured for two different cases of the four conditions (a total of 160 physician-patient encounters). We compared chart abstraction with standardized-patient reports for four aspects of the encounter: taking the history, examining the patient, making the diagnosis, and prescribing appropriate treatment. The sensitivity and specificity of chart abstraction were calculated.

RESULTS

The mean (+/- SD) chart abstraction score was 54% +/- 9%, substantially less than the mean score on the standardized-patient checklist of 68% +/- 9% (P <0.001). This finding was similar for all four conditions and at both sites. "False positives"-chart-recorded necessary care actions not reported by the standardized patients-resulted in a specificity of only 81%. The overall sensitivity of chart abstraction for necessary care was only 70%.

CONCLUSIONS

Chart abstraction underestimates the quality of care for common outpatient general medical conditions when compared with standardized-patient reports. The medical record is neither sensitive nor specific. Quality measurements derived from chart abstraction may have important shortcomings, particularly as the basis for drawing policy conclusions or making management decisions.

Addressing physician compensation and practice productivity

Medical groups are challenged to adopt a systematic, evidence-based approach to selecting a physician compensation method that supports the group's overall financial and organizational strategies, including managed care contracting strategies; is consistent with the philosophies, beliefs, and attitudes of the group's membership as they pertain to individual productivity; and can be supported by the organization's information technology, decision support, and management infrastructures. This article explains how research in physician profiling, benchmarking, general compensation theory, and physician productivity provides evidence that can serve as the foundation for a pragmatic approach to evaluating physician compensation method alternatives. It also presents a unique production-based compensation model for illustrative purposes.

The future of dental managed care in the US

Managed care is slowly becoming more prominent within dental benefit programs, with nearly 30% of insured people participating in a dental managed care plan. This number is expected to increase steadily over time, but not with the same speed that it has with medicine. In combination with the changing demographics of the country and the diminishing supply of dental personnel, managed care will have less power over the dental market place than it has had over medicine. In the future, changes will be dependent on the demand for dental treatment and also the supply of dentists. With increased demand, decreased personnel and a minority of Americans having dental insurance, dentists are in a better position than their medical counterparts. If the influence of managed care is not diminished, the effects on the practice of dentistry will be dramatic. Managed care will eventually introduce outcomes assessment, utilization management, and reduced fees. This will transform the practice of dentistry into a two-tiered system, treating fee-for-service and managed care patients differently.

How valid are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patients visits

OBJECTIVES

This study was designed to determine the optimal nonobservational method of measuring the delivery of outpatient medical services.

METHODS

As part of a multimethod study of the content of primary care practice, research nurses directly observed consecutive patient visits to 138 practicing family physicians. Data on services delivered were collected using a direct observation checklist, medical record review, and patient exit questionnaires. For each medical service, the sensitivity, specificity, and Kappa statistic were calculated for medical record review and patient exit questionnaires compared with direct observation. Interrater reliability among eight research nurses was calculated using the Kappa statistic for a separate sample of videotaped visits and medical records.

RESULTS

Visits by 4,454 patients were observed. Exit questionnaires were returned by 74% of patients. Research nurse interrater reliabilities were generally high. The specificity of both the medical record and the patient exit questionnaire was high for most services. The sensitivity of the medical record was low for measuring health habit counseling and moderate for physical examination, laboratory testing, and immunization. The patient exit questionnaire showed moderate to high sensitivity for health habit counseling and immunization and variable sensitivity for physical examination and laboratory services.

CONCLUSIONS

The validity of the medical record and patient questionnaire for measuring delivery of different health services varied with the service. This report can be used to choose the optimal nonobservational method of measuring the delivery of specific ambulatory medical services for research and physician profiling and to interpret existing health services research studies using these common measures.

Managed care of substance abuse disorders. Implications for generalist physicians

Given the rapidly changing landscape of both private and public health plan programs, it is foolhardy to suggest that a definitive characterization of managed care and substance abuse issues can be made. Among other topics, this article discusses goals of managed care organizations, managed care models, and physicians' roles in managed care.

Developing a quality improvement database using health insurance data: a guided tour with application to Medicare's National Claims History file

Health policy researchers are increasingly turning to insurance claims to provide timely information on cost, utilization, and quality trends in health care markets. This research offers an in-depth description of how to systematically transform raw inpatient and ambulatory claims data into useful information for health care management and research using the Health Care Financing Administration's National Claims History file as an example. The topics covered include: (a) understanding the contents and architecture of claims data, (b) creating analytic files from raw claims, (c) technical innovations for health policy studies, (d) assessing data accuracy, (d) the costs of using claims data, and (e) ensuring confidentiality. In summary, claims data are found to have great potential for quality of care analysis. As in any analysis, careful development of a database is required for scientific research. The methods outlined in this study offer health data novices as well as experienced analysts a series of strategies to maximize the value of claims data for health policy analysis.

A computerized system for reviewing medical records from physicians' offices

BACKGROUND

Review of clinical performance in office-based care is increasing in importance as more medical care shifts to outpatient settings. Decisions made in primary care settings can save lives and limit disability through prevention and early intervention in disease. Information is needed to assess quality of care by answering such questions as whether drugs are prescribed and monitored appropriately, follow-up on serious health threats is carried out promptly, or procedures are performed for appropriate indications. Moreover, data from medical records are essential to provide important clinical information not found in the more widely used administrative data sets. Managed care organizations, too, face the need to respond to requests for objective information about the quality of primary care that they provide. Those organizations now planning assessments of primary care will need to consider the cost of obtaining information from medical record review.

METHODS

The DEMPAQ Record Review System (DRRS) is a tool for peer review organizations (PROs) to use to review ambulatory care given to Medicare beneficiaries in physicians' offices. The system is described in terms of functions (activities commonly performed in the course of an office visit, such as drug prescribing), indicators (summary measures of quality for each key function of clinical care), and clinical items (for example, specific drugs or tests for each function and indicator)--a total of 263 indicators in all. A framework is provided for measuring the operational costs of a review system based on medical records.

RESULTS

The costs directly associated with a fully operational review system were less than $48 per case in Iowa and Alabama and $72 per case in Maryland. On average, reviewers spent about an hour per case signed on to DRRS; average review times declined over time with practice. About half the cost of the review process is accounted for by administrative costs. Therefore, once the effort has been made to obtain records, the additional cost of abstracting more data items is relatively low. Since samples of 300-500 records suffice to measure average performance for a region or state, costs per region/state approximate $15,000 to $25,000 per measurement cycle for assessment of a wide array of clinical areas.

SUMMARY

The cost of collecting information on quality of care from medical records using the review system falls within current budgets for PRO review. However, organizations planning to implement a quality improvement campaign should also consider the costs of analyzing the data, reporting information to physicians, and continuing to monitor changes in performance.