What makes quality assurance effective? Results from a randomized, controlled trial in 16 primary care group practices

Identifying and understanding benefits associated with return-on-investment from large-scale healthcare Quality Improvement programmes: an integrative systematic literature review

BACKGROUND

We previously developed a Quality Improvement (QI) Return-on-Investment (ROI) conceptual framework for large-scale healthcare QI programmes. We defined ROI as any monetary or non-monetary value or benefit derived from QI. We called the framework the QI-ROI conceptual framework. The current study describes the different categories of benefits covered by this framework and explores the relationships between these benefits.

METHODS

We searched Medline, Embase, Global health, PsycInfo, EconLit, NHS EED, Web of Science, Google Scholar, organisational journals, and citations, using ROI or returns-on-investment concepts (e.g., cost-benefit, cost-effectiveness, value) combined with healthcare and QI. Our analysis was informed by Complexity Theory in view of the complexity of large QI programmes. We used Framework analysis to analyse the data using a preliminary ROI conceptual framework that was based on organisational obligations towards its stakeholders. Included articles discussed at least three organisational benefits towards these obligations, with at least one financial or patient benefit. We synthesized the different QI benefits discussed.

RESULTS

We retrieved 10 428 articles. One hundred and two (102) articles were selected for full text screening. Of these 34 were excluded and 68 included. Included articles were QI economic, effectiveness, process, and impact evaluations as well as conceptual literature. Based on these literatures, we reviewed and updated our QI-ROI conceptual framework from our first study. Our QI-ROI conceptual framework consists of four categories: 1) organisational performance, 2) organisational development, 3) external outcomes, and 4) unintended outcomes (positive and negative). We found that QI benefits are interlinked, and that ROI in large-scale QI is not merely an end-outcome; there are earlier benefits that matter to organisations that contribute to overall ROI. Organisations also found positive aspects of negative unintended consequences, such as learning from failed QI.

DISCUSSION AND CONCLUSION

Our analysis indicated that the QI-ROI conceptual framework is made-up of multi-faceted and interconnected benefits from large-scale QI programmes. One or more of these may be desirable depending on each organisation's goals and objectives, as well as stage of development. As such, it is possible for organisations to deduce incremental benefits or returns-on-investments throughout a programme lifecycle that are relevant and legitimate.

Audit and feedback: effects on professional practice and healthcare outcomes

BACKGROUND

Audit and feedback is widely used as a strategy to improve professional practice either on its own or as a component of multifaceted quality improvement interventions. This is based on the belief that healthcare professionals are prompted to modify their practice when given performance feedback showing that their clinical practice is inconsistent with a desirable target. Despite its prevalence as a quality improvement strategy, there remains uncertainty regarding both the effectiveness of audit and feedback in improving healthcare practice and the characteristics of audit and feedback that lead to greater impact.

OBJECTIVES

To assess the effects of audit and feedback on the practice of healthcare professionals and patient outcomes and to examine factors that may explain variation in the effectiveness of audit and feedback.

SEARCH METHODS

We searched the Cochrane Central Register of Controlled Trials (CENTRAL) 2010, Issue 4, part of The Cochrane Library. www.thecochranelibrary.com, including the Cochrane Effective Practice and Organisation of Care (EPOC) Group Specialised Register (searched 10 December 2010); MEDLINE, Ovid (1950 to November Week 3 2010) (searched 09 December 2010); EMBASE, Ovid (1980 to 2010 Week 48) (searched 09 December 2010); CINAHL, Ebsco (1981 to present) (searched 10 December 2010); Science Citation Index and Social Sciences Citation Index, ISI Web of Science (1975 to present) (searched 12-15 September 2011).

SELECTION CRITERIA

Randomised trials of audit and feedback (defined as a summary of clinical performance over a specified period of time) that reported objectively measured health professional practice or patient outcomes. In the case of multifaceted interventions, only trials in which audit and feedback was considered the core, essential aspect of at least one intervention arm were included.

DATA COLLECTION AND ANALYSIS

All data were abstracted by two independent review authors. For the primary outcome(s) in each study, we calculated the median absolute risk difference (RD) (adjusted for baseline performance) of compliance with desired practice compliance for dichotomous outcomes and the median percent change relative to the control group for continuous outcomes. Across studies the median effect size was weighted by number of health professionals involved in each study. We investigated the following factors as possible explanations for the variation in the effectiveness of interventions across comparisons: format of feedback, source of feedback, frequency of feedback, instructions for improvement, direction of change required, baseline performance, profession of recipient, and risk of bias within the trial itself. We also conducted exploratory analyses to assess the role of context and the targeted clinical behaviour. Quantitative (meta-regression), visual, and qualitative analyses were undertaken to examine variation in effect size related to these factors.

MAIN RESULTS

We included and analysed 140 studies for this review. In the main analyses, a total of 108 comparisons from 70 studies compared any intervention in which audit and feedback was a core, essential component to usual care and evaluated effects on professional practice. After excluding studies at high risk of bias, there were 82 comparisons from 49 studies featuring dichotomous outcomes, and the weighted median adjusted RD was a 4.3% (interquartile range (IQR) 0.5% to 16%) absolute increase in healthcare professionals' compliance with desired practice. Across 26 comparisons from 21 studies with continuous outcomes, the weighted median adjusted percent change relative to control was 1.3% (IQR = 1.3% to 28.9%). For patient outcomes, the weighted median RD was -0.4% (IQR -1.3% to 1.6%) for 12 comparisons from six studies reporting dichotomous outcomes and the weighted median percentage change was 17% (IQR 1.5% to 17%) for eight comparisons from five studies reporting continuous outcomes. Multivariable meta-regression indicated that feedback may be more effective when baseline performance is low, the source is a supervisor or colleague, it is provided more than once, it is delivered in both verbal and written formats, and when it includes both explicit targets and an action plan. In addition, the effect size varied based on the clinical behaviour targeted by the intervention.

AUTHORS' CONCLUSIONS

Audit and feedback generally leads to small but potentially important improvements in professional practice. The effectiveness of audit and feedback seems to depend on baseline performance and how the feedback is provided. Future studies of audit and feedback should directly compare different ways of providing feedback.

Systematic review and meta-analysis of practice facilitation within primary care settings

PURPOSE

This study was a systematic review with a quantitative synthesis of the literature examining the overall effect size of practice facilitation and possible moderating factors. The primary outcome was the change in evidence-based practice behavior calculated as a standardized mean difference.

METHODS

In this systematic review, we searched 4 electronic databases and the reference lists of published literature reviews to find practice facilitation studies that identified evidence-based guideline implementation within primary care practices as the outcome. We included randomized and nonrandomized controlled trials and prospective cohort studies published from 1966 to December 2010 in English language only peer-reviewed journals. Reviews of each study were conducted and assessed for quality; data were abstracted, and standardized mean difference estimates and 95% confidence intervals (CIs) were calculated using a random-effects model. Publication bias, influence, subgroup, and meta-regression analyses were also conducted.

RESULTS

Twenty-three studies contributed to the analysis for a total of 1,398 participating practices: 697 practice facilitation intervention and 701 control group practices. The degree of variability between studies was consistent with what would be expected to occur by chance alone (I2 = 20%). An overall effect size of 0.56 (95% CI, 0.43-0.68) favored practice facilitation (z = 8.76; P <.001), and publication bias was evident. Primary care practices are 2.76 (95% CI, 2.18-3.43) times more likely to adopt evidence-based guidelines through practice facilitation. Meta-regression analysis indicated that tailoring (P = .05), the intensity of the intervention (P = .03), and the number of intervention practices per facilitator (P = .004) modified evidence-based guideline adoption.

CONCLUSION

Practice facilitation has a moderately robust effect on evidence-based guideline adoption within primary care. Implementation fidelity factors, such as tailoring, the number of practices per facilitator, and the intensity of the intervention, have important resource implications.

Implementing guidelines into clinical practice: what is the value?

RATIONALE AND OBJECTIVE

In budget-constrained health systems, decision makers need to consider both the costs and effects of introducing and actively implementing clinical guidance. We aim to demonstrate how, as an alternative to conventional methods, a total net benefit approach to economic evaluation can be used to inform decision making about guidelines and specific implementation strategies, like education or financial incentives.

METHODS

Aside from providing more detail on the decision framework, we describe how to collect and analyse the relevant data for calculating the total net benefit of guideline use and the value of implementation. We illustrate the process of decision analysis for a stylized example on improving diabetes care in the UK. For the analysis, economic evidence on intensified glycemic control and that on audit and feedback to promote control is combined with information on diabetes practice.

RESULTS

Our illustration demonstrates that the total net benefit of guideline use and the value of implementation can vary substantially, depending on the clinical intervention chosen, the health system being studied and the specific implementation strategies. This also holds for the threshold value for cost-effectiveness, the duration of guideline usage or validity, the size of the patient population served, and the trends and ceiling rates in the implementation of clinical guidance.

CONCLUSIONS

In comparison with conventional methods for health economic evaluation, a total net benefit approach allows for the explicit consideration of the current (or future) use of guidelines or guideline recommendations, the cost of implementation and the scope of clinical practice. Decisions made on the basis of the total net benefit of all plausible combinations of clinical guidance and implementation strategies provide optimal patient care and an efficient use of resources.

Modeling the value for money of changing clinical practice change: a stochastic application in diabetes care

BACKGROUND

Decision making about resource allocation for guideline implementation to change clinical practice is inevitably undertaken in a context of uncertainty surrounding the cost-effectiveness of both clinical guidelines and implementation strategies. Adopting a total net benefit approach, a model was recently developed to overcome problems with the use of combined ratio statistics when analyzing decision uncertainty.

OBJECTIVE

To demonstrate the stochastic application of the model for informing decision making about the adoption of an audit and feedback strategy for implementing a guideline recommending intensive blood glucose control in type 2 diabetes in primary care in the Netherlands.

METHODS

An integrated Bayesian approach to decision modeling and evidence synthesis is adopted, using Markov Chain Monte Carlo simulation in WinBUGs. Data on model parameters is gathered from various sources, with effectiveness of implementation being estimated using pooled, random-effects meta-analysis. Decision uncertainty is illustrated using cost-effectiveness acceptability curves and frontier.

RESULTS

Decisions about whether to adopt intensified glycemic control and whether to adopt audit and feedback alter for the maximum values that decision makers are willing to pay for health gain. Through simultaneously incorporating uncertain economic evidence on both guidance and implementation strategy, the cost-effectiveness acceptability curves and cost-effectiveness acceptability frontier show an increase in decision uncertainty concerning guideline implementation.

CONCLUSIONS

The stochastic application in diabetes care demonstrates that the model provides a simple and useful tool for quantifying and exploring the (combined) uncertainty associated with decision making about adopting guidelines and implementation strategies and, therefore, for informing decisions about efficient resource allocation to change clinical practice.

Implementing clinical protocols in oncology: quality gaps and the learning curve phenomenon

BACKGROUND

The quality improvement effort in clinical practice has focused mostly on 'performance quality', i.e. on the development of comprehensive, evidence-based guidelines. This study aimed to assess the 'conformance quality', i.e. the extent to which guidelines once developed are correctly and consistently applied. It also aimed to assess the existence of quality gaps in the treatment of certain patient segments as defined by age or gender and to investigate methods to improve overall conformance quality.

METHODS

A retrospective audit of clinical practice in a well-defined oncology setting was undertaken and the results compared to those obtained from prospectively applying an internally developed clinical protocol in the same setting and using specific tools to increase conformance quality.

RESULTS

All indicators showed improvement after the implementation of the protocol that in many cases reached statistical significance, while in the entire cohort advanced age was associated (although not significantly) with sub-optimal delivery of care. A 'learning curve' phenomenon in the implementation of quality initiatives was detected, with all indicators improving substantially in the second part of the prospective study.

CONCLUSIONS

Clinicians should pay separate attention to the implementation of chosen protocols and employ specific tools to increase conformance quality in patient care.

Electronic health record systems: the vehicle for implementing performance measures

Advances in information technology and recent national directives have the potential to support dramatic improvements in health care. Two key components are the implementation of functional electronic health record systems and widely accepted, evidence-based clinical performance measures for physicians. Midwest Heart Specialists, a 55-physician cardiovascular group at 14 locations in northern Illinois, has utilized an outpatient electronic health record system since 1997. Since 2003, the group has integrated cardiovascular measurement sets developed by the American Medical Association-convened Physician Consortium for Performance Improvement into its electronic health record system. With this integration, the group was able to capture data needed for internal quality assessment and improvement as part of routine outpatient care without the need for additional resources. Critical disease-management data for decision support are available continuously, resulting in improvements in health care. The reporting of these standardized data could be the foundation to support quality-based reimbursement strategies and physician office-based disease-management strategies.

Results of a collaborative quality improvement program on outcomes and costs in a tertiary critical care unit

OBJECTIVE

To demonstrate that by using the knowledge and skills of the primary care provider and by applying statistical and scientific principles of quality improvement, outcomes can be improved and costs significantly reduced.

DESIGN

A before and after quasi-experimentally designed trial using historical controls plus an analysis of costs in areas not influenced by intensive care unit (ICU) practice to control for possible secular changes.

SETTING

A tertiary ICU.

PATIENTS

All patients admitted to the above-mentioned ICU from January 1, 1991, through December 31, 1995.

INTERVENTIONS

a) A focused program that applied statistical and scientific quality improvement processes to the practice of intensive care. b) An organized effort to modify the culture, thinking, and behavior of the personnel who practice in the ICU.

MEASUREMENTS

Severity of illness, ICU and hospital lengths of stay, ICU and hospital mortality rates, total hospital costs as analyzed by the cost center, and measures of improvement in specific areas of care.

MAIN RESULTS

Significant improvement in glucose control, use of enteral feeding, antibiotic use, adult respiratory distress syndrome survival, laboratory use, blood gases use, radiograph use, and appropriate use of sedation. A severity adjusted total hospital cost reduction of $2,580,981 in 1991 dollars when comparing 1995 with the control year of 1991, with 87% of the reduction in those cost centers directly influenced by the intervention.

CONCLUSIONS

A focused quality improvement program in the ICU can have a beneficial impact on care and simultaneously reduce costs.

Trauma emergency unit: long-term evaluation of a quality assurance programme

OBJECTIVE

Long-term evaluation of a quality assurance programme (after an assessment in 1993).

DESIGN

Review of medical records.

SETTING

Emergency area of an orthopaedic, trauma, and plastic surgery unit in a French teaching hospital (Besancon).

SUBJECTS

1187 consecutive ambulatory patients' records, from July 1995.

MAIN MEASURES

Occurrence of near adverse events (at risk events causing situations which could lead to the occurrence of an adverse event).

RESULTS

71 near adverse events were identified (5.9% of the ambulatory visits). There was a significant decrease in the rate of near adverse events between 1993 (9.9% (2056 ambulatory visits, 204 near adverse events)), and 1995 (5.9% (1187 ambulatory visits, 71 near adverse events)), and significant change in the proportion of each category of adverse event (decrease in departures from prevention protocols).

CONCLUSIONS

Despite their limitations, the effectiveness and efficiency of quality assurance programmes seem to be real and valuable. Maintaining quality improvement requires conditions which include some of the basic principles of total quality management (leadership, participatory management, openness, continuous feed back). The organisation of this unit as a specialised trauma centre was also a determining factor in the feasibility of a quality assurance programme (specialisation and small size, high activity volume, management of the complete care process). Quality assurance is an important initial step towards quality improvement, that should precede consideration of a total quality management programme.

Quality improvement among primary care practitioners: an overall appraisal of results of the Ambulatory Care Medical Audit Demonstration Project

OBJECTIVES

The authors appraised the overall conclusions of a randomized, controlled trial of quality assurance in 16 primary care group practices, addressing the relevance of findings to health maintenance organizations in the 1990s.

METHODS

The framework was the analogy of opening the "black box" of quality assurance interventions to examine circumstances in which interventions worked.

RESULTS

External pressures for quality improvement were weak during the study and knowledge of continuous quality improvement principles lacking. Correspondingly, within study practices, pre-existing mechanisms lacked the rigorous data-driven approach and system focus of the quality assurance cycles conducted. Additional barriers to demonstrating an effect of quality assurance included pre-existing good performance, high variability in performance measurements, and lack of time within the study for radical re-design of systems. Improvement in performance for one guideline was impeded by change of practice recommendations during the study. Nevertheless, clinically and statistically significant improvements in quality were obtained in five of the seven remaining guidelines, with effects peaking after feedback of performance results. A sixth guideline showed improvement in practices in which the physician leader influenced colleagues to improve. The seventh guideline showed improvement that did not reach statistical significance, in part because of lack of statistical power.

CONCLUSIONS

This study demonstrated the effectiveness of cycles of quality measurement and improvement. The findings provide guidance for health-care practitioners and managers of the 1990s, for whom quality measurement and improvement has become a priority.

The Ambulatory Care Medical Audit Demonstration Project. Research design

The authors describe the design of and statistical analyses involved in the Ambulatory Care Medical Audit Demonstration Project, which tested feasibility, cost, and effectiveness of cycles that met quality assurance requirements in eight pediatric and eight general medicine group practices at four teaching hospitals and six health centers. The authors used a concurrent crossover design using randomized cycles of quality assurance so that a practice was a control site for one guideline and an experimental site for another. For 12 months before and 18 months during and after quality assurance experimental interventions, the authors measured practitioner conformance to review criteria for patient-care guidelines believed to improve outcomes, including four internal medicine patient-care guidelines (ie, follow-up of low hematocrit, cancer screening for women, follow-up of high serum glucose, and monitoring of patients treated with digoxin) and four pediatric patient-care guidelines (ie, follow-up of positive urine cultures, screening for disease and immunizing infants, management of acute gastroenteritis, and management of acute ear infection). The authors distinguished review criteria whose performance depended on personal efforts of practitioners from those that concerned performance dependent on the practice's system for reporting test results and calling patients to return for care.