Variation in Clinical Practice: A Priority Setting Approach to the Staged Funding of Quality Improvement

Local Level Economic Evaluation: What is it? What is its Value? Is it Sustainable?

In Australia, local health services with allocated budgets manage public hospital services for defined geographical areas. The authors were embedded in a local health service for around 2 years and undertook a range of local level economic evaluations for which three decision contexts were defined: intervention development, post-implementation and prioritisation. Despite difficulties in estimating opportunity costs and in the relevance of portfolio-based prioritisation approaches, economic evaluation added value to local decision-making. Development-focused (ex ante) economic evaluations used expert elicitation and calibration methods to synthesise published evidence with local health systems data to evaluate interventions to prevent hospital acquired complications. The use of economic evaluation facilitated the implementation of interventions with additional resource requirements. Decision analytic models were used alongside the implementation of larger scale, more complex service interventions to estimate counterfactual patient pathways, costs and outcomes, providing a transparent alternative to the statistical analyses of intervention effects, which were subject to high risk of bias. Economic evaluations of more established services had less impact due to data limitations and lesser executive interest. Prioritisation-focused economic evaluations compared costs, outcomes and processes of care for defined patient populations across alternative local health services to identify, understand and quantify the effects of unwarranted variation to inform priority areas for improvement within individual local health services. The sustained use of local level economic evaluation could be supported by embedding health economists in local continuous improvement units, perhaps with an initial focus on supporting the development and evaluation of prioritised new service interventions. Shared resources and critical mass are important, which could be facilitated through groups of embedded economists with joint appointments between different local health services and the same academic institution.

Practice Variation Research in Degenerative Lumbar Disc Surgery: A Literature Review on Design Characteristics and Outcomes

STUDY DESIGN

Literature review.

OBJECTIVE

To describe whether practice variation studies on surgery in patients with lumbar degenerative disc disease used adequate study methodology to identify unwarranted variation, and to inform quality improvement in clinical practice. Secondary aim was to describe whether variation changed over time.

METHODS

Literature databases were searched up to May 4^th, 2021. To define whether study design was appropriate to identify unwarranted variation, we extracted data on level of aggregation, study population, and case-mix correction. To define whether studies were appropriate to achieve quality improvement, data were extracted on outcomes, explanatory variables, description of scientific basis, and given recommendations. Spearman's rho was used to determine the association between the Extreme Quotient (EQ) and year of publication.

RESULTS

We identified 34 articles published between 1990 and 2020. Twenty-six articles (76%) defined the diagnosis. Prior surgery cases were excluded or adjusted for in 5 articles (15%). Twenty-three articles (68%) adjusted for case-mix. Variation in outcomes was analyzed in 7 articles (21%). Fourteen articles (41%) identified explanatory variables. Twenty-six articles (76%) described the evidence on effectiveness. Recommendations for clinical practice were given in 9 articles (26%). Extreme Quotients ranged between 1-fold and 15-fold variation and did not show a significant change over time (rho= -.33, P= .09).

CONCLUSIONS

Practice variation research on surgery in patients with degenerative disc disease showed important limitations to identify unwarranted variation and to achieve quality improvement by public reporting. Despite the availability of new evidence, we could not observe a significant decrease in variation over time.

Data Consult Service: Can we use observational data to address immediate clinical needs?

OBJECTIVE

A number of clinical decision support tools aim to use observational data to address immediate clinical needs, but few of them address challenges and biases inherent in such data. The goal of this article is to describe the experience of running a data consult service that generates clinical evidence in real time and characterize the challenges related to its use of observational data.

MATERIALS AND METHODS

In 2019, we launched the Data Consult Service pilot with clinicians affiliated with Columbia University Irving Medical Center. We created and implemented a pipeline (question gathering, data exploration, iterative patient phenotyping, study execution, and assessing validity of results) for generating new evidence in real time. We collected user feedback and assessed issues related to producing reliable evidence.

RESULTS

We collected 29 questions from 22 clinicians through clinical rounds, emails, and in-person communication. We used validated practices to ensure reliability of evidence and answered 24 of them. Questions differed depending on the collection method, with clinical rounds supporting proactive team involvement and gathering more patient characterization questions and questions related to a current patient. The main challenges we encountered included missing and incomplete data, underreported conditions, and nonspecific coding and accurate identification of drug regimens.

CONCLUSIONS

While the Data Consult Service has the potential to generate evidence and facilitate decision making, only a portion of questions can be answered in real time. Recognizing challenges in patient phenotyping and designing studies along with using validated practices for observational research are mandatory to produce reliable evidence.

The NSW Pathology Atlas of Variation: Part I-Identifying Emergency Departments With Outlying Laboratory Test-Ordering Practices

STUDY OBJECTIVE

Abdominal pain and chest pain are leading reasons for emergency department (ED) presentations, with laboratory tests frequently ordered to aid the diagnostic process. Our study aims to identify EDs with outlying laboratory ordering practices for patients presenting with undifferentiated abdominal pain and chest pain.

METHODS

This was a retrospective observational study of 519,597 patients who presented with the complaint of abdominal or chest pain at 44 major hospital EDs across New South Wales, Australia, from January 2017 to September 2018. For each condition, we evaluated the risk-adjusted rate of ordering at least 1 laboratory test and of each of the top 15 ordered tests. We used funnel plots to graph variations in test ordering and identify EDs with outlying test-ordering practices. EDs lying above or below the 99.8% funnel control limits were regarded as outliers.

RESULTS

From 3,360,152 unplanned presentations, abdominal pain and chest pain represented 8.8% (n=296,809) and 6.6% (n=222,788) of all cases, respectively. No major outliers were observed for ordering at least one laboratory test; however, variations were observed for individual tests. For abdominal pain, the top 3 tests with the highest ordering variation included glucose (20 outlier EDs), C-reactive protein (10 outliers), and calcium-magnesium-phosphate (7 outliers). For chest pain, the top 3 tests with the highest ordering variation were glucose (21 outlier EDs), C-reactive protein (17 outliers), and liver function test (14 outliers).

CONCLUSION

Identifying EDs with outlying laboratory-ordering practices is the first step in initiating context-specific evaluation of whether outlying variations are unwarranted.

Nurses' protocol-based care decision-making: a multiple case study

AIM

To describe and explain nurses' protocol-based care decision-making.

BACKGROUND

Protocol-based care is a strategy to reduce variability in clinical practice. There are no studies looking at protocol-based care decision-making. Understand this process is key to successful implementation.

METHOD

A multiple embedded case study was carried out. Nurses' protocol-based care decision-making was studied in three inpatient wards (medical, surgical and medical-surgical) of a university hospital in northern Spain. Data collection was performed between 2015 and 2016 including documentary analysis, non-participant observations, participant observations and interviews. Analysis of quantitative data involved descriptive statistics and qualitative data was submitted to Burnard's method of content analysis (1996). The data integration comprised the integration of the data set of each case separately and the integration of the findings resulting from the comparison of the cases. The following the thread method of data integration was used for this purpose. The SRQR guideline was used for reporting.

RESULTS

The multiple embedded case study revealed protocol-based care decision-making as a linear and variable process that depends on the context and consists of multiple interrelated elements, among which the risk perception is foremost.

CONCLUSION

This study has allowed progress in protocol-based care decision-making characterisation. This knowledge is crucial to support the design of educational and management strategies aimed at implementing protocol-based care.

RELEVANCE TO CLINICAL PRACTICE

Strategies to promote protocol-based care should address the contexts of practice and the ability of professionals' to accurately assess the degree of risk of clinical activity. Hence, it will promote quality of care, patient safety and efficiency in healthcare cost.

Evaluations of healthcare delivery models in Australia: a scoping review protocol

OBJECTIVE

The aim of this review is to identify and describe the evidence base of published primary, comparative healthcare delivery model evaluations that require the employment of additional healthcare practitioners undertaken in Australia.

INTRODUCTION

In Australia, formal processes are utilized in assessing the value of new pharmaceuticals and medical services, which inform decisions on whether to list new items on the Pharmaceutical Benefits Schedule and Medicare Benefits Schedule, respectively. There are no formal processes to aid in decision making on the funding of new, evaluated healthcare delivery models. This imbalance undervalues the available evidence on healthcare delivery models, leading to the sub-optimal allocation of resources between new health technologies and new healthcare delivery models within the Australian health system.

INCLUSION CRITERIA

Eligible studies will evaluate healthcare delivery models that require the employment of additional healthcare practitioners (either to replace existing practitioners of another type or to provide new services). Studies must include a comparator to evaluate a condition of interest being treated using alternative healthcare delivery models, or no treatment, and will involve observation of outcomes over a similar period of time. Studies in any Australian setting will be included. Interventions aimed at primary preventions will be excluded.

METHODS

PubMed, Embase and CINAHL will be searched for articles published from 2008. One reviewer will review titles, and then two reviewers will independently review abstracts to identify eligible studies. One reviewer will extract data on study characteristics and design. The results of the data extraction will be presented in a table with examples of case studies.

Use of health economic evaluation in the implementation and improvement science fields-a systematic literature review

Background

Economic evaluation can inform whether strategies designed to improve the quality of health care delivery and the uptake of evidence-based practices represent a cost-effective use of limited resources. We report a systematic review and critical appraisal of the application of health economic methods in improvement/implementation research.

Method

A systematic literature search identified 1668 papers across the Agris, Embase, Global Health, HMIC, PsycINFO, Social Policy and Practice, MEDLINE and EconLit databases between 2004 and 2016. Abstracts were screened in Rayyan database, and key data extracted into Microsoft Excel. Evidence was critically appraised using the Quality of Health Economic Studies (QHES) framework.

Results

Thirty studies were included—all health economic studies that included implementation or improvement as a part of the evaluation. Studies were conducted mostly in Europe (62%) or North America (23%) and were largely hospital-based (70%). The field was split between improvement (N = 16) and implementation (N = 14) studies. The most common intervention evaluated (43%) was staffing reconfiguration, specifically changing from physician-led to nurse-led care delivery. Most studies (N = 19) were ex-post economic evaluations carried out empirically—of those, 17 were cost effectiveness analyses. We found four cost utility analyses that used economic modelling rather than empirical methods. Two cost-consequence analyses were also found. Specific implementation costs considered included costs associated with staff training in new care delivery pathways, the impacts of new processes on patient and carer costs and the costs of developing new care processes/pathways. Over half (55%) of the included studies were rated ‘good’ on QHES. Study quality was boosted through inclusion of appropriate comparators and reporting of incremental analysis (where relevant); and diminished through use of post-hoc subgroup analysis, limited reporting of the handling of uncertainty and justification for choice of discount rates.

Conclusions

The quantity of published economic evaluations applied to the field of improvement and implementation research remains modest; however, quality is overall good. Implementation and improvement scientists should work closely with health economists to consider costs associated with improvement interventions and their associated implementation strategies. We offer a set of concrete recommendations to facilitate this endeavour.

The Impact of Risk Standardization on Variation in CT Use and Emergency Physician Profiling

OBJECTIVE

The purpose of this study is to use detailed electronic health record data to profile the use of condition-specific, risk-standardized imaging by emergency physicians.

MATERIALS AND METHODS

CT utilization in four emergency departments in a single health care system was retrospectively analyzed. The primary outcome for analysis was indication-specific, risk-standardized CT utilization. We constructed seven clinical cohorts on the basis of the presence or absence of a traumatic indication for the most frequently performed CT studies. Risk standardization was performed using machine learning algorithms and hierarchic logistic regression models. Variation in CT utilization for each cohort was analyzed using coefficients of variation and box plots, the effect of risk standardization on physician profiling was determined using slope diagrams and kappa values, and within-physician correlation was assessed using correlation coefficients and matrices.

RESULTS

For the seven cohorts, the number of physicians ordering more than 25 CT studies for a particular indication ranged from 70 to 88, and the number of ED visits ranged from 17,458 to 117,489. The unadjusted variation was large for each indication (coefficient of variation, 30.2-57.9). Risk standardization resulted in reduced but persistent variation for all indications (coefficient of variation, 12.3-22.3). Among indication-specific models, risk standardization resulted in reclassification by two or more deciles for 14.0-39.1% of physicians. The R value for within-physician correlation varied from 0.02 to 0.80 and was highest between chest and abdominal imaging for trauma.

CONCLUSION

In this multisite study of CT utilization, risk standardization had a substantial impact on variation in CT utilization and emergency physician profiling. Administrators and payers should include risk standardization in future measures of physician imaging to ensure valid assessment of performance and achieve improvements in emergency care value.

Management of fever and neutropenia in children with cancer: A survey of Australian and New Zealand practice

AIM

Variation in the management of fever and neutropenia (FN) in children is well described. The aim of this study was to explore the current management of FN across Australia and New Zealand and highlight areas for improvement.

METHODS

A practice survey was administered to paediatric health-care providers via four clinical and research networks. Using three clinical case vignettes, we explored risk stratification, empiric antibiotics, initial investigations, intravenous-oral switch, ambulatory management and antibiotic duration in children with cancer and FN.

RESULTS

A response was received from 104 participants from 16 different hospitals. FN guideline compliance was rated as moderate or poor by 24% of respondents, and seven different fever definitions were described. There was little variation in the selected empiric monotherapy and dual-therapy regimens, and almost all respondents recommended first-dose antibiotics within 1 h. However, 27 different empiric antibiotic combinations were selected for beta-lactam allergy. An incorrect risk status was assigned to the low-risk case by 27% of respondents and to the high-risk case by 41%. Compared to current practice, significantly more respondents would manage the low-risk case in the ambulatory setting provided adequate resources were in place (43 vs. 85%, P < 0.0001). There was variation in the use of empiric glycopeptides as well as use of aminoglycosides beyond 48 h.

CONCLUSION

Although the antibiotics selected for empiric management of FN are appropriate and consistent, variation and inaccuracies exist in risk stratification, the selection of monotherapy over dual therapy, empiric antibiotics chosen for beta-lactam allergy, use of glycopeptides and duration of aminoglycosides.

Variation in Care and Clinical Outcomes in Children Hospitalized With Orbital Cellulitis

OBJECTIVES

To describe variation in the care of children hospitalized with orbital cellulitis and to determine associations with length of stay (LOS), emergency department (ED) revisits, and hospital readmissions.

METHODS

By using the Pediatric Health Information System, we performed a multicenter, retrospective study of children aged 2 months to 18 years with a primary International Classification of Diseases, Ninth Revision, Clinical Modification discharge diagnosis code for orbital cellulitis from 2007 to 2014. We assessed hospital-level variation in the use of diagnostic tests, corticosteroids, and antibiotics individually and in aggregate for association with outcomes (LOS, ED revisits, readmissions) after risk-adjusting for important clinical and demographic factors.

RESULTS

A total of 1828 children met inclusion criteria. Complete blood cell counts (median [interquartile range]: 81.8% [66.7-89.6]), C-reactive protein levels (57.1% [22.2-84.0]), blood cultures (57.9% [48.9-63.6]), and computed tomography imaging (74.7% [66.7-81.0]) were the most frequently performed diagnostic tests, with significant variation observed across hospitals (all P < .001). Corticosteroids were used in 29.2% of children (interquartile range: 18.4-37.5). There was significant variation in antibiotic exposure across hospitals (P < .001). Increased total diagnostic test usage was associated with increased LOS (P = .044), but not with 30-day ED revisits (P = .176) or readmissions (P = .403).

CONCLUSIONS

Children hospitalized with orbital cellulitis experience wide variation in clinical management. Increased hospital-level usage is associated with increased LOS. Our findings highlight a critical need to identify treatment strategies that optimize resource use and outcomes for children hospitalized with orbital cellulitis.

Quality Improvement in Athletic Health Care

CONTEXT

  Quality improvement (QI) is a health care concept that ensures patients receive high-quality (safe, timely, effective, efficient, equitable, patient-centered) and affordable care. Despite its importance, the application of QI in athletic health care has been limited.

OBJECTIVES

  To describe the need for and define QI in health care, to describe how to measure quality in health care, and to present a QI case in athletic training.

DESCRIPTION

  As the athletic training profession continues to grow, a widespread engagement in QI efforts is necessary to establish the value of athletic training services for the patients that we serve. A review of the importance of QI in health care, historical perspectives of QI, tools to drive QI efforts, and examples of common QI initiatives is presented to assist clinicians in better understanding the value of QI for advancing athletic health care and the profession. Clinical and Research Advantages:  By engaging clinicians in strategies to measure outcomes and improve their patient care services, QI practice can help athletic trainers provide high-quality and affordable care to patients.