The use of control charts by laypeople and hospital decision-makers for guiding decision making

Retrospective evaluation of an intervention based on training sessions to increase the use of control charts in hospitals

BACKGROUND

Statistical process control charts (SPCs) distinguish signal from noise in quality and safety metrics and thus enable resources to be targeted towards the most suitable actions for improving processes and outcomes. Nevertheless, according to a recent study, SPCs are not widely used by hospital boards in England. To address this, an educational training initiative with training sessions lasting less than one and a half days was established to increase uptake of SPCs in board papers. This research evaluated the impact of the training sessions on the inclusion of SPCs in hospital board papers in England.

METHODS

We used a non-randomised controlled before and after design. Use of SPCs was examined in 40 publicly available board papers across 20 hospitals; 10 intervention hospitals and 10 control hospitals matched using hospital characteristics and time-period. Zero-inflated negative binomial regression models and t-tests compared changes in usage by means of a difference in difference approach.

RESULTS

Across the 40 board papers in our sample, we found 6287 charts. Control hospitals had 9/1585 (0.6%) SPCs before the intervention period and 23/1900 (1.2%) after the intervention period, whereas intervention hospitals increased from 89/1235 (7%) before to 328/1567 (21%) after the intervention period; a relative risk ratio of 9 (95% CI 3 to 32). The absolute difference in use of SPCs was 17% (95% CI 6% to 27%) in favour of the intervention group.

CONCLUSIONS

The results suggest that a scalable educational training initiative to improve use of SPCs within organisations can be effective. Future research could aim to overcome the limitations of observational research with an experimental design or seek to better understand mechanisms, decision-making and patient outcomes.

Establishing a global quality of care benchmark report

BACKGROUND

The Movember funded TrueNTH Global Registry (TNGR) aims to improve care by collecting and analysing a consistent dataset to identify variation in disease management, benchmark care delivery in accordance with best practice guidelines and provide this information to those in a position to enact change. We discuss considerations of designing and implementing a quality of care report for TNGR.

METHODS

Eleven working group sessions were held prior to and as reports were being built with representation from clinicians, data managers and investigators contributing to TNGR. The aim of the meetings was to understand current data display approaches, share literature review findings and ideas for innovative approaches. Preferred displays were evaluated with two surveys (survey 1: 5 clinicians and 5 non-clinicians, 83% response rate; survey 2: 17 clinicians and 18 non-clinicians, 93% response rate).

RESULTS

Consensus on dashboard design and three data-display preferences were achieved. The dashboard comprised two performance summary charts; one summarising site's relative quality indicator (QI) performance and another to summarise data quality. Binary outcome QIs were presented as funnel plots. Patient-reported outcome measures of function score and the extent to which men were bothered by their symptoms were presented in bubble plots. Time series graphs were seen as providing important information to supplement funnel and bubble plots. R Markdown was selected as the software program principally because of its excellent analytic and graph display capacity, open source licensing model and the large global community sharing program code enhancements.

CONCLUSIONS

International collaboration in creating and maintaining clinical quality registries has allowed benchmarking of process and outcome measures on a large scale. A registry report system was developed with stakeholder engagement to produce dynamic reports that provide user-specific feedback to 132 participating sites across 13 countries.

National Health Service (NHS) trust boards adopt statistical process control reporting: the impact of the Making Data Count Training Programme

Background

Red, amber, green (RAG) reports persist as the tool most commonly used by NHS trust boards to understand performance and gain assurance, despite statistical process control (SPC) being a more reliable way of presenting data over time. The aim of this study is to report board members’ feedback on an educational intervention focusing on the use of SPC in NHS trust performance reports, review the presence of SPC charts in performance reports and explore board members’ experience of behavioural changes in their board or fellow board members following the intervention.

Methods

A 90-minute board training session in the use of SPC—Making Data Count—was delivered to 61 NHS trust boards between November 2017 and July 2019. This paper describes the approach taken with boards to enable them to understand the limitations of RAG reports and the benefits of using SPC and analyses the extent to which the Making Data Count training has led to boards adopting SPC. The paper provides quantitative analysis of the increase in SPC use across the 61 participating boards, summaries from the board evaluation forms and qualitative reflections of seven senior leaders from four boards who consented to participate in post-training interviews with an independent evaluator.

Results

During the period covered by this study, 583 participants of board training provided feedback. 99% of respondents agreed that the training session was a good use of their time. 97% of respondents agreed that participating in the event would enhance their ability to make good decisions. A review of the presence of SPC charts in the board papers of the 61 trusts prior to the board training revealed that 72% contained 0–5 SPC charts. A review of the same trusts’ papers 6–12 months after the training revealed a significant increase in the presence of SPC with 85% of reports containing a minimum of six charts.

Conclusion

The Making Data Count education intervention has increased the use of SPC in board reports and has had some self-reported impact on individual and collective behavioural changes by board members, including reducing the amount of time wasted by boards discussing insignificant changes in data and providing a clearer focus on those issues requiring board attention. Further research is required to see if this immediate impact is sustained over time and to identify the key enablers and barriers to organisational adoption of SPC by boards in the NHS.

Monitoring Hospital Performance with Statistical Process Control After Total Hip and Knee Arthroplasty: A Study to Determine How Much Earlier Worsening Performance Can Be Detected

BACKGROUND

Given the low early revision rate after total hip arthroplasty (THA) and total knee arthroplasty (TKA), hospital performance is typically compared using 3 years of data. The purpose of this study was to assess how much earlier worsening hospital performance in 1-year revision rates after THA and TKA can be detected.

METHODS

All 86,468 THA and 73,077 TKA procedures performed from 2014 to 2016 and recorded in the Dutch Arthroplasty Register were included. Negative outlier hospitals were identified by significantly higher O/E (observed divided by expected) 1-year revision rates in a funnel plot. Monthly Shewhart p-charts (with 2 and 3-sigma control limits) and cumulative sum (CUSUM) charts (with 3.5 and 5 control limits) were constructed to detect a doubling of revisions (odds ratio of 2), generating a signal when the control limit was reached. The median number of months until generation of a first signal for negative outliers and the number of false signals for non-negative outliers were calculated. Sensitivity, specificity, and accuracy were calculated for all charts and control limit settings using outlier status in the funnel plot as the gold standard.

RESULTS

The funnel plot showed that 13 of 97 hospitals had significantly higher O/E 1-year revision rates and were negative outliers for THA and 7 of 98 hospitals had significantly higher O/E 1-year revision rates and were negative outliers for TKA. The Shewhart p-chart with the 3-sigma control limit generated 68 signals (34 false-positive) for THA and 85 signals (63 false-positive) for TKA. The sensitivity for THA and TKA was 92% and 100%, respectively; the specificity was 69% and 51%, respectively; and the accuracy was 72% and 54%, respectively. The CUSUM chart with a 5 control limit generated 18 signals (1 false-positive) for THA and 7 (1 false-positive) for TKA. The sensitivity was 85% and 71% for THA and TKA, respectively; the specificity was 99% for both; and the accuracy was 97% for both. The Shewhart p-chart with a 3-sigma control limit generated the first signal for negative outliers after a median of 10 months (interquartile range [IQR] = 2 to 18) for THA and 13 months (IQR = 5 to 18) for TKA. The CUSUM chart with a 5 control limit generated the first signal after a median of 18 months (IQR = 7 to 22) for THA and 21 months (IQR = 9 to 25) for TKA.

CONCLUSIONS

Monthly monitoring using CUSUM charts with a 5 control limit enables earlier detection of worsening 1-year revision rates with accuracy so that initiatives to improve care can start earlier.

A Novel Think Tank Program to Promote Innovation and Strategic Planning in Ophthalmic Surgery

Background

Continuous quality improvement is a pillar of all surgical groups. Innovation is a critical aspect to continuously improve, but traditional staff retreats have several disadvantages which limit their utility in identifying needs and developing innovative solutions. To address these challenges, we designed the novel Think Tank Program to spur innovation and strategic planning for an academic ophthalmology department including the Kellogg Eye Center 6 operating rooms.

Methods

The Think Tank program is a structured seven-phase program for faculty in small teams to identify, innovate, and implement meaningful change. Participants brainstormed problems and possible solutions to those problems, formed teams, acquired data, and implemented meaningful change in clinical care, research, education, and administration.

Results

The program generated 19 novel proposals and significant faculty engagement and discussion in improving the department. A case example of improving the operating room (OR) utilization resulted in improved OR utilization from 63.8% to 74.6% over a 3 month period before and after implementation. It also resulted in a reduction of cancelled or rescheduled surgeries within 2 weeks or surgery from 29.8% to 15.2%. This resulted in an estimated positive financial margin of over $141,000 to the institution in addition to improvement in patient, surgeon, and staff satisfaction with the quality of care.

Conclusions

Engaged faculty, critical data analysis, and value proposition analysis with data-driven metrics and accountability can result in a significant increase in OR utilization and reduction in surgical cancellations. Think Tank serves as a model transformative program to assist practices and institutions to best fulfill their mission while actively engaging and retaining their members.

Comparison of control charts for monitoring clinical performance using binary data

Background

Time series charts are increasingly used by clinical teams to monitor their performance, but statistical control charts are not widely used, partly due to uncertainty about which chart to use. Although there is a large literature on methods, there are few systematic comparisons of charts for detecting changes in rates of binary clinical performance data.

Methods

We compared four control charts for binary data: the Shewhart p-chart; the exponentially weighted moving average (EWMA) chart; the cumulative sum (CUSUM) chart; and the g-chart. Charts were set up to have the same long-term false signal rate. Chart performance was then judged according to the expected number of patients treated until a change in rate was detected.

Results

For large absolute increases in rates (>10%), the Shewhart p-chart and EWMA both had good performance, although not quite as good as the CUSUM. For small absolute increases (<10%), the CUSUM detected changes more rapidly. The g-chart is designed to efficiently detect decreases in low event rates, but it again had less good performance than the CUSUM.

Implications

The Shewhart p-chart is the simplest chart to implement and interpret, and performs well for detecting large changes, which may be useful for monitoring processes of care. The g-chart is a useful complement for determining the success of initiatives to reduce low-event rates (eg, adverse events). The CUSUM may be particularly useful for faster detection of problems with patient safety leading to increases in adverse event rates.