Performance of statistical process control methods for regional surgical site infection surveillance: a 10-year multicentre pilot study

Use of Statistical Process Control Methods for Early Detection of Healthcare Facility-Associated Nontuberculous Mycobacteria Outbreaks: A Single-Center Pilot Study

BACKGROUND

Nontuberculous mycobacteria (NTM) are emerging pathogens increasingly implicated in healthcare facility-associated (HCFA) infections and outbreaks. We analyzed the performance of statistical process control (SPC) methods in detecting HCFA NTM outbreaks.

METHODS

We retrospectively analyzed 3 NTM outbreaks that occurred from 2013 to 2016 at a tertiary care hospital. The outbreaks consisted of pulmonary Mycobacterium abscessus complex (MABC) acquisition, cardiac surgery-associated extrapulmonary MABC infection, and a bronchoscopy-associated pseudo-outbreak of Mycobacterium avium complex (MAC). We analyzed monthly case rates of unique patients who had positive respiratory cultures for MABC, non-respiratory cultures for MABC, and bronchoalveolar lavage cultures for MAC, respectively. For each outbreak, we used these rates to construct a pilot moving average (MA) SPC chart with a rolling baseline window. We also explored the performance of numerous alternative control charts, including exponentially weighted MA, Shewhart, and cumulative sum charts.

RESULTS

The pilot MA chart detected each outbreak within 2 months of outbreak onset, preceding actual outbreak detection by an average of 6 months. Over a combined 117 months of pre-outbreak and post-outbreak surveillance, no false-positive SPC signals occurred (specificity, 100%). Prospective use of this chart for NTM surveillance could have prevented an estimated 108 cases of NTM. Six high-performing alternative charts detected all outbreaks during the month of onset, with specificities ranging from 85.7% to 94.9%.

CONCLUSIONS

SPC methods have potential to substantially improve HCFA NTM surveillance, promoting early outbreak detection and prevention of NTM infections. Additional study is needed to determine the best application of SPC for prospective HCFA NTM surveillance in other settings.

Grand rounds in methodology: four critical decision points in statistical process control evaluations of quality improvement initiatives

Quality improvement (QI) projects often employ statistical process control (SPC) charts to monitor process or outcome measures as part of ongoing feedback, to inform successive Plan-Do-Study-Act cycles and refine the intervention (formative evaluation). SPC charts can also be used to draw inferences on effectiveness and generalisability of improvement efforts (summative evaluation), but only if appropriately designed and meeting specific methodological requirements for generalisability. Inadequate design decreases the validity of results, which not only reduces the chance of publication but could also result in patient harm and wasted resources if incorrect conclusions are drawn. This paper aims to bring together much of what has been written in various tutorials, to suggest a process for using SPC in QI projects. We highlight four critical decision points that are often missed, how these are inter-related and how they affect the inferences that can be drawn regarding effectiveness of the intervention: (1) the need for a stable baseline to enable drawing inferences on effectiveness; (2) choice of outcome measures to assess effectiveness, safety and intervention fidelity; (3) design features to improve the quality of QI projects; (4) choice of SPC analysis aligned with the type of outcome, and reporting on the potential influence of other interventions or secular trends.These decision points should be explicitly reported for readers to interpret and judge the results, and can be seen as supplementing the Standards for Quality Improvement Reporting Excellence guidelines. Thinking in advance about both formative and summative evaluation will inform more deliberate choices and strengthen the evidence produced by QI projects.

Early recognition and response to increases in surgical site infections using optimised statistical process control charts-The early 2RIS trial: A multicentre stepped wedge cluster randomised controlled trial

BACKGROUND

Traditional approaches for surgical site infection (SSI) surveillance have deficiencies that delay detection of SSI outbreaks and other clinically important increases in SSI rates. We investigated whether use of optimised statistical process control (SPC) methods and feedback for SSI surveillance would decrease rates of SSI in a network of US community hospitals.

METHODS

We conducted a stepped wedge cluster randomised trial of patients who underwent any of 13 types of common surgical procedures across 29 community hospitals in the Southeastern United States. We divided the 13 procedures into six clusters; a cluster of procedures at a single hospital was the unit of randomisation and analysis. In total, 105 clusters were randomised to 12 groups of 8-10 clusters. All participating clusters began the trial in a 12-month baseline period of control or "traditional" SSI surveillance, including prospective analysis of SSI rates and consultative support for SSI outbreaks and investigations. Thereafter, a group of clusters transitioned from control to intervention surveillance every three months until all clusters received the intervention. Electronic randomisation by the study statistician determined the sequence by which clusters crossed over from control to intervention surveillance. The intervention was the addition of weekly application of optimised SPC methods and feedback to existing traditional SSI surveillance methods. Epidemiologists were blinded to hospital identity and randomisation status while adjudicating SPC signals of increased SSI rates, but blinding was not possible during SSI investigations. The primary outcome was the overall SSI prevalence rate (PR=SSIs/100 procedures), evaluated via generalised estimating equations with a Poisson regression model. Secondary outcomes compared traditional and optimised SPC signals that identified SSI rate increases, including the number of formal SSI investigations generated and deficiencies identified in best practices for SSI prevention. This trial was registered at ClinicalTrials.gov, NCT03075813.

FINDINGS

Between Mar 1, 2016, and Feb 29, 2020, 204,233 unique patients underwent 237,704 surgical procedures. 148,365 procedures received traditional SSI surveillance and feedback alone, and 89,339 procedures additionally received the intervention of optimised SPC surveillance. The primary outcome of SSI was assessed for all procedures performed within participating clusters. SSIs occurred after 1171 procedures assigned control surveillance (prevalence rate [PR] 0.79 per 100 procedures), compared to 781 procedures that received the intervention (PR 0·87 per 100 procedures; model-based PR ratio 1.10, 95% CI 0.94-1.30, p=0.25). Traditional surveillance generated 24 formal SSI investigations that identified 120 SSIs with deficiencies in two or more perioperative best practices for SSI prevention. In comparison, optimised SPC surveillance generated 74 formal investigations that identified 458 SSIs with multiple best practice deficiencies.

INTERPRETATION

The addition of optimised SPC methods and feedback to traditional methods for SSI surveillance led to greater detection of important SSI rate increases and best practice deficiencies but did not decrease SSI rates. Additional research is needed to determine how to best utilise SPC methods and feedback to improve adherence to SSI quality measures and prevent SSIs.

FUNDING

Agency for Healthcare Research and Quality.

Increasing the use of perioperative risk scoring in emergency laparotomy: nationwide quality improvement programme

Background

Emergency laparotomy is associated with high morbidity and mortality. The early identification of high-risk patients allows for timely perioperative care and appropriate resource allocation. The aim of this study was to develop a nationwide surgical trainee-led quality improvement (QI) programme to increase the use of perioperative risk scoring in emergency laparotomy.

Methods

The programme was structured using the active implementation framework in 15 state-funded Irish hospitals to guide the staged implementation of perioperative risk scoring. The primary outcome was a recorded preoperative risk score for patients undergoing an emergency laparotomy at each site.

Results

The rate of patients undergoing emergency laparotomy receiving a perioperative risk score increased from 0–11 per cent during the exploratory phase to 35–100 per cent during the full implementation phase. Crucial factors for implementing changes included an experienced central team providing implementation support, collaborator engagement, and effective communication and social relationships.

Conclusions

A trainee-led QI programme increased the use of perioperative risk assessment in patients undergoing emergency laparotomy, with the potential to improve patient outcomes and care delivery.

Surgical Site Infections

Surgical site infections (SSIs) are among the most common and most costly health care-associated infections, leading to adverse patient outcomes and death. Wound contamination occurs with each incision, but proven strategies exist to decrease the risk of SSI. In particular, improved adherence to evidence-based preventive measures related to appropriate antimicrobial prophylaxis can decrease the rate of SSI. Aggressive surgical debridement and effective antimicrobial therapy are needed to optimize the treatment of SSI.

Introduction of a standardised maternity early warning system: indicative data from a before-and-after study at a large pilot site before national rollout in Aotearoa New Zealand

Strong evidence now demonstrates that recognition and response systems using standardised early warning scores can help prevent harm associated with in-hospital clinical deterioration in non-pregnant adult patients. However, a standardised maternity-specific early warning system has not yet been agreed in the UK. In Aotearoa New Zealand, following the nationwide implementation of the standardised New Zealand Early Warning Score (NZEWS) for adult inpatients, a modified maternity-specific variation (NZMEWS) was piloted in a major tertiary hospital in Auckland, before national rollout. Following implementation in July 2018, we observed a significant and sustained reduction in severe maternal morbidity as measured by emergency response calls to women who were very unwell (emergency response team call), and a non-significant reduction in cardiorespiratory arrest team calls. Emergency response team calls to maternity wards fell from a median of 0.8 per 100 births at baseline (January 2017-May 2018) to 0.6 per 100 births monthly (from March 2019 to December 2020) (p < 0.0001). Cardiorespiratory arrest team calls to maternity wards fell from 0.14 per 100 births per quarter (quarter 1 2017-quarter 2 2018) to 0.09 calls per 100 births per quarter after NZMEWS was introduced (quarter 3 2018-quarter 4 2020) (p = 0.2593). These early results provide evidence that NZMEWS can detect and prevent deterioration of pregnant women, although there are multiple factors that may have contributed to the reduction in emergency response calls noted.

Monitoring scheme for early detection of coronavirus and other respiratory virus outbreaks

In December 2019, an outbreak of pneumonia caused by a novel coronavirus (severe acute respiratory syndrome coronavirus 2 [SARS-CoV-2]) began in Wuhan, China. SARS-CoV-2 exhibited efficient person-to-person transmission of what became labeled as COVID-19. It has spread worldwide with over 83,000,000 infected cases and more than 1,800,000 deaths to date (December 31, 2020). This research proposes a statistical monitoring scheme in which an optimized np control chart is utilized by sentinel metropolitan airports worldwide for early detection of coronavirus and other respiratory virus outbreaks. The sample size of this chart is optimized to ensure the best overall performance for detecting a wide range of shifts in the infection rate, based on the available resources, such as the inspection rate and the allowable false alarm rate. The effectiveness of the proposed optimized np chart is compared with that of the traditional np chart with a predetermined sample size under both sampling inspection and 100% inspection. For a variety of scenarios including a real case, the optimized np control chart is found to substantially outperform its traditional counterpart in terms of the average number of infections. Therefore, this control chart has potential to be an effective tool for early detection of respiratory virus outbreaks, promoting early outbreak investigation and mitigation.

A Multidisciplinary Quality Improvement Initiative to Facilitate Penicillin Allergy Delabeling Among Hospitalized Pediatric Patients

BACKGROUND

Penicillin allergy is reported in up to 10% of the general population; however, >90% of patients reporting an allergy are tolerant. Patients labeled as penicillin allergic have longer hospital stays, increased exposure to suboptimal antibiotics, and an increased risk of methicillin-resistant Staphylococcus aureus and Clostridioides difficile. The primary aim with our quality improvement initiative was to increase penicillin allergy delabeling to at least 10% among all hospitalized pediatric patients reporting a penicillin allergy with efforts directed toward patients determined to be low risk for true allergic reaction.

METHODS

Our quality improvement project included several interventions: the development of a multidisciplinary clinical care pathway to identify eligible patients, workflow optimization to support delabeling, an educational intervention, and participation in our institution's quality improvement incentive program. Our interventions were targeted to facilitate appropriate delabeling by the primary hospital medicine team. Statistical process control charts were used to assess the impact of this intervention pre- and postpathway implementation.

RESULTS

After implementation of the clinical pathway, the percentage of patients admitted to hospital medicine delabeled of their penicillin allergy by discharge increased to 11.7%. More than one-half of those delabeled (51.2%) received a penicillin-based antimicrobial at time of discharge. There have been no adverse events or allergic reactions requiring emergency medication administration since pathway implementation.

CONCLUSIONS

Our quality improvement initiative successfully increased the rate of penicillin allergy delabeling among low-risk hospitalized pediatric patients, allowing for increased use of optimal antibiotics.

Monitoring Hospital Performance with Statistical Process Control After Total Hip and Knee Arthroplasty: A Study to Determine How Much Earlier Worsening Performance Can Be Detected

BACKGROUND

Given the low early revision rate after total hip arthroplasty (THA) and total knee arthroplasty (TKA), hospital performance is typically compared using 3 years of data. The purpose of this study was to assess how much earlier worsening hospital performance in 1-year revision rates after THA and TKA can be detected.

METHODS

All 86,468 THA and 73,077 TKA procedures performed from 2014 to 2016 and recorded in the Dutch Arthroplasty Register were included. Negative outlier hospitals were identified by significantly higher O/E (observed divided by expected) 1-year revision rates in a funnel plot. Monthly Shewhart p-charts (with 2 and 3-sigma control limits) and cumulative sum (CUSUM) charts (with 3.5 and 5 control limits) were constructed to detect a doubling of revisions (odds ratio of 2), generating a signal when the control limit was reached. The median number of months until generation of a first signal for negative outliers and the number of false signals for non-negative outliers were calculated. Sensitivity, specificity, and accuracy were calculated for all charts and control limit settings using outlier status in the funnel plot as the gold standard.

RESULTS

The funnel plot showed that 13 of 97 hospitals had significantly higher O/E 1-year revision rates and were negative outliers for THA and 7 of 98 hospitals had significantly higher O/E 1-year revision rates and were negative outliers for TKA. The Shewhart p-chart with the 3-sigma control limit generated 68 signals (34 false-positive) for THA and 85 signals (63 false-positive) for TKA. The sensitivity for THA and TKA was 92% and 100%, respectively; the specificity was 69% and 51%, respectively; and the accuracy was 72% and 54%, respectively. The CUSUM chart with a 5 control limit generated 18 signals (1 false-positive) for THA and 7 (1 false-positive) for TKA. The sensitivity was 85% and 71% for THA and TKA, respectively; the specificity was 99% for both; and the accuracy was 97% for both. The Shewhart p-chart with a 3-sigma control limit generated the first signal for negative outliers after a median of 10 months (interquartile range [IQR] = 2 to 18) for THA and 13 months (IQR = 5 to 18) for TKA. The CUSUM chart with a 5 control limit generated the first signal after a median of 18 months (IQR = 7 to 22) for THA and 21 months (IQR = 9 to 25) for TKA.

CONCLUSIONS

Monthly monitoring using CUSUM charts with a 5 control limit enables earlier detection of worsening 1-year revision rates with accuracy so that initiatives to improve care can start earlier.

Detection of calibration drift in clinical prediction models to inform model updating

Model calibration, critical to the success and safety of clinical prediction models, deteriorates over time in response to the dynamic nature of clinical environments. To support informed, data-driven model updating strategies, we present and evaluate a calibration drift detection system. Methods are developed for maintaining dynamic calibration curves with optimized online stochastic gradient descent and for detecting increasing miscalibration with adaptive sliding windows. These methods are generalizable to support diverse prediction models developed using a variety of learning algorithms and customizable to address the unique needs of clinical use cases. In both simulation and case studies, our system accurately detected calibration drift. When drift is detected, our system further provides actionable alerts by including information on a window of recent data that may be appropriate for model updating. Simulations showed these windows were primarily composed of data accruing after drift onset, supporting the potential utility of the windows for model updating. By promoting model updating as calibration deteriorates rather than on pre-determined schedules, implementations of our drift detection system may minimize interim periods of insufficient model accuracy and focus analytic resources on those models most in need of attention.

Identification of prosthetic hip and knee joint infections using administrative databases-A validation study

OBJECTIVE

To determine whether combinations of diagnosis and procedures codes can improve the detection of prosthetic hip and knee joint infections from administrative databases.

DESIGN

We performed a validation study of all readmissions from January 1, 2010, until December 31, 2016, following primary arthroplasty comparing the diagnosis and procedure codes obtained from an administrative database based upon the International Classification of Disease, Tenth Revision (ICD-10) to the reference standard of chart review.

SETTING

Four tertiary-care hospitals in Toronto, Canada, from 2010 to 2016.

PARTICIPANTS

Individuals who had a primary arthroplasty were identified using procedure codes.

INTERVENTION

Chart review of readmissions identified the presence of a prosthetic joint infection and, if present, the surgical procedure performed.

RESULTS

Overall, 27,802 primary arthroplasties were performed. Among 8,844 readmissions over a median follow-up of 669 days (interquartile range, 256-1,249 days), a PJI was responsible for or present in 586 of 8,844 (6.6%). Diagnosis codes alone exhibited a sensitivity of 0.88 (95% CI, 0.85-0.92) and positive predictive value (PPV) of 0.78 (95% CI, 0.74-0.82) for detecting a PJI. Combining a PJI diagnosis code with procedure codes for an arthroplasty and the insertion of a peripherally inserted central catheter improved detection: sensitivity was 0.92 (95% CI, 0.88-0.94) and PPV was 0.78 (95% CI, 0.74-0.82). However, procedure codes were unable to identify the specific surgical approach to PJI treatment.

CONCLUSIONS

Compared to PJI diagnosis codes, combinations of diagnosis and procedure codes improve the detection of a PJI in administrative databases.

Large-scale empirical optimisation of statistical control charts to detect clinically relevant increases in surgical site infection rates

OBJECTIVE

Surgical site infections (SSIs) are common costly hospital-acquired conditions. While statistical process control (SPC) use in healthcare has increased, limited rigorous empirical research compares and optimises these methods for SSI surveillance. We sought to determine which SPC chart types and design parameters maximise the detection of clinically relevant SSI rate increases while minimising false alarms.

DESIGN

Systematic retrospective data analysis and empirical optimisation.

METHODS

We analysed 12 years of data on 13 surgical procedures from a network of 58 community hospitals. Statistically significant SSI rate increases (signals) at individual hospitals initially were identified using 50 different SPC chart variations (Shewhart or exponentially weighted moving average, 5 baseline periods, 5 baseline types). Blinded epidemiologists evaluated the clinical significance of 2709 representative signals of potential outbreaks (out of 5536 generated), rating them as requiring 'action' or 'no action'. These ratings were used to identify which SPC approaches maximised sensitivity and specificity within a broader set of 3600 individual chart variations (additional baseline variations and chart types, including moving average (MA), and five control limit widths) and over 32 million dual-chart combinations based on different baseline periods, reference data (network-wide vs local hospital SSI rates), control limit widths and other calculation considerations. Results were validated with an additional year of data from the same hospital cohort.

RESULTS

The optimal SPC approach to detect clinically important SSI rate increases used two simultaneous MA charts calculated using lagged rolling baseline windows and 1 SD limits. The first chart used 12-month MAs with 18-month baselines and best identified small sustained increases above network-wide SSI rates. The second chart used 6-month MAs with 3-month baselines and best detected large short-term increases above individual hospital SSI rates. This combination outperformed more commonly used charts, with high sensitivity (0.90; positive predictive value=0.56) and practical specificity (0.67; negative predictive value=0.94).

CONCLUSIONS

An optimised combination of two MA charts had the best performance for identifying clinically relevant small but sustained above-network SSI rates and large short-term individual hospital increases.

Evaluation of the Collaborative Use of an Evidence-Based Care Bundle in Emergency Laparotomy

Question

Is a quality improvement collaborative approach to implementation of a care bundle associated with reductions in mortality from emergency laparotomy?

Findings

In this study of a collaborative project involving 28 hospitals and a total of 14 809 patients, reductions in mortality and length of stay were seen after implementation of a care bundle. Improvement took time to occur and was not seen until the second year of the collaborative project.

Meaning

The findings suggest that hospitals should consider adopting a care bundle approach and participating in a collaborative group to see improvement in outcomes for patients undergoing emergency laparotomy.

Importance

Patients undergoing emergency laparotomy have high mortality, but few studies exist to improve outcomes for these patients.

Objective

To assess whether a collaborative approach to implement a 6-point care bundle is associated with reduction in mortality and length of stay and improvement in the delivery of standards of care across a group of hospitals.

Design, Setting, and Participants

The Emergency Laparotomy Collaborative (ELC) was a UK-based prospective quality improvement study of the implementation of a care bundle provided to patients requiring emergency laparotomy between October 1, 2015, and September 30, 2017. Participants were 28 National Health Service hospitals and emergency surgical patients who were treated at these hospitals and whose data were entered into the National Emergency Laparotomy Audit (NELA) database. Post-ELC implementation outcomes were compared with baseline data from July 1, 2014, to September 30, 2015. Data entry and collection were performed through the NELA.

Interventions

A 6-point, evidence-based care bundle was used. The bundle included prompt measurement of blood lactate levels, early review and treatment for sepsis, transfer to the operating room within defined time goals after the decision to operate, use of goal-directed fluid therapy, postoperative admission to an intensive care unit, and multidisciplinary involvement of senior clinicians in the decision and delivery of perioperative care. Change management and leadership coaching were provided to ELC leadership teams.

Main Outcome and Measures

Primary outcomes were in-hospital mortality, both crude and Portsmouth Physiological and Operative Severity Score for the enumeration of Mortality and morbidity (P-POSSUM) risk-adjusted, and length of stay. Secondary outcomes were the changes after implementation of the separate metrics in the care bundle.

Results

A total of 28 hospitals participated in the ELC and completed the project. The baseline group included 5562 patients (2937 female [52.8%] and a mean [range] age of 65.3 [18.0-114.0] years), whereas the post-ELC group had 9247 patients (4911 female [53.1%] and a mean [range] age of 65.0 [18.0-99.0] years). Unadjusted mortality rate decreased from 9.8% at baseline to 8.3% in year 2 of the project, and so did risk-adjusted mortality from a baseline of 5.3% to 4.5% post-ELC. Mean length of stay decreased from 20.1 days during year 1 to 18.9 days during year 2. Significant changes in 5 of the 6 metrics in the care bundle were achieved.

Conclusions and Relevance

A collaborative approach using a quality improvement methodology and a care bundle appeared to be effective in reducing mortality and length of stay in emergency laparotomy, suggesting that hospitals should adopt such an approach to see better patient outcomes and care delivery performance.