Low Rate of Completion of Recommended Tests and Referrals in an Academic Primary Care Practice with Resident Trainees

BACKGROUND

A frequent, preventable cause of diagnostic errors involves failure to follow up on diagnostic tests, referrals, and symptoms-termed "failure to close the diagnostic loop." This is particularly challenging in a resident practice where one third of physicians graduate annually, and rates of patient loss due to these transitions may lead to more opportunities for failure to close diagnostic loops. The aim of this study was to determine the prevalence of failure of loop closure in a resident primary care clinic compared to rates in the faculty practice and identify factors contributing to failure.

METHODS

This retrospective cohort study included all patient visits from January 1, 2018, to December 31, 2021, at two academic medical center-based primary care practices where residents and faculty practice in the same setting. The primary outcome was prevalence of failure to close the loop for (1) dermatology referrals, (2) colonoscopy, and (3) cardiac stress testing. The primary predictor was resident vs. faculty status of the ordering provider. The authors present an unadjusted analysis and the results of a multivariable logistic regression analysis incorporating all patient factors to determine their association with loop closure.

RESULTS

Of 12,282 orders for referrals and tests for the three studied areas, 1,929 (15.7%) were ordered by a resident physician. Of resident orders for all three tests, 52.9% were completed within the designated time vs. 58.4% for orders placed by attending physicians (p < 0.01). In an unadjusted analysis by test type, a similar trend was seen for colonoscopy (51.4% completion rate for residents vs. 57.5% for attending physicians, p < 0.01) and for cardiac stress testing (55.7% completion rate for residents vs. 61.2% for attending physicians), though a difference was not seen for dermatology referrals (64.2% completion rate for residents vs. 63.7% for attending physicians). In an adjusted analysis, patients with resident orders were less likely than attendings to close the loop for all test types combined (odds ratio 0.88, 95% confidence interval 0.79-0.98), with low rates of test completion for both physician groups.

CONCLUSION

Loop closure for three diagnostic interventions was low for patients in both faculty and resident primary care clinics, with lower loop closure rates in resident clinics. Failure to close diagnostic loops presents a safety challenge in primary care and is of particular concern for training programs.

Do patients who read visit notes on the patient portal have a higher rate of "loop closure" on diagnostic tests and referrals in primary care? A retrospective cohort study

OBJECTIVES

The 2021 US Cures Act may engage patients to help reduce diagnostic errors/delays. We examined the relationship between patient portal registration with/without note reading and test/referral completion in primary care.

MATERIALS AND METHODS

Retrospective cohort study of patients with visits from January 1, 2018 to December 31, 2021, and order for (1) colonoscopy, (2) dermatology referral for concerning lesions, or (3) cardiac stress test at 2 academic primary care clinics. We examined differences in timely completion ("loop closure") of tests/referrals for (1) patients who used the portal and read ≥1 note (Portal + Notes); (2) those with a portal account but who did not read notes (Portal Account Only); and (3) those who did not register for the portal (No Portal). We estimated the predictive probability of loop closure in each group after adjusting for socio-demographic and clinical factors using multivariable logistic regression.

RESULTS

Among 12 849 tests/referrals, loop closure was more common among Portal+Note-readers compared to their counterparts for all tests/referrals (54.2% No Portal, 57.4% Portal Account Only, 61.6% Portal+Notes, P < .001). In adjusted analysis, compared to the No Portal group, the odds of loop closure were significantly higher for Portal Account Only (OR 1.2; 95% CI, 1.1-1.4), and Portal+Notes (OR 1.4; 95% CI, 1.3-1.6) groups. Beyond portal registration, note reading was independently associated with loop closure (P = .002).

DISCUSSION AND CONCLUSION

Compared to no portal registration, the odds of loop closure were 20% higher in tests/referrals for patients with a portal account, and 40% higher in tests/referrals for note readers, after controlling for sociodemographic and clinical factors. However, important safety gaps from unclosed loops remain, requiring additional engagement strategies.

Completion of Recommended Tests and Referrals in Telehealth vs In-Person Visits

Importance

Use of telehealth has increased substantially in recent years. However, little is known about whether the likelihood of completing recommended tests and specialty referrals-termed diagnostic loop closure-is associated with visit modality.

Objectives

To examine the prevalence of diagnostic loop closure for tests and referrals ordered at telehealth visits vs in-person visits and identify associated factors.

Design, Setting, and Participants

In a retrospective cohort study, all patient visits from March 1, 2020, to December 31, 2021, at 1 large urban hospital-based primary care practice and 1 affiliated community health center in Boston, Massachusetts, were evaluated.

Main Measures

Prevalence of diagnostic loop closure for (1) colonoscopy referrals (screening and diagnostic), (2) dermatology referrals for suspicious skin lesions, and (3) cardiac stress tests.

Results

The study included test and referral orders for 4133 patients (mean [SD] age, 59.3 [11.7] years; 2163 [52.3%] women; 203 [4.9%] Asian, 1146 [27.7%] Black, 2362 [57.1%] White, and 422 [10.2%] unknown or other race). A total of 1151 of the 4133 orders (27.8%) were placed during a telehealth visit. Of the telehealth orders, 42.6% were completed within the designated time frame vs 58.4% of those ordered during in-person visits and 57.4% of those ordered without a visit. In an adjusted analysis, patients with telehealth visits were less likely to close the loop for all test types compared with those with in-person visits (odds ratio, 0.55; 95% CI, 0.47-0.64).

Conclusions

The findings of this study suggest that rates of loop closure were low for all test types across all visit modalities but worse for telehealth. Failure to close diagnostic loops presents a patient safety challenge in primary care that may be of particular concern during telehealth encounters.

Early recognition and response to increases in surgical site infections using optimised statistical process control charts-The early 2RIS trial: A multicentre stepped wedge cluster randomised controlled trial

BACKGROUND

Traditional approaches for surgical site infection (SSI) surveillance have deficiencies that delay detection of SSI outbreaks and other clinically important increases in SSI rates. We investigated whether use of optimised statistical process control (SPC) methods and feedback for SSI surveillance would decrease rates of SSI in a network of US community hospitals.

METHODS

We conducted a stepped wedge cluster randomised trial of patients who underwent any of 13 types of common surgical procedures across 29 community hospitals in the Southeastern United States. We divided the 13 procedures into six clusters; a cluster of procedures at a single hospital was the unit of randomisation and analysis. In total, 105 clusters were randomised to 12 groups of 8-10 clusters. All participating clusters began the trial in a 12-month baseline period of control or "traditional" SSI surveillance, including prospective analysis of SSI rates and consultative support for SSI outbreaks and investigations. Thereafter, a group of clusters transitioned from control to intervention surveillance every three months until all clusters received the intervention. Electronic randomisation by the study statistician determined the sequence by which clusters crossed over from control to intervention surveillance. The intervention was the addition of weekly application of optimised SPC methods and feedback to existing traditional SSI surveillance methods. Epidemiologists were blinded to hospital identity and randomisation status while adjudicating SPC signals of increased SSI rates, but blinding was not possible during SSI investigations. The primary outcome was the overall SSI prevalence rate (PR=SSIs/100 procedures), evaluated via generalised estimating equations with a Poisson regression model. Secondary outcomes compared traditional and optimised SPC signals that identified SSI rate increases, including the number of formal SSI investigations generated and deficiencies identified in best practices for SSI prevention. This trial was registered at ClinicalTrials.gov, NCT03075813.

FINDINGS

Between Mar 1, 2016, and Feb 29, 2020, 204,233 unique patients underwent 237,704 surgical procedures. 148,365 procedures received traditional SSI surveillance and feedback alone, and 89,339 procedures additionally received the intervention of optimised SPC surveillance. The primary outcome of SSI was assessed for all procedures performed within participating clusters. SSIs occurred after 1171 procedures assigned control surveillance (prevalence rate [PR] 0.79 per 100 procedures), compared to 781 procedures that received the intervention (PR 0·87 per 100 procedures; model-based PR ratio 1.10, 95% CI 0.94-1.30, p=0.25). Traditional surveillance generated 24 formal SSI investigations that identified 120 SSIs with deficiencies in two or more perioperative best practices for SSI prevention. In comparison, optimised SPC surveillance generated 74 formal investigations that identified 458 SSIs with multiple best practice deficiencies.

INTERPRETATION

The addition of optimised SPC methods and feedback to traditional methods for SSI surveillance led to greater detection of important SSI rate increases and best practice deficiencies but did not decrease SSI rates. Additional research is needed to determine how to best utilise SPC methods and feedback to improve adherence to SSI quality measures and prevent SSIs.

FUNDING

Agency for Healthcare Research and Quality.

Lessons Learned in Implementing a Chronic Opioid Therapy Management System

OBJECTIVES

Opioid misuse has resulted in significant morbidity and mortality in the United States, and safer opioid use represents an important challenge in the primary care setting. This article describes a research collaborative of health service researchers, systems engineers, and clinicians seeking to improve processes for safer chronic opioid therapy management in an academic primary care center. We present implementation results and lessons learned along with an intervention toolkit that others may consider using within their organization.

METHODS

Using iterative improvement lifecycles and systems engineering principles, we developed a risk-based workflow model for patients on chronic opioids. Two key safe opioid use process metrics-percent of patients with recent opioid treatment agreements and urine drug tests-were identified, and processes to improve these measures were designed, tested, and implemented. Focus groups were conducted after the conclusion of implementation, with barriers and lessons learned identified via thematic analysis.

RESULTS

Initial surveys revealed a lack of knowledge regarding resources available to patients and prescribers in the primary care clinic. In addition, 18 clinicians (69%) reported largely "inheriting" (rather than initiating) their chronic opioid therapy patients. We tracked 68 patients over a 4-year period. Although process measures improved, full adherence was not achieved for the entire population. Barriers included team structure, the evolving opioid environment, and surveillance challenges, along with disruptions resulting from the 2019 novel coronavirus.

CONCLUSIONS

Safe primary care opioid prescribing requires ongoing monitoring and management in a complex environment. The application of a risk-based approach is possible but requires adaptability and redundancies to be reliable.

Evaluating a patient safety learning laboratory to create an interdisciplinary ecosystem for health care innovation

BACKGROUND

In response to the complexity, challenges, and slow pace of innovation, health care organizations are adopting interdisciplinary team approaches. Systems engineering, which is oriented to creating new, scalable processes that perform with higher reliability and lower costs, holds promise for driving innovation in the face of challenges to team performance. A patient safety learning laboratory (lab) can be an essential aspect of fostering interdisciplinary team innovation across multiple projects and organizations by creating an ecosystem focused on deploying systems engineering methods to accomplish process redesign.

PURPOSE

We sought to identify the role and activities of a learning ecosystem that support interdisciplinary team innovation through evaluation of a patient safety learning lab.

METHODS

Our study included three participating learning lab project teams. We applied a mixed-methods approach using a convergent design that combined data from qualitative interviews of team members conducted as teams neared the completion of their redesign projects, as well as evaluation questionnaires administered throughout the 4-year learning lab.

RESULTS

Our results build on learning theories by showing that successful learning ecosystems continually create alignment between interdisciplinary teams' activities, organizational context, and innovation project objectives. The study identified four types of alignment, interpersonal/interprofessional, informational, structural, and processual, and supporting activities for alignment to occur.

CONCLUSION

Interdisciplinary learning ecosystems have the potential to foster health care improvement and innovation through alignment of team activities, project goals, and organizational contexts.

PRACTICE IMPLICATIONS

This study applies to interdisciplinary teams tackling multilevel system challenges in their health care organization and suggests that the work of such teams benefits from the four types of alignment. Alignment on all four dimensions may yield best results.

93. Early Recognition and Response to Increases in Surgical Site Infections (SSI) using Optimized Statistical Process Control (SPC) Charts – the Early 2RIS Trial: A Multicenter Stepped Wedge Cluster Randomized Controlled Trial (RCT)

Background

Traditional approaches for SSI surveillance have deficiencies that can delay detection of SSI outbreaks and other clinically important increases in SSI rates. Optimized SPC methods for SSI surveillance have not been prospectively evaluated.

Methods

We conducted a prospective multicenter stepped wedge cluster RCT to evaluate the performance of SSI surveillance and feedback performed with optimized SPC plus traditional surveillance methods compared to traditional surveillance alone. We divided 13 common surgical procedures into 6 clusters (Table 1). A cluster of procedures at a single hospital was the unit of randomization and analysis, and 105 total clusters across 29 community hospitals were randomized to 12 groups of 8-10 clusters (Figure 1). After a 12-month baseline observation period (3/2016-2/2017), the SPC surveillance intervention was serially implemented according to stepped wedge assignment over a 36-month intervention period (3/2017-2/2020) until all 12 groups of clusters had received the intervention. The primary outcome was the overall SSI prevalence rate (PR=SSIs/100 procedures), evaluated with a GEE model with Poisson distribution.

Table 1

Figure 1

Schematic for stepped wedge design. The 12-month baseline observation period was followed by the 36-month intervention period, comprised of 12 3-month steps.

Results

Our trial involved prospective surveillance of 237,704 procedures that resulted in 1,952 SSIs (PR=0.82). The overall SSI PR did not differ significantly between clusters of procedures assigned to SPC surveillance (781 SSIs/89,339 procedures; PR=0.87) and those assigned to traditional surveillance (1,171 SSIs/148,365 procedures; PR=0.79; PR ratio=1.10 [95% CI, 0.94–1.30]; P=.25) (Table 2). SPC surveillance identified 104 SSI rate increases that required formal investigations, compared to only 25 investigations generated by traditional surveillance. Among 10 best practices for SSI prevention, 453 of 502 (90%) SSIs analyzed due to SPC detection of SSI rate increases had at least 2 deficiencies (Table 3).

Table 2

Poisson regression models comparing surgical site infection (SSI) prevalence rates for procedure clusters receiving statistical process control surveillance to SSI rates for clusters receiving traditional control surveillance.

Table 3

Compliance with 10 best practices for surgical site infection (SSI) prevention among 502 SSIs analyzed during SSI investigations generated by statistical process control surveillance.

Conclusion

SPC methods more frequently detected important SSI rate increases associated with deficiencies in SSI prevention best practices than traditional surveillance; however, feedback of this information did not lead to SSI rate reductions. Further study is indicated to determine the best application of SPC methods to improve adherence to SSI quality measures and prevent SSIs.

Disclosures

Arthur W. Baker, MD, MPH, Medincell (Advisor or Review Panel member) Susan S. Huang, MD, MPH, Medline (Other Financial or Material Support, Conducted studies in which participating hospitals and nursing homes received contributed antiseptic and cleaning products)Molnlycke (Other Financial or Material Support, Conducted studies in which participating hospitals and nursing homes received contributed antiseptic and cleaning products)Stryker (Sage) (Other Financial or Material Support, Conducted studies in which participating hospitals and nursing homes received contributed antiseptic and cleaning products)Xttrium (Other Financial or Material Support, Conducted studies in which participating hospitals and nursing homes received contributed antiseptic and cleaning products)

Increasing efficiency and accuracy of magnetic interaction calculations in colloidal simulation through machine learning

Calculating the magnetic interaction between magnetic particles that are positioned in close proximity to one another is a surprisingly challenging task. Exact solutions for this interaction exist either through numerical expansion of multipolar interactions or through solving Maxwell's equations with a finite element solver. These approaches can take hours for simple configurations of three particles. Meanwhile, across a range of scientific and engineering problems, machine learning approaches have been developed as fast computational platforms for solving complex systems of interest when large data sets are available. In this paper, we bring the touted benefits of recent advances in science-based machine learning algorithms to bear on the problem of modeling the magnetic interaction between three particles. We investigate this approach using diverse machine learning systems including physics informed neural networks. We find that once the training data has been collected and the model has been initiated, simulation times are reduced from hours to mere seconds while maintaining remarkable accuracy. Despite this promise, we also try to lay bare the current challenges of applying machine learning to these and more complex colloidal systems.

Systems engineering analysis of diagnostic referral closed-loop processes

BACKGROUND

Closing loops to complete diagnostic referrals remains a significant patient safety problem in most health systems, with 65%-73% failure rates and significant delays common despite years of improvement efforts, suggesting new approaches may be useful. Systems engineering (SE) methods increasingly are advocated in healthcare for their value in studying and redesigning complex processes.

OBJECTIVE

Conduct a formative SE analysis of process logic, variation, reliability and failures for completing diagnostic referrals originating in two primary care practices serving different demographics, using dermatology as an illustrating use case.

METHODS

An interdisciplinary team of clinicians, systems engineers, quality improvement specialists, and patient representatives collaborated to understand processes of initiating and completing diagnostic referrals. Cross-functional process maps were developed through iterative group interviews with an urban community-based health centre and a teaching practice within a large academic medical centre. Results were used to conduct an engineering process analysis, assess variation within and between practices, and identify common failure modes and potential solutions.

RESULTS

Processes to complete diagnostic referrals involve many sub-standard design constructs, with significant workflow variation between and within practices, statistical instability and special cause variation in completion rates and timeliness, and only 21% of all process activities estimated as value-add. Failure modes were similar between the two practices, with most process activities relying on low-reliability concepts (eg, reminders, workarounds, education and verification/inspection). Several opportunities were identified to incorporate higher reliability process constructs (eg, simplification, consolidation, standardisation, forcing functions, automation and opt-outs).

CONCLUSION

From a systems science perspective, diagnostic referral processes perform poorly in part because their fundamental designs are fraught with low-reliability characteristics and mental models, including formalised workaround and rework activities, suggesting a need for different approaches versus incremental improvement of existing processes. SE perspectives and methods offer new ways of thinking about patient safety problems, failures and potential solutions.

Patient engagement in system redesign teams: a process of social identity

PURPOSE

Studies demonstrate how patient roles in system redesign teams reflect a continuum of involvement and influence. This research shows the process by which patients move through this continuum and effectively engage within redesign projects.

DESIGN/METHODOLOGY/APPROACH

The authors studied members of redesign teams, consisting of 5-10 members: clinicians, systems engineers, health system staff and patient(s), from three health systems working on separate projects in a patient safety learning lab. Weekly team meetings were observed, January 2016-April 2018, 17 semi-structured interviews were conducted and findings through a patient focus group were refined. Grounded theory was used to analyze field notes and transcripts.

FINDINGS

Results show how the social identity process enables patients to move through stages in a patient engagement continuum (informant, partner and active change agent). Initially, patient and team member perceptions of the patient's role influence their respective behaviors (activating, directing, framing and sharing). Subsequently, patient and team member behaviors influence patient contributions on the team, which can redefine patient and team member perceptions of the patient's role.

ORIGINALITY/VALUE

As health systems grow increasingly complex and become more interested in responding to patient expectations, understanding how to effectively engage patients on redesign teams gains importance. This research investigates how and why patient engagement on redesign teams changes over time and what makes different types of patient roles valuable for team objectives. Findings have implications for how redesign teams can better prepare, anticipate and support the changing role of engaged patients.

Systems Analysis of a Dedicated Ambulatory Respiratory Unit for Seeing and Ensuring Follow-up of Patients With COVID-19 Symptoms

COVID-19 necessitated significant care redesign, including new ambulatory workflows to handle surge volumes, protect patients and staff, and ensure timely reliable care. Opportunities also exist to harvest lessons from workflow innovations to benefit routine care. We describe a dedicated COVID-19 ambulatory unit for closing testing and follow-up loops characterized by standardized workflows and electronic communication, documentation, and order placement. More than 85% of follow-ups were completed within 24 hours, with no observed staff, nor patient infections associated with unit operations. Identified issues include role confusion, staffing and gatekeeping bottlenecks, and patient reluctance to visit in person or discuss concerns with phone screeners.

Antibiotic stewardship from a decision-making, behavioral economics, and incentive design perspective

Antibiotic-resistant infections cause over 20 thousand deaths and $20 billion annually in the United States. Antibiotic prescribing decision making can be described as a "tragedy of the commons" behavioral economics problem, for which individual best interests affecting human decision-making lead to suboptimal societal antibiotic overuse. In 2015, the U.S. federal government announced a $1.2 billion National Action Plan to combat resistance and reduce antibiotic use by 20% in inpatient settings and 50% in outpatient settings by 2020. We develop and apply a behavioral economics model based on game theory and "tragedy of the commons" concepts to help illustrate why rational individuals may not practice ideal stewardship and how to potentially structure three specific alternate approaches to accomplish these objectives (collective cooperative management, usage taxes, resistance penalties), based on Ostrom's economic governance principles. Importantly, while each approach can effectively incentivize ideal stewardship, the latter two do so with 10-30% lower utility to all providers. Encouraging local or state-level self-managed cooperative stewardship programs thus is preferred to national taxes and penalties, in contrast with current trends and with similar implications in other countries.

Early recognition and response to increases in surgical site infections using optimized statistical process control charts-the Early 2RIS Trial: a multicenter cluster randomized controlled trial with stepped wedge design

Background

Surgical site infections (SSIs) cause significant patient suffering. Surveillance and feedback of SSI rates is an evidence-based strategy to reduce SSIs, but traditional surveillance methods are slow and prone to bias. The objective of this cluster randomized controlled trial (RCT) is to determine if using optimized statistical process control (SPC) charts for SSI surveillance and feedback lead to a reduction in SSI rates compared to traditional surveillance.

Methods

The Early 2RIS Trial is a prospective, multicenter cluster RCT using a stepped wedge design. The trial will be performed in 29 hospitals in the Duke Infection Control Outreach Network (DICON) and 105 clusters over 4 years, from March 2016 through February 2020; year one represents a baseline period; thereafter, 8–9 clusters will be randomized to intervention every 3 months over a 3-year period using a stepped wedge randomization design. All patients who undergo one of 13 targeted procedures at study hospitals will be included in the analysis; these procedures will be included in one of six clusters: cardiac, orthopedic, gastrointestinal, OB-GYN, vascular, and spinal. All clusters will undergo traditional surveillance for SSIs; once randomized to intervention, clusters will also undergo surveillance and feedback using optimized SPC charts. Feedback on surveillance data will be provided to all clusters, regardless of allocation or type of surveillance. The primary endpoint is the difference in rates of SSI between the SPC intervention compared to traditional surveillance and feedback alone.

Discussion

The traditional approach for SSI surveillance and feedback has several major deficiencies because SSIs are rare events. First, traditional statistical methods require aggregation of measurements over time, which delays analysis until enough data accumulate. Second, traditional statistical tests and resulting p values are difficult to interpret. Third, analyses based on average SSI rates during predefined time periods have limited ability to rapidly identify important, real-time trends. Thus, standard analytic methods that compare average SSI rates between arbitrarily designated time intervals may not identify an important SSI rate increase on time unless the “signal” is very strong. Therefore, novel strategies for early identification and investigation of SSI rate increases are needed to decrease SSI rates. While SPC charts are used throughout industry and healthcare to improve and optimize processes, including other types of healthcare-associated infections, they have not been evaluated as a tool for SSI surveillance and feedback in a randomized trial.

Trial registration

ClinicalTrials.govNCT03075813, Registered March 9, 2017.

A mixed methods analysis of access barriers to dermatology care in a rural state

AIMS

To identify significant patient and system access barriers and facilitators to dermatology care in one rural health system with limited dermatology appointment availability.

DESIGN

Mixed methods study using data from electronic medical records, patient surveys, stakeholder semi-structured interviews, and service area dermatologist demographics. Retrospective data were collected between 1 January 2017-1 March 2018, and interviews and surveys were conducted between June 1-August 31, 2018. Participants were recruited from two primary care practices in one rural Maine regional health system.

METHODS

Findings from thematic analyses, descriptive statistics, and statistical modelling were integrated using Chi-square tests for homogeneity to develop a unified understanding. Statistical modelling using odd-ratio logistic and linear regression were performed for each outcome variable of interest.

RESULTS

Urgent referrals by primary care increased the likelihood of dermatology care overall (OR: 6.771; p = .007) and at nearby sites with limited availability (OR: 4.024; p = .024), but not at geographically further sites with higher capacities (p = .844). Referral under-diagnosis occurred in 20.8% of those biopsied. Older (p = .041) or non-working (p = .021) patients were more likely to remain unevaluated than seek more available but geographically further care.

CONCLUSIONS

In rural areas with scarce appointment availability, primary care provider diagnostic accuracy may be an important barrier of dermatology care receipt and health outcomes, especially among at-risk populations.

IMPACT

Although melanoma mortality rates are decreasing throughout the US, little is known about why rates in Maine continue to rise. This study applied a comprehensive approach to identify several patient and system access barriers to dermatology care in one underserved rural regional health system. While specific to this population and large service area, these findings will inform improvement efforts here and support broader future research efforts aimed at understanding and improving health outcomes in this rural state.

Potential Community and Campus Covid-19 Outcomes Under University and College Reopening Scenarios

BACKGROUND

Significant uncertainty exists in many countries about the safety of, and best strategies for, reopening college and university campuses until the Covid-19 pandemic is better controlled. Little also is known about the effects on-campus students may have on local higher-risk communities. We aimed to estimate potential community and campus Covid-19 exposures, infections, and mortality due to various university reopening and precaution plans under current ranges of assumptions and uncertainties.

METHODS

We developed and calibrated campus-only, community-only, and campus-x-community epidemic differential equation and agent-based models. Input parameters for campus and surrounding communities were estimated via published and grey literature, scenario development, expert opinion, accuracy optimization algorithms, and Monte Carlo simulation; models were cross-validated against each other using February-June 2020 data from heterogeneous U.S. counties and states. Campus opening plans (spanning various fully open, hybrid, and fully virtual approaches) were identified from websites and publications. All scenarios were simulated assuming 16-week semesters and estimated ranges for Covid-19 prevalence among community residents and arriving students, precaution compliance, contact frequency, virus attack rates, and tracing and isolation effectiveness. Additional student and community exposures, infections, and mortality were estimated under each scenario, with 10% trimmed medians, standard deviations, and probability intervals computed to omit extreme outlier scenarios. Factorial analyses were conducted to identify intervention inputs with largest and smallest effects.

RESULTS

As a base case with no precautions (or no compliance), predicted 16-week student infections and mortality under normal operations ranged significantly from 471 to 9,495 (median: 2,286, SD: 2,627) and 0 to 123 (median: 9, SD: 14) per 10,000 students, respectively. The maximum active exposures across a semester was 15.76% of all students warranting tracing. Total additional community exposures, infections, and mortality ranged from 1 to 187, 13 to 820, and 1 to 21 per 10,000 residents, respectively. 1% and 5% of on-campus students were infected after a mean (SD) of 11 (3) and 76 (17) days, respectively; >10% students infected by the end of a semester in 34.8% of scenarios, with the greatest increase (first inflection point) occurring on aver-age on day 84 (SD: 10.2 days). Common reopening precautions reduced infections by 24% to 26% and mortality by 36% to 50% in both populations. Uncertainties in many factors, however, produced tremendous variability in all results, ranging from medians by -67% to +342%.

CONCLUSIONS

Consequences on community and student Covid-19 exposures, infections, and mortality of reopening physical campuses are very highly unpredictable, depending on a combination of random chance, controllable (e.g. physical layouts), and uncontrollable (e.g. human behavior) factors. Implications include needs for criteria to adapt campus operations mid-semester, methods to detect when necessary, and contingency plans for doing so.

Assessing the cognitive and work load of an inpatient safety dashboard in the context of opioid management

For health information technology to realize its potential to improve flow, care, and patient safety, applications should be intuitive to use and burden neutral for frontline clinicians. We assessed the impact of a patient safety dashboard on clinician cognitive and work load within a simulated information-seeking task for safe inpatient opioid medication management. Compared to use of an electronic health record for the same task, the dashboard was associated with significantly reduced time on task, mouse clicks, and mouse movement (each p < 0.001), with no significant increases in cognitive load nor task inaccuracy. Cognitive burden was higher for users with less experience, possibly partly attributable to usability issues identified during this study. Findings underscore the importance of assessing the usability, cognitive, and work load analysis during the design and implementation of health information technology applications.

Use, Perceived Usability, and Barriers to Implementation of a Patient Safety Dashboard Integrated within a Vendor EHR

BACKGROUND

Preventable adverse events continue to be a threat to hospitalized patients. Clinical decision support in the form of dashboards may improve compliance with evidence-based safety practices. However, limited research describes providers' experiences with dashboards integrated into vendor electronic health record (EHR) systems.

OBJECTIVE

This study was aimed to describe providers' use and perceived usability of the Patient Safety Dashboard and discuss barriers and facilitators to implementation.

METHODS

The Patient Safety Dashboard was implemented in a cluster-randomized stepped wedge trial on 12 units in neurology, oncology, and general medicine services over an 18-month period. Use of the Dashboard was tracked during the implementation period and analyzed in-depth for two 1-week periods to gather a detailed representation of use. Providers' perceptions of tool usability were measured using the Health Information Technology Usability Evaluation Scale (rated 1-5). Research assistants conducted field observations throughout the duration of the study to describe use and provide insight into tool adoption.

RESULTS

The Dashboard was used 70% of days the tool was available, with use varying by role, service, and time of day. On general medicine units, nurses logged in throughout the day, with many logins occurring during morning rounds, when not rounding with the care team. Prescribers logged in typically before and after morning rounds. On neurology units, physician assistants accounted for most logins, accessing the Dashboard during daily brief interdisciplinary rounding sessions. Use on oncology units was rare. Satisfaction with the tool was highest for perceived ease of use, with attendings giving the highest rating (4.23). The overall lowest rating was for quality of work life, with nurses rating the tool lowest (2.88).

CONCLUSION

This mixed methods analysis provides insight into the use and usability of a dashboard tool integrated within a vendor EHR and can guide future improvements and more successful implementation of these types of tools.

Access to specialty healthcare in urban versus rural US populations: a systematic literature review

Background

Access to healthcare is a poorly defined construct, with insufficient understanding of differences in facilitators and barriers between US urban versus rural specialty care. We summarize recent literature and expand upon a prior conceptual access framework, adapted here specifically to urban and rural specialty care.

Methods

A systematic review was conducted of literature within the CINAHL, Medline, PubMed, PsycInfo, and ProQuest Social Sciences databases published between January 2013 and August 2018. Search terms targeted peer-reviewed academic publications pertinent to access to US urban or rural specialty healthcare. Exclusion criteria produced 67 articles. Findings were organized into an existing ten-dimension care access conceptual framework where possible, with additional topics grouped thematically into supplemental dimensions.

Results

Despite geographic and demographic differences, many access facilitators and barriers were common to both populations; only three dimensions did not contain literature addressing both urban and rural populations. The most commonly represented dimensions were availability and accommodation, appropriateness, and ability to perceive. Four new identified dimensions were: government and insurance policy, health organization and operations influence, stigma, and primary care and specialist influence.

Conclusions

While findings generally align with a preexisting framework, they also suggest several additional themes important to urban versus rural specialty care access.

Large-scale empirical optimisation of statistical control charts to detect clinically relevant increases in surgical site infection rates

OBJECTIVE

Surgical site infections (SSIs) are common costly hospital-acquired conditions. While statistical process control (SPC) use in healthcare has increased, limited rigorous empirical research compares and optimises these methods for SSI surveillance. We sought to determine which SPC chart types and design parameters maximise the detection of clinically relevant SSI rate increases while minimising false alarms.

DESIGN

Systematic retrospective data analysis and empirical optimisation.

METHODS

We analysed 12 years of data on 13 surgical procedures from a network of 58 community hospitals. Statistically significant SSI rate increases (signals) at individual hospitals initially were identified using 50 different SPC chart variations (Shewhart or exponentially weighted moving average, 5 baseline periods, 5 baseline types). Blinded epidemiologists evaluated the clinical significance of 2709 representative signals of potential outbreaks (out of 5536 generated), rating them as requiring 'action' or 'no action'. These ratings were used to identify which SPC approaches maximised sensitivity and specificity within a broader set of 3600 individual chart variations (additional baseline variations and chart types, including moving average (MA), and five control limit widths) and over 32 million dual-chart combinations based on different baseline periods, reference data (network-wide vs local hospital SSI rates), control limit widths and other calculation considerations. Results were validated with an additional year of data from the same hospital cohort.

RESULTS

The optimal SPC approach to detect clinically important SSI rate increases used two simultaneous MA charts calculated using lagged rolling baseline windows and 1 SD limits. The first chart used 12-month MAs with 18-month baselines and best identified small sustained increases above network-wide SSI rates. The second chart used 6-month MAs with 3-month baselines and best detected large short-term increases above individual hospital SSI rates. This combination outperformed more commonly used charts, with high sensitivity (0.90; positive predictive value=0.56) and practical specificity (0.67; negative predictive value=0.94).

CONCLUSIONS

An optimised combination of two MA charts had the best performance for identifying clinically relevant small but sustained above-network SSI rates and large short-term individual hospital increases.

85. Use of Dual Statistical Process Control Charts for Early Detection of Surgical Site Infection Outbreaks at a Community Hospital Network

Background

We recently showed that the empirical use of a combination of 2 moving average (MA) statistical process control (SPC) charts was highly sensitive and specific for detecting potentially important increases in surgical site infection (SSI) rates. We performed this follow-up study to examine the performance of these same SPC charts when applied to known SSI outbreaks.

Methods

We retrospectively applied 2 MA SPC charts to all 30 SSI outbreaks investigated from 2007 to 2015 in a network of over 50 community hospitals. These outbreaks were detected via routine SSI surveillance activities that occurred in the network. We reviewed prior outbreak investigation documentation to determine the estimated time of outbreak onset and time of traditional surveillance outbreak detection. The first SPC chart utilized procedure-specific, composite SSI data from the hospital network for its baseline; the baseline for the second chart was calculated from SSI data from the outbreak hospital undergoing analysis. Both charts used rolling baseline windows but varied in baseline window size, rolling baseline lag, and MA window size. SPC chart outbreak detection occurred when either chart had a data point above the upper control limit of 1 standard deviation. Time of SPC detection was compared with both time of outbreak onset and time of traditional surveillance detection.

Results

With the dual chart approach, SPC detected all 30 outbreaks, including detection of 25 outbreaks (83%) prior to their estimated onset (Figure 1). SPC detection occurred a median of 16 months (interquartile range, 12–21 months) prior to the date of traditional outbreak detection, which never occurred prior to outbreak onset. Both individual SPC charts exhibited at least 90% sensitivity in outbreak detection, but the dual chart approach showed superior sensitivity and speed of detection (Figure 2).

Conclusion

A strategy that employed optimized, dual MA SPC charts retrospectively detected all SSI outbreaks that occurred over 9 years in a network of community hospitals. SPC outbreak detection occurred earlier than traditional surveillance detection. These optimized SPC charts merit prospective study to evaluate their ability to promote early detection of SSI clusters in real-world scenarios.

Disclosures

All Authors: No reported Disclosures.

1232. Potential Health and Cost Outcomes of Optimized Statistical Process Control Use for Surgical Site Infection Surveillance

Background

Surgical site infections (SSIs) are common (160,000–300,000 per year in the United States) and costly ($6,000–$25,500 per event) healthcare-associated infections with potentially lethal outcomes (2.1%–6.7% mortality rate). A prior analysis by our group suggested that statistical process control (SPC) can detect SSI outbreaks earlier than traditional epidemiological surveillance methods. This study aimed to quantify the potential impact of SPC surveillance on patient outcomes (prevented SSIs and deaths) and healthcare costs.

Methods

We retrospectively analyzed 30 SSI outbreaks occurring over a period of 8 years in a network of 50 community hospitals from the Southeastern United States. We applied 24 control chart variations, including 2 optimized for SSI surveillance, 6 with expert-defined pre-outbreak baselines (used in our pilot study), 4 with lagged rolling baselines (idem), and 12 common practice ones (using rolling baselines with no lag or fixed baselines). The charts used procedure-specific data from either the outbreak hospital or the entire network to compute baseline SSI rates. We calculated the average SSI rates during, before and after the outbreaks, and the months elapsed between SPC and traditional detection. We then used these values to estimate the number of SSIs that could have been prevented by SPC, and corresponding deaths avoided and cost savings (Figure 1).

Results

Optimized charts detected 96% of the outbreaks earlier than traditional surveillance, while pilot study and common practice charts did so only 65% (58%) of the time (Figure 2). Optimized charts could potentially prevent 15.2 SSIs, 0.64 deaths, and save $226,000 in excess care costs per outbreak. Overall, charts using network baselines performed better than those relying on local hospital data. Commonly used variations were the least effective, but were still able to improve on traditional surveillance (Figure 3).

Conclusion

SPC methods provide a great opportunity to prevent infections and deaths and generate cost savings, ultimately improving patient safety and care quality. While common practice SPC charts can also speed up outbreak detection, optimized SPC methods have a significantly higher potential to prevent SSIs and reduce healthcare costs.

Disclosures

All authors: No reported disclosures.

2426. Performance of Statistical Process Control Charts for Detecting Clinically-Significant Increases in Clostridium difficile Infection Rates

Background

Clostridium difficile infections (CDIs) are the most common type of healthcare-associated infection in the United States, with an estimated annual incidence of 500,000 cases and excess healthcare costs of $5 billion per year. The prevalence and severity of CDIs have been increasing in recent years, making it of vital importance to detect outbreaks sufficiently early to minimize negative health outcomes. Statistical process control (SPC) methods have proven to be a versatile tool in healthcare, enabling near real-time monitoring of adverse events rates and thereby improving patients’ health. The aim of this study was to investigate the performance of SPC in detecting clinically significant increases in CDI rates.

Methods

We retrospectively analyzed monthly CDI rates at 6 community hospitals in the Duke Infection Control Outreach Network from 2009–2017. Detected CDIs were stratified by surveillance system (LabID or traditional), infection source, recurrence type, and diagnostic test (nucleic acid amplification or enzyme-linked immunosorbent assay). Recurrent and community-associated CDIs were excluded from all analyses. Several variations of Shewhart and exponentially-weighted moving average (EWMA) u-charts were applied to each hospital (Figure 1), including using different baseline types (global, fixed, or rolling) and baseline data streams (hospital or network-wide). To help assess chart performance, epidemiologists determined the clinical relevance (yes/no) of 167 statistical signals generated using earlier iterations of these charts. Performance was quantified via sensitivity, specificity, and accuracy.

Results

EWMA u-charts with global network-wide baselines performed the best (Figure 2), detecting the largest number of clinically relevant signals (56% sensitivity) with high specificity (96%). Charts utilizing network-wide baselines were generally more accurate than those using local hospital data for that purpose (accuracy of 46–72% vs. 43–45%). Similarly, charts with fixed baselines performed better than those with rolling ones (accuracy of 43–62% vs. 43–47%).

Conclusion

SPC charts are easily applicable to CDI surveillance; however, their parameters would need to be optimized to maximize utility.

Disclosures

All authors: No reported disclosures.

Performance of statistical process control methods for regional surgical site infection surveillance: a 10-year multicentre pilot study

BACKGROUND

Traditional strategies for surveillance of surgical site infections (SSI) have multiple limitations, including delayed and incomplete outbreak detection. Statistical process control (SPC) methods address these deficiencies by combining longitudinal analysis with graphical presentation of data.

METHODS

We performed a pilot study within a large network of community hospitals to evaluate performance of SPC methods for detecting SSI outbreaks. We applied conventional Shewhart and exponentially weighted moving average (EWMA) SPC charts to 10 previously investigated SSI outbreaks that occurred from 2003 to 2013. We compared the results of SPC surveillance to the results of traditional SSI surveillance methods. Then, we analysed the performance of modified SPC charts constructed with different outbreak detection rules, EWMA smoothing factors and baseline SSI rate calculations.

RESULTS

Conventional Shewhart and EWMA SPC charts both detected 8 of the 10 SSI outbreaks analysed, in each case prior to the date of traditional detection. Among detected outbreaks, conventional Shewhart chart detection occurred a median of 12 months prior to outbreak onset and 22 months prior to traditional detection. Conventional EWMA chart detection occurred a median of 7months prior to outbreak onset and 14 months prior to traditional detection. Modified Shewhart and EWMA charts additionally detected several outbreaks earlier than conventional SPC charts. Shewhart and SPC charts had low false-positive rates when used to analyse separate control hospital SSI data.

CONCLUSIONS

Our findings illustrate the potential usefulness and feasibility of real-time SPC surveillance of SSI to rapidly identify outbreaks and improve patient safety. Further study is needed to optimise SPC chart selection and calculation, statistical outbreak detection rules and the process for reacting to signals of potential outbreaks.

Can improved specialty access moderate emergency department overuse?: Effect of neurology appointment delays on ED visits

Background

Delayed access to specialty care may increase inappropriate emergency department (ED) visits. However, the details of this relationship after referral to a specialist are unknown.

Methods

The correlations in an academic medical center between time to new neurology patient appointments and nonurgent ED use are explored in this study. Access was measured as the number of days between the scheduling and outpatient appointment dates. A series of statistical analyses including correlation analysis, regressions, and hypothesis tests were conducted to investigate possible associations between delayed access to specialty care and ED visits, as well as the effect of ED visits on specialty care cancellation and no-show rates.

Results

Of 19,505 new neurology patients, 310 visited an ED prior to their appointment, 95.2% (295) of whom had poor access (defined here as exceeding 21 days). Patients with access >21 days for new visits were 6.6 times more likely to visit the ED before their appointment date, 19% within the first week after scheduling. Patients who visited the ED between their booking and appointment dates were 2.3 times more likely to cancel or fail to attend their appointment.

Conclusion

These results suggest that long access delays in specialty referrals can significantly increase ED costs and congestion. Further studies in other specialties to explore this relationship are warranted.

Review of Research Trends and Methods in Nano Environmental, Health, and Safety Risk Analysis

Despite the many touted benefits of nanomaterials, concerns remain about their possible environmental, health, and safety (EHS) risks in terms of their toxicity, long-term accumulation effects, or dose-response relationships. The published studies on EHS risks of nanomaterials have increased significantly over the past decade and half, with most focused on nanotoxicology. Researchers are still learning about health consequences of nanomaterials and how to make environmentally responsible decisions regarding their production. This article characterizes the scientific literature on nano-EHS risk analysis to map the state-of-the-art developments in this field and chart guidance for the future directions. First, an analysis of keyword co-occurrence networks is investigated for nano-EHS literature published in the past decade to identify the intellectual turning points and research trends in nanorisk analysis studies. The exposure groups targeted in emerging nano-EHS studies are also assessed. System engineering methods for risk, safety, uncertainty, and system reliability analysis are reviewed, followed by detailed descriptions where applications of these methods are utilized to analyze nanomaterial EHS risks. Finally, the trends, methods, future directions, and opportunities of system engineering methods in nano-EHS research are discussed. The analysis of nano-EHS literature presented in this article provides important insights on risk assessment and risk management tools associated with nanotechnology, nanomanufacturing, and nano-enabled products.

Thresholds and accuracy in screening tools for early detection of psychopathology

BACKGROUND

The accuracy of any screening instrument designed to detect psychopathology among children is ideally assessed through rigorous comparison to 'gold standard' tests and interviews. Such comparisons typically yield estimates of what we refer to as 'standard indices of diagnostic accuracy', including sensitivity, specificity, positive predictive value (PPV), and negative predictive value. However, whereas these statistics were originally designed to detect binary signals (e.g., diagnosis present or absent), screening questionnaires commonly used in psychology, psychiatry, and pediatrics typically result in ordinal scores. Thus, a threshold or 'cut score' must be applied to these ordinal scores before accuracy can be evaluated using such standard indices. To better understand the tradeoffs inherent in choosing a particular threshold, we discuss the concept of 'threshold probability'. In contrast to PPV, which reflects the probability that a child whose score falls at or above the screening threshold has the condition of interest, threshold probability refers specifically to the likelihood that a child whose score is equal to a particular screening threshold has the condition of interest.

METHOD

The diagnostic accuracy and threshold probability of two well-validated behavioral assessment instruments, the Child Behavior Checklist Total Problem Scale and the Strengths and Difficulties Questionnaire total scale were examined in relation to a structured psychiatric interview in three de-identified datasets.

RESULTS

Although both screening measures were effective in identifying groups of children at elevated risk for psychopathology in all samples (odds ratios ranged from 5.2 to 9.7), children who scored at or near the clinical thresholds that optimized sensitivity and specificity were unlikely to meet criteria for psychopathology on gold standard interviews.

CONCLUSIONS

Our results are consistent with the view that screening instruments should be interpreted probabilistically, with attention to where along the continuum of positive scores an individual falls.

Health systems engineering fellowship: curriculum and program development

Industrial engineering and related disciplines have been used widely in improvement efforts in many industries. These approaches have been less commonly attempted in health care. One factor limiting application is the limited workforce resulting from a lack of specific education and professional development in health systems engineering (HSE). The authors describe the development of an HSE fellowship within the United States Department of Veterans Affairs, Veterans Health Administration (VA). This fellowship includes a novel curriculum based on specifically established competencies for HSE. A 1-year HSE curriculum was developed and delivered to fellows at several VA engineering resource centers over several years. On graduation, a majority of the fellows accepted positions in the health care field. Challenges faced in developing the fellowship are discussed. Advanced educational opportunities in applied HSE have the potential to develop the workforce capacity needed to improve the quality of health care.

Generalizability of a simple approach for predicting hospital admission from an emergency department

OBJECTIVES

The objective was to test the generalizability, across a range of hospital sizes and demographics, of a previously developed method for predicting and aggregating, in real time, the probabilities that emergency department (ED) patients will be admitted to a hospital inpatient unit.

METHODS

Logistic regression models were developed that estimate inpatient admission probabilities of each patient upon entering an ED. The models were based on retrospective development (n = 4,000 to 5,000 ED visits) and validation (n = 1,000 to 2,000 ED visits) data sets from four heterogeneous hospitals. Model performance was evaluated using retrospective test data sets (n = 1,000 to 2,000 ED visits). For one hospital the developed model also was applied prospectively to a test data set (n = 910 ED visits) coded by triage nurses in real time, to compare results to those from the retrospective single investigator-coded test data set.

RESULTS

The prediction models for each hospital performed reasonably well and typically involved just a few simple-to-collect variables, which differed for each hospital. Areas under receiver operating characteristic curves (AUC) ranged from 0.80 to 0.89, R(2) correlation coefficients between predicted and actual daily admissions ranged from 0.58 to 0.90, and Hosmer-Lemeshow goodness-of-fit statistics of model accuracy had p > 0.01 with one exception. Data coded prospectively by triage nurses produced comparable results.

CONCLUSIONS

The accuracy of regression models to predict ED patient admission likelihood was shown to be generalizable across hospitals of different sizes, populations, and administrative structures. Each hospital used a unique combination of predictive factors that may reflect these differences. This approach performed equally well when hospital staff coded patient data in real time versus the research team retrospectively.

Predicting emergency department inpatient admissions to improve same-day patient flow

OBJECTIVES

The objectives were to evaluate three models that use information gathered during triage to predict, in real time, the number of emergency department (ED) patients who subsequently will be admitted to a hospital inpatient unit (IU) and to introduce a new methodology for implementing these predictions in the hospital setting.

METHODS

Three simple methods were compared for predicting hospital admission at ED triage: expert opinion, naïve Bayes conditional probability, and a generalized linear regression model with a logit link function (logit-linear). Two months of data were gathered from the Boston VA Healthcare System's 13-bed ED, which receives approximately 1,100 patients per month. Triage nurses were asked to estimate the likelihood that each of 767 triaged patients from that 2-month period would be admitted after their ED treatment, by placing them into one of six categories ranging from low to high likelihood. Logit-linear regression and naïve Bayes models also were developed using retrospective data and used to estimate admission probabilities for each patient who entered the ED within a 2-month time frame, during triage hours (1,160 patients). Predictors considered included patient age, primary complaint, provider, designation (ED or fast track), arrival mode, and urgency level (emergency severity index assigned at triage).

RESULTS

Of the three methods considered, logit-linear regression performed the best in predicting total bed need, with a receiver operating characteristic (ROC) area under the curve (AUC) of 0.887, an R(2) of 0.58, an average estimation error of 0.19 beds per day, and on average roughly 3.5 hours before peak demand occurred. Significant predictors were patient age, primary complaint, bed type designation, and arrival mode (p < 0.0001 for all factors). The naïve Bayesian model had similar positive predictive value, with an AUC of 0.841 and an R(2) of 0.58, but with average difference in total bed need of approximately 2.08 per day. Triage nurse expert opinion also had some predictive capability, with an R(2) of 0.52 and an average difference in total bed need of 1.87 per day.

CONCLUSIONS

Simple probability models can reasonably predict ED-to-IU patient volumes based on basic data gathered at triage. This predictive information could be used for improved real-time bed management, patient flow, and discharge processes. Both statistical models were reasonably accurate, using only a minimal number of readily available independent variables.

Foundations of opioid risk management

Increased abuse and diversion of prescription opioids has been a consequence of the increased availability of opioids to address the widespread problem of undertreated pain. Opioid risk management refers to the effort to minimize harms associated with opioid therapy while maintaining appropriate access to therapy. Management of these linked public health issues requires a coordinated and balanced effort among a disparate group of stakeholders at the federal, state, industry, practitioner, and patient levels. This paper reviews the principles of opioid risk management by examining the epidemiology of prescription opioid abuse in the United States; identifying key stakeholders involved in opioid risk management and their responsibilities for managing or monitoring opioid abuse and diversion; and summarizing the mechanisms currently used to monitor and address prescription opioid abuse. Limitations of current approaches, and emerging directions in opioid risk management, are also presented.

Statistical process control as a tool for research and healthcare improvement

Improvement of health care requires making changes in processes of care and service delivery. Although process performance is measured to determine if these changes are having the desired beneficial effects, this analysis is complicated by the existence of natural variation-that is, repeated measurements naturally yield different values and, even if nothing was done, a subsequent measurement might seem to indicate a better or worse performance. Traditional statistical analysis methods account for natural variation but require aggregation of measurements over time, which can delay decision making. Statistical process control (SPC) is a branch of statistics that combines rigorous time series analysis methods with graphical presentation of data, often yielding insights into the data more quickly and in a way more understandable to lay decision makers. SPC and its primary tool-the control chart-provide researchers and practitioners with a method of better understanding and communicating data from healthcare improvement efforts. This paper provides an overview of SPC and several practical examples of the healthcare applications of control charts.

Risk-adjusted sequential probability ratio tests and longitudinal surveillance methods

Regardless of the exact method employed, the application of statistical process controL SPRTs, or related longitudinal analysis methods can significantly improve the ability to monitor clinical processes and outcomes. Incorporation and adaptation of risk-adjustment and rare events into these methods represent important contributions to their use in health care. Fostering greater and more widespread use of these methods, however, remains a significant challenge. Hopefully studies such as those by Spiegelhalter et al. will lead to more awareness of their value for contributing to a safer health care system.

Binary cumulative sums and moving averages in nosocomial infection cluster detection

Clusters of nosocomial infection often occur undetected, at substantial cost to the medical system and individual patients. We evaluated binary cumulative sum (CUSUM) and moving average (MA) control charts for automated detection of nosocomial clusters. We selected two outbreaks with genotyped strains and used resistance as inputs to the control charts. We identified design parameters for the CUSUM and MA (window size, k, alpha, Beta, p(0), p(1)) that detected both outbreaks, then calculated an associated positive predictive value (PPV) and time until detection (TUD) for sensitive charts. For CUSUM, optimal performance (high PPV, low TUD, fully sensitive) was for 0.1 < or = alpha < or = 0.25 and 0.2 < or = Beta < or = 0.25, with p(0) = 0.05, with a mean TUD of 20 (range 8-43) isolates. Mean PPV was 96.5% (relaxed criteria) to 82.6% (strict criteria). MAs had a mean PPV of 88.5% (relaxed criteria) to 46.1% (strict criteria). CUSUM and MA may be useful techniques for automated surveillance of resistant infections.

Controlling methicillin-resistant Staphylococcus aureus: a feedback approach using annotated statistical process control charts

OBJECTIVES

To investigate the benefit of a hospitalwide feedback program regarding methicillin-resistant Staphylococcus aureus (MRSA), using annotated statistical process control charts.

DESIGN

Retrospective and prospective analysis of MRSA rates using statistical process control charts.

PARTICIPANTS

Twenty-four medical, medical specialty, surgical, intensive care, and cardiothoracic care wards and units at four Glasgow Royal Infirmary hospitals.

METHODS

Annotated control charts were applied to prospective and historical monthly data on MRSA cases from each ward and unit during a 46-month period from January 1997 through September 2000. Results were fed back from December 1999 and then on a regular monthly basis to medical staff, ward managers, senior managers, and hotel services.

RESULTS

Monthly reductions in the MRSA acquisition rate started 2 months after the introduction of the feedback program and have continued to the present time. The overall MRSA rate currently is approximately 50% lower than when the program began and has become more consistent and less variable within departments throughout Glasgow Royal Infirmary. The control charts have helped to detect rate changes and manage resources more effectively. Medical and nursing staff and managers also report that they find this the most positive form of MRSA feedback they have received.

CONCLUSIONS

Feedback programs that provide current information to front-line staff and incorporate annotated control charts can be effective in reducing the rate of MRSA.

Number-between g-type statistical quality control charts for monitoring adverse events

Alternate Shewhart-type statistical control charts, called "g" and "h" charts, are developed and evaluated for monitoring the number of cases between hospital-acquired infections and other adverse events, such as heart surgery complications, catheter-related infections, surgical site infections, contaminated needle sticks, and other iatrically induced outcomes. These new charts, based on inverse sampling from geometric and negative binomial distributions, are simple to use and can exhibit significantly greater detection power over conventional binomial-based approaches, particularly for infrequent events and low "defect" rates. A companion article illustrates several interesting properties of these charts and design modifications that significantly can improve their statistical properties, operating characteristics, and sensitivity.

Performance of number-between g-type statistical control charts for monitoring adverse events

Alternate Shewhart-type statistical control charts, called "g" and "h" charts, have been developed for monitoring the number of cases between hospital-acquired infections and other adverse events, such as heart surgery complications, catheter-related infections, surgical site infections, contaminated needle sticks, medication errors, and other care-induced concerns. This article investigates the statistical properties of these new charts and illustrates several design considerations that significantly can improve their operating characteristics and sensitivity, including the use of within-limit rules, a new in-control rule, redefined Bernoulli trials, and probability-based limits. These new charts are based on inverse sampling from geometric and negative binomial distributions, are simple for practitioners to use, and in some cases exhibit significantly greater detection power over conventional binomial-based approaches, particularly for infrequent events and low "defect" rates.

Development of a web-based multifacility healthcare surveillance information system

This article describes recent work to develop a Web-based statistical surveillance information system to monitor in real time the status of the U.S. Air Force's worldwide healthcare network. The intent is to incorporate statistical and related methods in order to identify unusual events and patterns of concern in large, highly distributed organizations. The work recently received an award from Vice President Gore for reinventing government.

Statistical quality control methods in infection control and hospital epidemiology, Part II: Chart use, statistical properties, and research issues

This is the second in a two-part series discussing and illustrating the application of statistical process control (SPC) in hospital epidemiology. The basic philosophical and theoretical foundations of statistical quality control and their relation to epidemiology are emphasized in order to expand the mutual understanding and cross-fertilization between these two disciplines. Part I provided an overview of the philosophy and general approach of SPC, illustrated common types of control charts, and provided references for further information or statistical formulae. Part II now discusses alternate possible SPC approaches, statistical properties of control charts, chart-design issues and optimal control limit widths, some common misunderstandings, and more advanced issues. The focus of both articles is mostly nonmathematical, emphasizing important concepts and practical examples rather than academic theory and exhaustive calculations.

Statistical quality control methods in infection control and hospital epidemiology, part I: Introduction and basic theory

This article is the first in a two-part series discussing and illustrating the application of statistical process control (SPC) to processes often examined by hospital epidemiologists. The basic philosophical and theoretical foundations of statistical quality control and their relation to epidemiology are emphasized in order to expand mutual understanding and cross-fertilization between these two disciplines. Part I provides an overview of quality engineering and SPC, illustrates common types of control charts, and provides references for further information or statistical formulae. Part II discusses statistical properties of control charts, issues of chart design and optimal control limit widths, alternate possible SPC approaches to infection control, some common misunderstandings, and more advanced issues. The focus of both articles is mostly non-mathematical, emphasizing important concepts and practical examples rather than academic theory and exhaustive calculations.