Strategies to enhance remote monitoring adherence among patients with cardiovascular implantable electronic devices

Background

Remote monitoring (RM) of patients with cardiovascular implantable electronic devices (CIEDs) (pacemakers and implantable cardioverter-defibrillators) has a Class 1, Level of Evidence A Heart Rhythm Society recommendation. Yet RM adherence varies widely across settings, and factors associated with variation are not understood.

Objective

The purpose of this study was to identify strategies for supporting RM across Veterans Health Administration (VHA) facilities.

Methods

In a national evaluation, we surveyed and interviewed 27 nurses, medical instrument technicians, and advanced practice providers across 26 VHA facilities (following approximately 15,000 CIED patients). Participants were selected based on overall patient adherence by facility, which ranged from 46%-96%. Questions covered RM adherence strategies, manufacturer resources, organizational characteristics, and workflows for optimizing adherence.

Results

All clinicians reported that RM adherence was extremely important (53.8%), very important (34.6%), or important (11.5%) for improving patient outcomes. High performing facilities prioritized consistent patient education about RM and evaluated nonadherence using dashboards and manufacturer web sites. High performing facilities instituted clear standard operating procedures that defined staff responsibilities and facilitated efficient contact with nonadherent patients and then family members by phone and then mail. Clinicians based at high performing facilities spent twice as many hours per week (9.1) on average managing RM adherence compared to other facilities (4.5). Effective communication (internally and with non-VHA care partners) and use of CIED manufacturer resources were essential. Facilities that were not high performing rarely used these strategies.

Conclusion

Clinicians can support high RM adherence by emphasizing patient education, regularly assessing and addressing nonadherence using staff protocols, and engaging CIED manufacturers.

Study protocol: maintaining preventive care during public health emergencies through effective coordination

BACKGROUND

Screening lies at the heart of preventive care. However, COVID-19 dramatically disrupted routine screening efforts, resulting in excess mortality not directly attributable to COVID-19. Screening rates during COVID varied markedly by facility and clinical condition, suggesting susceptibilities in screening and referral process workflow. To better understand these susceptibilities and identify new practices to mitigate interrupted care, we propose a qualitative study comparing facilities that exhibited high, low, and highly variable performance (respectively) in screening rates before and during the pandemic. We will be guided by Weaver et al.'s multi-team systems (MTS) model of coordination, using cancer and mental health screening rates as exemplars.

METHOD

Qualitative analysis of interviews and focus groups with primary care personnel, leadership, and patients at 10 VA medical centers. We will select sites based on rurality, COVID-19 caseload at the beginning of the pandemic, and performance on five outpatient clinical performance indicators of cancer and mental health screening. Sites will be categorized into one of five screening performance groups: high performers, low performers, improvers, plummeters, and highly variable. We will create process maps for each performance measure to create a workflow baseline and then interview primary care leadership to update the map at each site. We will clinician conduct focus groups to elicit themes regarding clinician coordination patterns (e.g., handoffs), strategies, and barriers/facilitators to screening during COVID. We will also conduct patient interviews to examine their screening experience during this period, for context. All interviews and focus groups will be audio-recorded, transcribed, and enhanced by field notes. We will analyze clinician transcripts and field notes using iterative, rapid analysis. Patient interviews will be analyzed using inductive/deductive content analysis.

DISCUSSION

Our study represents a unique opportunity to inform the multi-team systems literature by identifying specific forms of information exchange, collective problem solving, and decision-making associated with higher and improved clinical performance. Specifically, our study aims to detect the specific points in the screening and referral process most susceptible to disruption and coordination processes that, if changed, will yield the highest value. Findings apply to future pandemics or any event with the potential to disrupt care.

What can we learn from COVID-19?: examining the resilience of primary care teams

Introduction

The COVID-19 pandemic continues to place an unprecedented strain on the US healthcare system, and primary care is no exception. Primary care services have shifted toward a team-based approach for delivering care in the last decade. COVID-19 placed extraordinary stress on primary care teams at the forefront of the pandemic response efforts. The current work applies the science of effective teams to examine the impact of COVID-19-a crisis or adverse event-on primary care team resilience.

Methods

Little empirical research has been done testing the theory of team resilience during an extremely adverse crisis event in an applied team setting. Therefore, we conducted an archival study by using large-scale national data from the Veterans Health Administration to understand the characteristics and performance of 7,023 Patient Aligned Care Teams (PACTs) during COVID-19.

Results

Our study found that primary care teams maintained performance in the presence of adversity, indicating possible team resilience. Further, team coordination positively predicted team performance (B = 0.53) regardless of the level of adversity a team was experiencing.

Discussion

These findings in turn attest to the need to preserve team coordination in the presence of adversity. Results carry implications for creating opportunities for teams to learn and adjust to an adverse event to maintain performance and optimize team-member well-being. Teamwork can act as a protective factor against high levels of workload, burnout, and turnover, and should be studied further for its role in promoting team resilience.

Nurse Leader Perceptions of Data in the Veterans Health Administration: A Qualitative Evaluation

Healthcare systems and nursing leaders aim to make evidence-based nurse staffing decisions. Understanding how nurses use and perceive available data to support safe staffing can strengthen learning healthcare systems and support evidence-based practice, particularly given emerging data availability and specific nursing challenges in data usability. However, current literature offers sparse insight into the nature of data use and challenges in the inpatient nurse staffing management context. We aimed to investigate how nurse leaders experience using data to guide their inpatient staffing management decisions in the Veterans Health Administration, the largest integrated healthcare system in the United States. We conducted semistructured interviews with 27 Veterans Health Administration nurse leaders across five management levels, using a constant comparative approach for analysis. Participants primarily reported using data for quality improvement, organizational learning, and organizational monitoring and support. Challenges included data fragmentation, unavailability and unsuitability to user need, lack of knowledge about available data, and untimely reporting. Our findings suggest that prioritizing end-user experience and needs is necessary to better govern evidence-based data tools for improving nursing care. Continuous nurse leader involvement in data governance is integral to ensuring high-quality data for end-user nurses to guide their decisions impacting patient care.

Impact of the COVID-19 Pandemic on Medical Education during Inpatient Internal Medicine Rounds

OBJECTIVES

Inpatient rounding is a foundational component of medical education in academic hospitals. The coronavirus 2019 (COVID-19) pandemic disrupted traditional inpatient rounding practices. The objectives of this study were to describe how Internal Medicine inpatient team rounding changed because of COVID-19-related precautions and the effect of these changes on education during rounds.

METHODS

During February to March 2021, we conducted four virtual focus groups with medical and physician assistant students, interns, upper-level residents, and attending physicians at the Michael E. DeBakey Veterans Affairs Medical Center in Houston, Texas, and designed a codebook to categorize focus group commentary.

RESULTS

Focus groups revealed that students believed that certain physical-distancing measures in place early on during the pandemic were ineffective and significantly limited their ability to evaluate patients in person. Residents described increased stress levels related to potential severe acute respiratory-coronavirus 2 exposure and limited time at the bedside, which affected their confidence with clinical assessments. Rounding-team fragmentation precluded the entire team learning from all of the patients on the team's census. Loss of intrateam camaraderie impaired the development of comfortable learning environments.

CONCLUSIONS

This study evaluated Internal Medicine team member focus groups to describe how the COVID-19 pandemic affected medical education during rounds. Academic teaching programs can adapt the findings from this study to address and prevent pandemic-related gaps in medical education during rounds now and during future potential disruptions to medical education.

Developing and validating a comprehensive measure of coordination in patient aligned care teams

Background/Objective

Despite numerous extant measures assessing context-specific elements of care coordination, we are unaware of any comprehensive, team-based instrument that measures the requisite mechanisms and conditions required to coordinate successfully. In this study we develop and validate the psychometric properties of the Coordination Practices Survey, a context-agnostic measure of coordination for primary care teams.

Methods

Coordination items were developed based on a systematic literature review; items from previously developed scales were adapted and new items were created as needed; all items were refined after subject matter expert review and feedback. We collected data from Primary Care teams drawn from 1200 Veterans Health Administration (VHA) medical centers and outpatient clinics nationwide. 1645 primary care team members from 512 patient aligned care teams in the Veterans Health Administration completed the survey from 2015 to 2016. Psychometric properties were assessed after data collection using Cronbach’s alpha, intraclass correlations and multilevel confirmatory factor analysis to assess the factor structure.

Results

Our findings confirmed the psychometric properties of two distinguishable subscales of coordination: (a) Accountability and (b) Common Understanding. The within- and between-team latent structure of each subscale exhibited adequate fit to the data, as well as appropriately high Cronbach’s alpha and intraclass correlations. There was insufficient variability in responses to the predictability subscale to properly assess its psychometric properties.

Conclusion

With context-specific validation, our subscales of accountability and common understanding may be used to assess coordination processes in other contexts for both research and operational applications.

Supplementary information

The online version contains supplementary material available at 10.1186/s12913-022-08590-2.

Increasing physician participation as subjects in scientific and quality improvement research

Background

The twenty-first century has witnessed an exponential increase in healthcare quality research. As such activities become more prevalent, physicians are increasingly needed to participate as subjects in research and quality improvement (QI) projects. This raises an important ethical question: how should physicians be remunerated for participating as research and/or QI subjects?

Financial versus non-monetary incentives for participation

Research suggests participation in research and QI is often driven by conditional altruism, the idea that although initial interest in enrolling in research is altruistic or prosocial, decisions to actually perform study tasks are cost–benefit driven. Thus, the three models commonly employed to appropriately compensate participants (in-kind compensation such as travel reimbursement, paying market rates for the subject’s time, and paying market rates for the activity asked of the participant) are a poor fit when the participant is a clinician, largely due to the asymmetry between cost and benefit or value to the participant. Non-monetary alternatives such as protected time for participation, continuing education or maintenance of certification credit, or professional development materials, can provide viable avenues for reducing this asymmetry.

Conclusion

Research and QI are integral to the betterment of medicine and healthcare. To increase physician participation in these activities as the subject of study, new models are needed that clarify the physician’s role in research and QI as a subject. Non-monetary approaches are recommended to successfully and ethically encourage research and QI participation, and thus incorporate these activities as a normal part of the ethical clinician’s and successful learning healthcare system’s world view.

Analysis of an Antibiotic Stewardship Program for Asymptomatic Bacteriuria in the Veterans Affairs Health Care System

IMPORTANCE

Antibiotic stewardship for asymptomatic bacteriuria (ASB) is an important quality improvement target. Understanding how to implement successful antibiotic stewardship interventions is limited.

OBJECTIVE

To evaluate the effectiveness of a quality improvement stewardship intervention on reducing unnecessary urine cultures and antibiotic use in patients with ASB.

DESIGN, SETTING, AND PARTICIPANTS

This interrupted time series quality improvement study was performed at the acute inpatient medical and long-term care units of 4 intervention sites and 4 comparison sites in the Veterans Affairs (VA) health care system from October 1, 2017, through April 30, 2020. Participants included the clinicians who order or collect urine cultures and who order, dispense, or administer antibiotics. Clinical outcomes were measured in all patients in a study unit during the study period. Data were analyzed from July 6, 2020, to May 24, 2021.

INTERVENTION

Case-based teaching on how to apply an evidence-based algorithm to distinguish urinary tract infection and ASB. The intervention was implemented through external facilitation by a centralized coordinating center, with a site champion at each intervention site serving as an internal facilitator.

MAIN OUTCOMES AND MEASURES

Urine culture orders and days of antibiotic therapy (DOT) and length of antibiotic therapy in days (LOT) associated with urine cultures, standardized by 1000 bed-days, were obtained from the VA's Corporate Data Warehouse.

RESULTS

Of 11 299 patients included, 10 703 (94.7%) were men, with a mean (SD) age of 72.6 (11.8) years. The decrease in urine cultures before and after the intervention was not significant in intervention sites per segmented regression analysis (-0.04 [95% CI, -0.17 to 0.09]; P = .56). However, difference-in-differences analysis comparing intervention with comparison sites found a significant reduction in the number of urine cultures ordered by 3.24 urine cultures per 1000 bed-days (P = .003). In the segmented regression analyses, the relative percentage decrease of DOT in the postintervention period at the intervention sites was 21.7% (P = .007), from 46.1 (95% CI, 28.8-63.4) to 37.0 (95% CI, 22.6-51.4) per 1000 bed-days. The relative percentage decrease of LOT in the postintervention period at the intervention sites was 21.0% (P = .001), from 36.7 (95% CI, 23.2-50.2) to 29.6 (95% CI, 18.2-41.0) per 1000 bed-days.

CONCLUSIONS AND RELEVANCE

The findings of this quality improvement study suggest that an individualized intervention for antibiotic stewardship for ASB was associated with a decrease in urine cultures and antibiotic use when implemented at multiple sites via external and internal facilitation. The electronic health record database-derived outcome measures and centralized facilitation approach are both suitable for dissemination.

Purpose, Subject, and Consumer Comment on "Perceived Burden Due to Registrations for Quality Monitoring and Improvement in Hospitals: A Mixed Methods Study"

Zegers and colleagues’ study codifies the perceived burden of quality monitoring and improvement stemming from the work by clinicians of registering (documenting) quality information in the medical record. We agree with Zegers and colleagues’ recommendation that a smaller, more effective and curated set of measures is needed to reduce burden, confusion, and expense. We further note that focusing on validity of clinical evidence behind individual measures is critical, but insufficient. We therefore extend Zegers and colleagues’ work through a pragmatic, tripartite heuristic. To assess the value of and curate a targeted set of performance measures, we propose concentrating on the relationships among three factors: (1) The purpose of the performance measure, (2) the subject being evaluated, and (3) the consumer using information for decision-making. Our proposed tripartite framework lays the groundwork for executing the evidence-based recommendations proposed by Zegers et al, and provides a path forward for more effective healthcare performance-measurement systems.

Acceptability and feasibility of video-based coaching to enhance clinicians' communication skills with patients

BACKGROUND

Despite a growing call to train clinicians in interpersonal communication skills, communication training is either not offered or is minimally effective, if at all. A critical need exists to develop new ways of teaching communication skills that are effective and mindful of clinician time pressures. We propose a program that includes real-time observation and video-based coaching to teach clinician communication skills. In this study, we assess acceptability and feasibility of the program using clinician interviews and surveys.

METHODS

The video-based coaching intervention targets five patient-centered communication behaviors. It uses trained communication coaches and live feed technology to provide coaching that is brief (less than 15 min), timely (same day) and theory-informed. Two coaches were trained to set up webcams and observe live video feeds of clinician visits in rooms nearby. As coaches watched and recorded the visit, they time stamped illustrative clips in real time. Video clips were a critical element of the program. During feedback sessions, coaches used video clips to promote discussion and self-reflection. They also used role play and guided practice techniques to enforce new tips. Clinicians included residents (n = 15), fellows (n = 4), attending physicians (n = 3), and a nurse practitioner (n = 1) at two primary care clinics in Houston, Texas. We administered surveys to clinicians participating in the program. The survey included questions on quality and delivery of feedback, and credibility of the coaches. We also interviewed clinicians following the intervention. We used rapid analysis to identify themes within the interviews.

RESULTS

Survey measures showed high feasibility and acceptability ratings from clinicians, with mean item scores ranging from 6.4 to 6.8 out of 7 points. Qualitative analysis revealed that clinicians found that 1) coaches were credible and supportive, 2) feedback was useful, 3) video-clips allowed for self-reflection, 4) getting feedback on the same day was useful, and 5) use of real patients preferred over standardized patients.

CONCLUSIONS

Video-based coaching can help clinicians learn new communication skills in a way that is clinician-centered, brief and timely. Our study demonstrates that real-time coaching using live feed and video technology is an acceptable and feasible way of teaching communication skills.

Prevalence of Effective Audit-and-Feedback Practices in Primary Care Settings: A Qualitative Examination Within Veterans Health Administration

OBJECTIVE

The purpose of this study is to uncover and catalog the various practices for delivering and disseminating clinical performance in various Veterans Affairs (VA) locations and to evaluate their quality against evidence-based models of effective feedback as reported in the literature.

BACKGROUND

Feedback can enhance clinical performance in subsequent performance episodes. However, evidence is clear that the way in which feedback is delivered determines whether performance is harmed or improved.

METHOD

We purposively sampled 16 geographically dispersed VA hospitals based on high, low, consistently moderate, and moderately average highly variable performance on a set of 17 outpatient clinical performance measures. We excluded four sites due to insufficient interview data. We interviewed four key personnel from each location (n = 48) to uncover effective and ineffective audit and feedback strategies. Interviews were transcribed and analyzed qualitatively using a framework-based content analysis approach to identify emergent themes.

RESULTS

We identified 102 unique strategies used to deliver feedback. Of these strategies, 64 (62.74%) have been found to be ineffective according to the audit-and-feedback research literature. Comparing features common to effective (e.g., individually tailored, computerized feedback reports) versus ineffective (e.g., large staff meetings) strategies, most ineffective strategies delivered feedback in meetings, whereas strategies receiving the highest effectiveness scores delivered feedback via visually understood reports that did not occur in a group setting.

CONCLUSIONS

Findings show that current practices are leveraging largely ineffective feedback strategies. Future research should seek to identify the longitudinal impact of current feedback and audit practices on clinical performance.

APPLICATION

Feedback in primary care has little standardization and does not follow available evidence for effective feedback design. Future research in this area is warranted.

An evidence-based, structured, expert approach to selecting essential indicators of primary care quality

BACKGROUND

The purpose of this article is to illustrate the application of an evidence-based, structured performance measurement methodology to identify, prioritize, and (when appropriate) generate new measures of health care quality, using primary care as a case example. Primary health care is central to the health care system and health of the American public; thus, ensuring high quality is essential. Due to its complexity, ensuring high-quality primary care requires measurement frameworks that can assess the quality of the infrastructure, workforce configurations, and processes available. This paper describes the use of the Productivity Measurement and Enhancement System (ProMES) to compile a targeted set of such measures, prioritized according to their contribution and value to primary care.

METHODS

We adapted ProMES to select and rank existing primary care measures according to value to the primary care clinic. Nine subject matter experts (SMEs) consisting of clinicians, hospital leaders and national policymakers participated in facilitated expert elicitation sessions to identify objectives of performance, corresponding measures, and priority rankings.

RESULTS

The SMEs identified three fundamental objectives: access, patient-health care team partnerships, and technical quality. The SMEs also selected sixteen performance indicators from the 44 pre-vetted, currently existing measures from three different data sources for primary care. One indicator, Team 2-Day Post Discharge Contact Ratio, was selected as an indicator of both team partnerships and technical quality. Indicators were prioritized according to value using the contingency functions developed by the SMEs.

CONCLUSION

Our article provides an actionable guide to applying ProMES, which can be adapted to the needs of various industries, including measure selection and modification from existing data sources, and proposing new measures. Future work should address both logistical considerations (e.g., data capture, common data/programming language) and lingering measurement challenges, such as operationalizating measures to be meaningful and interpretable across health care settings.

964. Impact of the COVID-19 Pandemic on Bedside Medical Education: A Mixed-Methods Study

Background

The COVID-19 pandemic obligated academic medical programs to substantially alter the traditional Internal Medicine (IM) rounding model to decrease risk of inpatient nosocomial viral transmission. Our study aimed to describe how IM rounding practices changed during the COVID-19 pandemic and to understand the impacts of these changes on medical education.

Methods

We conducted a two-phase, mixed-methods study of inpatient IM rounding team practices at a large academic hospital in Houston, TX. In the first phase (January-February 2021), we organized and audio-recorded 4 virtual (Zoom) focus groups. Each included 5-6 rounding team members, divided by: attendings; senior residents; interns; and medical and physician assistant students. In the second phase (March-May 2021), we performed 6 direct observations of IM teams during rounds. Two observers systematically recorded variables such as time spent on non-bedside versus bedside rounds, number of each team member type entering patient rooms for bedside teaching, and types of personal protective equipment (PPE) worn.

Results

Topics discussed during focus groups included comparisons of rounding team size, rounding duration, physical distancing and PPE use, bedside education, communication methods, and patient safety before and after March 2020. Perceptions of changes in each topic were generally consistent across groups (Table 1). Direct observation data showed that team rounding styles remained diverse in the proportion of rounding time spent in an office versus on the wards, and in the number and types of team members entering patient rooms. IM team members uniformly wore respiratory PPE when entering all patient rooms; use of eye protection varied. Teams spent more total time discussing patients with or suspected to have COVID-19 compared to patients without COVID-19 (median 24 min versus 13 min, p< 0.0001).

Conclusion

Our results suggest that the COVID-19 pandemic adversely impacted bedside medical education, even into Spring of 2021. Conclusions from this study can be used to 1) address educational gaps related to COVID-19 pandemic-associated rounding changes and 2) create innovative methods of providing high-quality clinical education that will be minimally impacted by future respiratory virus pandemics.

Disclosures

Prathit A. Kulkarni, M.D., Vessel Health, Inc. (Grant/Research Support)

An Alternative Method of Public Reporting of Comparative Hospital Quality and Performance Data for Transparency Initiatives

BACKGROUND

Hospital performance comparisons for transparency initiatives may be inadequate if peer comparison groups are poorly defined.

OBJECTIVE

The objective of this study was to evaluate a new approach identifying hospital peers for comparison.

DESIGN/SETTING

We used Mahalanobis distance as a new method of developing peer-specific groupings for hospitals to incorporate both external and internal complexity. We compared the overlap in groups with an existing method used by the Veterans' Health Administration's Office for Productivity, Efficiency, and Staffing (OPES).

PARTICIPANTS

One hundred twenty-two acute-care Veterans' Health Administration's Medical Facilities as defined in the OPES fiscal year 2014 report.

MEASURES

Using 15 variables in 9 categories developed from expert input, including both hospital internal measures and community-based external measures, we used principal components analysis and calculated Mahalanobis distance between each hospital pair. This method accounts for correlation between variables and allows for variables having different variances. We identified the 50 closest hospitals, then eliminated any potential peer whose score on the first component was >1 SD from the reference hospital. We compared overlap with OPES measures.

RESULTS

Of 15 variables, 12 have SDs exceeding 25% of their means. The first 2 components of our analysis explain 24.8% and 18.5% of variation among hospitals. Eight of 9 variables scaling positively on the first component measure internal complexity, aligning with OPES groups. Four of 5 variables scaling positively on the second component but not the first are factors from the policy environment; this component reflects a dimension not considered in OPES groups.

CONCLUSION

Individualized peers that incorporate external complexity generate more nuanced comparators to evaluate quality.

Value of Interprofessional Education: The VA Quality Scholars Program

ABSTRACT

Interprofessional collaboration (IPC) has been shown to improve healthcare quality and patient safety; however, formal interprofessional education (IPE) training is insufficient. The VA Quality Scholars (VAQS) program exists to develop interprofessional leaders and scholars in healthcare improvement. The purpose of this study was to examine the impact of integrating interprofessional healthcare learners and designing an interprofessional curriculum for the national VAQS program. VAQS alumni (graduates from 2001 to 2017) across eight national sites (n = 102 [53.1%]) completed a web-based survey to assess alumni perceptions of IPC skill development during the program and IPC skill utilization in their careers. Alumni from 2009 and earlier were physicians; alumni after 2009 came from diverse health professional backgrounds. Overall, IPC and teamwork was identified as the most used skill (n = 82, 70%) during their career. When comparing the pre-IPE period and the post-IPE period, post-IPE alumni identified IPC and teamwork as the area of greatest skill development (n = 38). Integrating interprofessional trainees and robust IPE curricula enhanced an established and successful quality improvement (QI) training program. VAQS alumni endorsed the importance of IPC skills during their careers. The VAQS program is an example of how health professionals can successfully learn IPC skills in healthcare QI.

Development of a Hybrid, Interprofessional, Interactive Quality Improvement Curriculum as a Model for Continuing Professional Development

Over the past 20 years, there has been an increased focus on quality improvement (QI) in health care, which is critical in achieving care that is patient-centered, safer, timelier, and more effective, efficient, and equitable. At the center of this movement is QI education, which is known to lead to learning, behavior change, and improved outcomes. However, there is a need for the development and provision of long-duration, interactive, interprofessional training in QI, to allow for in-depth learning and application of learned skills. To this end, we designed a curriculum for an established interprofessional, interactive, web-based QI fellowship for doctorally prepared clinicians. Curricular content is delivered virtually to geographically dispersed learners over a 2-year time span. The didactic curriculum and experiential learning opportunities provide learners with the foundational knowledge and practical skills to engage in-and eventually, lead-QI initiatives around the country. Evaluation of learner satisfaction and cognitive, affective, and skills-based learning has found that this model is an effective method to train geographically distributed learners. A hybrid training structure is used, where learners interact with the material through 3 distinct delivery modes: (1) virtual instruction in QI topics; (2) face-to-face training, mentorship, and the opportunity for practical application of applied knowledge and skills through the completion of QI projects; and (3) opportunities for other types of training, tailored to each learner's Individual Development Plan. This training program model holds value for QI learning in various health care settings, which are interprofessional by nature. These foundational concepts of hybrid learning to distributed learners-wherein an instructor delivers curriculum in small, face-to-face batches, interprofessional learning is supplemented in a virtual, longitudinal manner, and learners are allowed the opportunity to put skills into action for real-world problems in interdisciplinary clinical teams-can be applied in a multitude of settings, with comparatively lower time and cost expenditure than traditional training programs.

Examining changes in interprofessional attitudes associated with virtual interprofessional training

Interprofessional care is essential in healthcare, but prior work has shown that physicians and nurses tend to have different perceptions about working interprofessionally (interprofessional attitudes). Although training has been shown to improve interprofessional attitudes, providing traditional face to face training is logistically challenging in the healthcare setting. The current study examined whether a virtual interprofessional training program could improve interprofessional attitudes for nurses and physicians. Among a sample of 35 physicians and nurses, results suggested that engagement in a virtual interprofessional training program was associated with improvements in interprofessional attitudes (i.e., perceived ability to work with, value in working with, and comfort in working with other professions) (p = .002), with attitudes improving an average of 0.25 points on a six-point scale (Cohen's d = 0.52). As a secondary aim, results showed that the magnitude of change in interprofessional attitudes did not differ significantly between physicians and nurses. Altogether, results suggest that virtual interprofessional training appears to be a suitable way to begin to improve interprofessional attitudes for both physicians and nurses.

Reinvigorating stagnant science: implementation laboratories and a meta-laboratory to efficiently advance the science of audit and feedback

Audit and feedback (A&F) is a commonly used quality improvement (QI) approach. A Cochrane review indicates that A&F is generally effective and leads to modest improvements in professional practice but with considerable variation in the observed effects. While we have some understanding of factors that enhance the effects of A&F, further research needs to explore when A&F is most likely to be effective and how to optimise it. To do this, we need to move away from two-arm trials of A&F compared with control in favour of head-to-head trials of different ways of providing A&F. This paper describes implementation laboratories involving collaborations between healthcare organisations providing A&F at scale, and researchers, to embed head-to-head trials into routine QI programmes. This can improve effectiveness while producing generalisable knowledge about how to optimise A&F. We also describe an international meta-laboratory that aims to maximise cross-laboratory learning and facilitate coordination of A&F research.

Impact of team configuration and team stability on primary care quality

Background

The science of effective teams is well documented; far less is known, however, about how specific team configurations may impact primary care team effectiveness. Further, teams experiencing frequent personnel changes (perhaps as a consequence of poor implementation) may have difficulty delivering effective, continuous, well-coordinated care. This study aims to examine the extent to which primary care clinics in the Veterans Health Administration have implemented team configurations consistent with recommendations based on the Patient-Centered Medical Home model and the extent to which adherence to said recommendations, team stability, and role stability impact healthcare quality. Specifically, we expect to find better clinical outcomes in teams that adhere to recommended team configurations, teams whose membership and configuration are more stable over time, and teams whose clinical manager role is more stable over time.

Methods/design

We will employ a combination of social network analysis and multilevel modeling to conduct a database review of variables extracted from the Veterans Health Administration’s Team Assignments Report (TAR) (one of the largest, most diverse existing national samples of primary care teams (n_teams > 7000)), as well as other employee and clinical data sources. To ensure the examination of appropriate clinical outcomes, we will enlist a team of subject matter experts to select a concise set of clear, prioritized primary care performance metrics. We will accomplish this using the Productivity Measurement and Enhancement System, an evidence-based methodology for developing and implementing performance measurement.

Discussion

We are unaware of other studies of healthcare teams that consider team size, composition, and configuration longitudinally or with sample sizes of this magnitude. Results from this study can inform primary care team implementation policy and practice in both private- and public-sector clinics, such that teams are configured optimally, with adequate staffing, and the right mix of roles within the team.

Trial registration

Not applicable—this study does not involve interventions on human participants.

Electronic supplementary material

The online version of this article (10.1186/s13012-019-0864-8) contains supplementary material, which is available to authorized users.

Mental models of audit and feedback in primary care settings

BACKGROUND

Audit and feedback has been shown to be instrumental in improving quality of care, particularly in outpatient settings. The mental model individuals and organizations hold regarding audit and feedback can moderate its effectiveness, yet this has received limited study in the quality improvement literature. In this study we sought to uncover patterns in mental models of current feedback practices within high- and low-performing healthcare facilities.

METHODS

We purposively sampled 16 geographically dispersed VA hospitals based on high and low performance on a set of chronic and preventive care measures. We interviewed up to 4 personnel from each location (n = 48) to determine the facility's receptivity to audit and feedback practices. Interview transcripts were analyzed via content and framework analysis to identify emergent themes.

RESULTS

We found high variability in the mental models of audit and feedback, which we organized into positive and negative themes. We were unable to associate mental models of audit and feedback with clinical performance due to high variance in facility performance over time. Positive mental models exhibit perceived utility of audit and feedback practices in improving performance; whereas, negative mental models did not.

CONCLUSIONS

Results speak to the variability of mental models of feedback, highlighting how facilities perceive current audit and feedback practices. Findings are consistent with prior research  in that variability in feedback mental models is associated with lower performance.; Future research should seek to empirically link mental models revealed in this paper to high and low levels of clinical performance.

Evaluation of a Distance Learning Curriculum for Interprofessional Quality Improvement Leaders

As health care systems move toward value-based care, training future leaders in quality improvement (QI) is essential. Web-based training allows for broad dissemination of QI knowledge to geographically distributed learners. The authors conducted a longitudinal evaluation of a structured, synchronous web-based, advanced QI curriculum that facilitated engagement and real-time feedback. Learners (n = 54) were satisfied (overall satisfaction; M = 3.31/4.00), and there were improvements in cognitive (immediate QI knowledge tests; P = .02), affective (self-efficacy of QI skills; P < .001), and skill-based learning (Quality Improvement Knowledge Application Tool; P < .001). There was significant improvement in affective transfer (interprofessional attitudes on the job; p < .01) but no significant change on cognitive (distal QI knowledge test; P = .91), or skill-based transfer (self-reported interprofessional collaboration job skills; P = .23). The findings suggest that this model can be effective to train geographically distributed future QI leaders.

Protocol to disseminate a hospital-site controlled intervention using audit and feedback to implement guidelines concerning inappropriate treatment of asymptomatic bacteriuria

BACKGROUND

Antimicrobial stewardship to combat the spread of antibiotic-resistant bacteria has become a national priority. This project focuses on reducing inappropriate use of antimicrobials for asymptomatic bacteriuria (ASB), a very common condition that leads to antimicrobial overuse in acute and long-term care. We previously conducted a successful intervention, entitled "Kicking Catheter Associated Urinary Tract Infection (CAUTI): the No Knee-Jerk Antibiotics Campaign," to decrease guideline-discordant ordering of urine cultures and antibiotics for ASB. The current objective is to facilitate implementation of a scalable version of the Kicking CAUTI campaign across four geographically diverse Veterans Health Administration facilities while assessing what aspects of an antimicrobial stewardship intervention are essential to success and sustainability.

METHODS

This project uses an interrupted time series design with four control sites. The two main intervention tools are (1) an evidence-based algorithm that distills the guidelines into a streamlined clinical pathway and (2) case-based audit and feedback to train clinicians to use the algorithm. Our conceptual framework for the development and implementation of this intervention draws on May's General Theory of Implementation. The intervention is directed at providers in acute and long-term care, and the goal is to reduce inappropriate screening for and treatment of ASB in all patients and residents, not just those with urinary catheters. The start-up for each facility consists of centrally-led phone calls with local site champions and baseline surveys. Case-based audit and feedback will begin at a given site after the start-up period and continue for 12 months, followed by a sustainability assessment. In addition to the clinical outcomes, we will explore the relationship between the dose of the intervention and clinical outcomes.

DISCUSSION

This project moves from a proof-of-concept effectiveness study to implementation involving significantly more sites, and uses the General Theory of Implementation to embed the intervention into normal processes of care with usual care providers. Aspects of implementation that will be explored include dissemination, internal and external facilitation, and organizational partnerships. "Less is More" is the natural next step from our prior successful Kicking CAUTI intervention, and has the potential to improve patient care while advancing the science of implementation.

Model depicting aspects of audit and feedback that impact physicians' acceptance of clinical performance feedback

Background

Audit and feedback (A&F) is a strategy that has been used in various disciplines for performance and quality improvement. There is limited research regarding medical professionals’ acceptance of clinical-performance feedback and whether feedback impacts clinical practice. The objectives of our research were to (1) investigate aspects of A&F that impact physicians’ acceptance of performance feedback; (2) determine actions physicians take when receiving feedback; and (3) determine if feedback impacts physicians’ patient-management behavior.

Methods

In this qualitative study, we employed grounded theory methods to perform a secondary analysis of semi-structured interviews with 12 VA primary care physicians. We analyzed a subset of interview questions from the primary study, which aimed to determine how providers of high, low and moderately performing VA medical centers use performance feedback to maintain and improve quality of care, and determine perceived utility of performance feedback.

Results

Based on the themes emergent from our analysis and their observed relationships, we developed a model depicting aspects of the A&F process that impact feedback acceptance and physicians’ patient-management behavior. The model is comprised of three core components – Reaction, Action and Impact – and depicts elements associated with feedback recipients’ reaction to feedback, action taken when feedback is received, and physicians modifying their patient-management behavior. Feedback characteristics, the environment, external locus-of-control components, core values, emotion and the assessment process induce or deter reaction, action and impact.

Feedback characteristics (content and timeliness), and the procedural justice of the assessment process (unjust penalties) impact feedback acceptance. External locus-of-control elements (financial incentives, competition), the environment (patient volume, time constraints) and emotion impact patient-management behavior. Receiving feedback generated intense emotion within physicians. The underlying source of the emotion was the assessment process, not the feedback. The emotional response impacted acceptance, impelled action or inaction, and impacted patient-management behavior. Emotion intensity was associated with type of action taken (defensive, proactive, retroactive).

Conclusions

Feedback acceptance and impact have as much to do with the performance assessment process as it does the feedback. In order to enhance feedback acceptance and the impact of feedback, developers of clinical performance systems and feedback interventions should consider multiple design elements.

Theory-based and evidence-based design of audit and feedback programmes: examples from two clinical intervention studies

BACKGROUND

Audit and feedback (A&F) is a common intervention used to change healthcare provider behaviour and, thus, improve healthcare quality. Although A&F can be effective its effectiveness varies, often due to the details of how A&F interventions are implemented. Some have suggested that a suitable conceptual framework is needed to organise the elements of A&F and also explain any observed differences in effectiveness. Through two examples from applied research studies, this article demonstrates how a suitable explanatory theory (in this case Kluger & DeNisi's Feedback Intervention Theory (FIT)) can be systematically applied to design better feedback interventions in healthcare settings.

METHODS

Case 1: this study's objective was to reduce inappropriate diagnosis of catheter-associated urinary tract infections (CAUTI) in inpatient wards. Learning to identify the correct clinical course of action from the case details was central to this study; consequently, the feedback intervention featured feedback elements that FIT predicts would best activate learning processes (framing feedback in terms of group performance and providing of correct solution information). We designed a highly personalised, interactive, one-on-one intervention with healthcare providers to improve their capacity to distinguish between CAUTI and asymptomatic bacteruria (ASB) and treat ASB appropriately. Case 2: Simplicity and scalability drove this study's intervention design, employing elements that FIT predicted positively impacted effectiveness yet still facilitated deployment and scalability (eg, delivered via computer, delivered in writing). We designed a web-based, report-style feedback intervention to help primary care physicians improve their care of patients with hypertension.

RESULTS

Both studies exhibited significant improvements in their desired outcome and in both cases interventions were received positively by feedback recipients.

SUMMARY

A&F has been a popular, yet inconsistently implemented and variably effective tool for changing healthcare provider behaviour and, improving healthcare quality. Through the systematic use of theory such as FIT, robust feedback interventions can be designed that yield greater effectiveness. Future work should look to comparative effectiveness of specific design elements and contextual factors that identify A&F as the optimal intervention to effectuate healthcare provider behaviour change.

TRIAL REGISTRATION NUMBER

NCT01052545, NCT00302718; post-results.

A Novel Method for Assessing Task Complexity in Outpatient Clinical-Performance Measures

BACKGROUND

Clinical-performance measurement has helped improve the quality of health-care; yet success in attaining high levels of quality across multiple domains simultaneously still varies considerably. Although many sources of variability in care quality have been studied, the difficulty required to complete the clinical work itself has received little attention.

OBJECTIVE

We present a task-based methodology for evaluating the difficulty of clinical-performance measures (CPMs) by assessing the complexity of their component requisite tasks.

DESIGN

Using Functional Job Analysis (FJA), subject-matter experts (SMEs) generated task lists for 17 CPMs; task lists were rated on ten dimensions of complexity, and then aggregated into difficulty composites.

PARTICIPANTS

Eleven outpatient work SMEs; 133 VA Medical Centers nationwide.

MAIN MEASURES

Clinical Performance: 17 outpatient CPMs (2000-2008) at 133 VA Medical Centers nationwide. Measure Difficulty: for each CPM, the number of component requisite tasks and the average rating across ten FJA complexity scales for the set of tasks comprising the measure.

KEY RESULTS

Measures varied considerably in the number of component tasks (M = 10.56, SD = 6.25, min = 5, max = 25). Measures of chronic care following acute myocardial infarction exhibited significantly higher measure difficulty ratings compared to diabetes or screening measures, but not to immunization measures ([Formula: see text] = 0.45, -0.04, -0.05, and -0.06 respectively; F (3, 186) = 3.57, p = 0.015). Measure difficulty ratings were not significantly correlated with the number of component tasks (r = -0.30, p = 0.23).

CONCLUSIONS

Evaluating the difficulty of achieving recommended CPM performance levels requires more than simply counting the tasks involved; using FJA to assess the complexity of CPMs' component tasks presents an alternate means of assessing the difficulty of primary-care CPMs and accounting for performance variation among measures and performers. This in turn could be used in designing performance reward programs, or to match workflow to clinician time and effort.

Linking clinician interaction and coordination to clinical performance in Patient-Aligned Care Teams

Background

Team coordination within clinical care settings is a critical component of effective patient care. Less is known about the extent, effectiveness, and impact of coordination activities among professionals within VA Patient-Aligned Care Teams (PACTs). This study will address these gaps by describing the specific, fundamental tasks and practices involved in PACT coordination, their impact on performance measures, and the role of coordination task complexity.

Methods/design

First, we will use a web-based survey of coordination practices among 1600 PACTs in the national VHA. Survey findings will characterize PACT coordination practices and assess their association with clinical performance measures. Functional job analysis, using 6–8 subject matter experts who are 3rd and 4th year residents in VA Primary Care rotations, will be utilized to identify the tasks involved in completing clinical performance measures to standard. From this, expert ratings of coordination complexity will be used to determine the level of coordinative complexity required for each of the clinical performance measures drawn from the VA External Peer Review Program (EPRP). For objective 3, data collected from the first two methods will evaluate the effect of clinical complexity on the relationships between measures of PACT coordination and their ratings on the clinical performance measures.

Discussion

Results from this study will support successful implementation of coordinated team-based work in clinical settings by providing knowledge regarding which aspects of care require the most complex levels of coordination and how specific coordination practices impact clinical performance.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-015-0368-0) contains supplementary material, which is available to authorized users.

Study protocol: identifying and delivering point-of-care information to improve care coordination

BACKGROUND

The need for deliberately coordinated care is noted by many national-level organizations. The Department of Veterans Affairs (VA) recently transitioned primary care clinics nationwide into Patient Aligned Care Teams (PACTs) to provide more accessible, coordinated, comprehensive, and patient-centered care. To better serve this purpose, PACTs must be able to successfully sequence and route interdependent tasks to appropriate team members while also maintaining collective situational awareness (coordination). Although conceptual frameworks of care coordination exist, few explicitly articulate core behavioral markers of coordination or the related information needs of team members attempting to synchronize complex care processes across time for a shared patient population. Given this gap, we partnered with a group of frontline primary care personnel at ambulatory care sites to identify the specific information needs of PACT members that will enable them to coordinate their efforts to provide effective, coordinated care. The study has three objectives: (1) development of measurable, prioritized point-of-care criteria for effective PACT coordination; (2) identifying the specific information needed at the point of care to optimize coordination; and (3) assessing the effect of adopting the aforementioned coordination standards on PACT clinicians' coordination behaviors.

METHODS/DESIGN

The study consists of three phases. In phase 1, we will employ the Productivity Measurement and Enhancement System (ProMES), a structured approach to performance measure creation from industrial/organizational psychology, to develop coordination measures with a design team of 6-10 primary care personnel; in phase 2, we will conduct focus groups with the phase 1 design team to identify point-of-care information needs. Phase 3 is a two-arm field experiment (n PACT = 28/arm); intervention arm PACTs will receive monthly feedback reports using the measures developed in phase 1 and attend brief monthly feedback sessions. Control arm PACTs will receive no intervention. PACTs will be followed prospectively for up to 1 year.

DISCUSSION

This project combines both action research and implementation science methods to address important gaps in the existing care coordination literature using a partnership-based research design. It will provide an evidence-based framework for care coordination by employing a structured methodology for a systematic approach to care coordination in PACT settings and identifying the information needs that produce the most successful coordination of care.

TRIAL REGISTRATION

ISRCTN15412521.

Electronic alerts and clinician turnover: the influence of user acceptance

OBJECTIVES

Use of certain components of electronic health records (EHRs), such as EHR-based alerting systems (EASs), might reduce provider satisfaction, a strong precursor to turnover. We examined the impact of factors likely to influence providers' acceptance of an alerting system, designed to facilitate electronic communication in outpatient settings, on provider satisfaction, intentions to quit, and turnover.

STUDY DESIGN AND METHODS

We conducted a cross-sectional Web-based survey of EAS-related practices from a nationwide sample of primary care providers (PCPs) practicing at Department of Veterans Affairs (VA) medical facilities. Of 5001 invited VA PCPs, 2590 completed the survey. We relied on Venkatesh's Unified Theory of Acceptance and Use of Technology to create survey measures of 4 factors likely to impact user acceptance of EAS: supportive norms, monitoring/ feedback, training, and providers' perceptions of the value (PPOV) of EASs to provider effectiveness. Facility-level PCP turnover was measured via the VA's Service Support Center Human Resources Cube. Hypotheses were tested using structural equation modeling.

RESULTS

After accounting for intercorrelations among predictors, monitoring/feedback regarding EASs significantly predicted intention to quit (b = 0.30, P < .01), and PPOV of EASs predicted both overall provider satisfaction (b = 0.58, P < .01) and facility-level provider turnover levels (b = -0.19, P < .05), all without relying on any intervening mechanisms.

CONCLUSIONS

Design, implementation, and use of EASs might impact provider satisfaction and retention. Institutions should consider strategies to help providers perceive greater value in these clinical tools.

How context affects electronic health record-based test result follow-up: a mixed-methods evaluation

OBJECTIVES

Electronic health record (EHR)-based alerts can facilitate transmission of test results to healthcare providers, helping ensure timely and appropriate follow-up. However, failure to follow-up on abnormal test results (missed test results) persists in EHR-enabled healthcare settings. We aimed to identify contextual factors associated with facility-level variation in missed test results within the Veterans Affairs (VA) health system.

DESIGN, SETTING AND PARTICIPANTS

Based on a previous survey, we categorised VA facilities according to primary care providers' (PCPs') perceptions of low (n=20) versus high (n=20) risk of missed test results. We interviewed facility representatives to collect data on several contextual factors derived from a sociotechnical conceptual model of safe and effective EHR use. We compared these factors between facilities categorised as low and high perceived risk, adjusting for structural characteristics.

RESULTS

Facilities with low perceived risk were significantly more likely to use specific strategies to prevent alerts from being lost to follow-up (p=0.0114). Qualitative analysis identified three high-risk scenarios for missed test results: alerts on tests ordered by trainees, alerts 'handed off' to another covering clinician (surrogate clinician), and alerts on patients not assigned in the EHR to a PCP. Test result management policies and procedures to address these high-risk situations varied considerably across facilities.

CONCLUSIONS

Our study identified several scenarios that pose a higher risk for missed test results in EHR-based healthcare systems. In addition to implementing provider-level strategies to prevent missed test results, healthcare organisations should consider implementing monitoring systems to track missed test results.

Baby-MONITOR: a composite indicator of NICU quality

BACKGROUND AND OBJECTIVES

NICUs vary in the quality of care delivered to very low birth weight (VLBW) infants. NICU performance on 1 measure of quality only modestly predicts performance on others. Composite measurement of quality of care delivery may provide a more comprehensive assessment of quality. The objective of our study was to develop a robust composite indicator of quality of NICU care provided to VLBW infants that accurately discriminates performance among NICUs.

METHODS

We developed a composite indicator, Baby-MONITOR, based on 9 measures of quality chosen by a panel of experts. Measures were standardized, equally weighted, and averaged. We used the California Perinatal Quality Care Collaborative database to perform across-sectional analysis of care given to VLBW infants between 2004 and 2010. Performance on the Baby-MONITOR is not an absolute marker of quality but indicates overall performance relative to that of the other NICUs. We used sensitivity analyses to assess the robustness of the composite indicator, by varying assumptions and methods.

RESULTS

Our sample included 9023 VLBW infants in 22 California regional NICUs. We found significant variations within and between NICUs on measured components of the Baby-MONITOR. Risk-adjusted composite scores discriminated performance among this sample of NICUs. Sensitivity analysis that included different approaches to normalization, weighting, and aggregation of individual measures showed the Baby-MONITOR to be robust (r = 0.89-0.99).

CONCLUSIONS

The Baby-MONITOR may be a useful tool to comprehensively assess the quality of care delivered by NICUs.

Examining clinical performance feedback in Patient-Aligned Care Teams

BACKGROUND

The move to team-based models of health care represents a fundamental shift in healthcare delivery, including major changes in the roles and relationships among clinical personnel. Audit and feedback of clinical performance has traditionally focused on the provider; however, a team-based model of care may require different approaches.

OBJECTIVE

Identify changes in audit and feedback of clinical performance to primary care clinical personnel resulting from implementing team-based care in their clinics.

DESIGN

Semi-structured interviews with primary care clinicians, their department heads, and facility leadership at 16 geographically diverse VA Medical Centers, selected purposively by their clinical performance profile.

PARTICIPANTS

An average of three interviewees per VA medical center, selected from physicians, nurses, and primary care and facility directors who participated in 1-hour interviews.

APPROACH

Interviews focused on how clinical performance information is fed back to clinicians, with particular emphasis on external peer-review program measures and changes in feedback associated with team-based care implementation. Interview transcripts were analyzed, using techniques adapted from grounded theory and content analysis.

KEY RESULTS

Ownership of clinical performance still rests largely with the provider, despite transitioning to team-based care. A panel-management information tool emerged as the most prominent change to clinical performance feedback dissemination, and existing feedback tools were seen as most effective when monitored by the nurse members of the team. Facilities reported few, if any, appreciable changes to the assessment of clinical performance since transitioning to team-based care.

CONCLUSIONS

Although new tools have been created to support higher-quality clinical performance feedback to primary care teams, such tools have not necessarily delivered feedback consistent with a team-based approach to health care. Audit and feedback of clinical performance has remained largely unchanged, despite material differences in roles and responsibilities of team members. Future research should seek to unpack the nuances of team-based audit and feedback, to better align feedback with strategic clinical goals.

Overtreatment of asymptomatic bacteriuria: identifying provider barriers to evidence-based care

BACKGROUND

Inappropriate use of antibiotics to treat asymptomatic bacteriuria (ASB) is a significant contributor to antibiotic overuse in hospitalized patients despite evidence-based guidelines on ASB management. We surveyed whether accurate knowledge of how to manage catheter-associated urine cultures was associated with level of training, familiarity with ASB guidelines, and various cognitive-behavioral constructs.

METHODS

We used a survey to measure respondents' knowledge of how to manage catheter-associated bacteriuria, familiarity with the content of the relevant Infectious Diseases Society of America guidelines, and cognitive-behavioral constructs. The survey was administered to 169 residents and staff providers.

RESULTS

The mean knowledge score was 57.5%, or slightly over one-half of the questions answered correctly. The overall knowledge score improved significantly with level of training (P < .0001). Only 42% of respondents reported greater than minimal recall of ASB guideline contents. Self-efficacy, behavior, risk perceptions, social norms, and guideline familiarity were individually correlated with knowledge score (P < .01). In multivariable analysis, behavior, risk perception, and year of training were correlated with knowledge score (P < .05).

CONCLUSIONS

Knowledge of how to manage catheter-associated bacteriuria according to evidence-based guidelines increases with experience. Addressing both knowledge gaps and relevant cognitive biases early in training may decrease the inappropriate use of antibiotics to treat ASB.

More than a score: a qualitative study of ancillary benefits of performance measurement

BACKGROUND

Prior research has examined clinical effects of performance measurement systems. To the extent that non-clinical effects have been researched, the focus has been on negative unintended consequences. Yet, these same systems may also have ancillary benefits for patients and providers--that is, benefits that extend beyond improvements on clinical measures. The purpose of this study is to identify and describe potential ancillary benefits of performance measures as perceived by primary care staff and facility leaders in a large US healthcare system.

METHODS

In-person individual semistructured interviews were conducted with 59 primary care staff and facility leaders at four Veterans Health Administration facilities. Transcribed interviews were coded and organised into thematic categories.

RESULTS

Interviewed staff observed that local performance measurement implementation practices can result in increased patient knowledge and motivation. These effects on patients can lead to improved performance scores and additional ancillary benefits. Performance measurement implementation can also directly result in ancillary benefits for the patients and providers. Patients may experience greater satisfaction with care and psychosocial benefits associated with increased provider-patient communication. Ancillary benefits of performance measurement for providers include increased pride in individual or organisational performance and greater confidence that one's practice is grounded in evidence-based medicine.

CONCLUSIONS

A comprehensive understanding of the effects of performance measurement systems needs to incorporate ancillary benefits as well as effects on clinical performance scores and negative unintended consequences. Although clinical performance has been the focus of most evaluations of performance measurement to date, both patient care and provider satisfaction may improve more rapidly if all three categories of effects are considered when designing and evaluating performance measurement systems.

Recruiting clinical personnel as research participants: a framework for assessing feasibility

Increasing numbers of research studies test interventions for clinicians in addition to or instead of interventions for patients. Although previous studies have enumerated barriers to patient enrolment in clinical trials, corresponding barriers have not been identified for enrolling clinicians as subjects. We propose a framework of metrics for evidence-based estimation of time and resources required for recruiting clinicians as research participants, and present an example from a federally funded study. Our framework proposes metrics for tracking five steps in the recruitment process: gaining entry into facilities, obtaining accurate eligibility and contact information, reaching busy clinicians, assessing willingness to participate, and scheduling participants for data collection. We analyzed recruitment records from a qualitative study exploring performance feedback at US Department of Veterans Affairs Medical Centers (VAMCs); five recruiters sought to reach two clinicians at 16 facilities for a one-hour interview. Objective metrics were calculable for all five steps; metric values varied considerably across facilities. Obtaining accurate contact information slowed down recruiting the most. We conclude that successfully recruiting even small numbers of employees requires considerable resourcefulness and more calendar time than anticipated. Our proposed framework provides an empirical basis for estimating research-recruitment timelines, planning subject-recruitment strategies, and assessing the research accessibility of clinical sites.

Effects of individual physician-level and practice-level financial incentives on hypertension care: a randomized trial

IMPORTANCE

Pay for performance is intended to align incentives to promote high-quality care, but results have been contradictory.

OBJECTIVE

To test the effect of explicit financial incentives to reward guideline-recommended hypertension care.

DESIGN, SETTING, AND PARTICIPANTS

Cluster randomized trial of 12 Veterans Affairs outpatient clinics with 5 performance periods and a 12-month washout that enrolled 83 primary care physicians and 42 nonphysician personnel (eg, nurses, pharmacists).

INTERVENTIONS

Physician-level (individual) incentives, practice-level incentives, both, or none. Intervention participants received up to 5 payments every 4 months; all participants could access feedback reports.

MAIN OUTCOMES AND MEASURES

Among a random sample, number of patients achieving guideline-recommended blood pressure thresholds or receiving an appropriate response to uncontrolled blood pressure, number of patients prescribed guideline-recommended medications, and number who developed hypotension.

RESULTS

Mean (SD) total payments over the study were $4270 ($459), $2672 ($153), and $1648 ($248) for the combined, individual, and practice-level interventions, respectively. The unadjusted baseline and final percentages and the adjusted absolute change over the study in patients meeting the combined blood pressure/appropriate response measure were 75% to 84% and 8.84% (95% CI, 4.20% to 11.80%) for the individual group, 80% to 85% and 3.70% (95% CI, 0.24% to 7.68%) for the practice-level group, 79% to 88% and 5.54% (95% CI, 1.92% to 9.52%) for the combined group, and 86% to 86% and 0.47% (95% CI, -3.12% to 4.04%) for the control group. The adjusted absolute estimated difference in the change between the proportion of patients with blood pressure control/appropriate response for individual incentive and control groups was 8.36% (95% CI, 2.40% to 13.00%; P=.005). The other incentive groups did not show a significant change compared with controls for this outcome. For medications, the unadjusted baseline and final percentages and the adjusted absolute change were 61% to 73% and 9.07% (95% CI, 4.52% to 13.44%), 56% to 65% and 4.98% (95% CI, 0.64% to 10.08%), 65% to 80% and 7.26% (95% CI, 2.92% to 12.48%), and 63% to 72% and 4.35% (95% CI, -0.28% to 9.28%), respectively. These changes in the use of guideline-recommended medications were not significant in any of the incentive groups compared with controls, nor was the incidence of hypotension. The effect of the incentive was not sustained after a washout.

CONCLUSIONS AND RELEVANCE

Individual financial incentives, but not practice-level or combined incentives, resulted in greater blood pressure control or appropriate response to uncontrolled blood pressure; none of the incentives resulted in greater use of guideline-recommended medications or increased incidence of hypotension compared with controls. Further research is needed on the factors that contributed to these findings.

TRIAL REGISTRATION

clinicaltrials.gov Identifier: NCT00302718.

Development and validation of an algorithm to recalibrate mental models and reduce diagnostic errors associated with catheter-associated bacteriuria

Background

Overtreatment of catheter-associated bacteriuria is a quality and safety problem, despite the availability of evidence-based guidelines. Little is known about how guidelines-based knowledge is integrated into clinicians’ mental models for diagnosing catheter-associated urinary tract infection (CA-UTI). The objectives of this research were to better understand clinicians’ mental models for CA-UTI, and to develop and validate an algorithm to improve diagnostic accuracy for CA-UTI.

Methods

We conducted two phases of this research project. In phase one, 10 clinicians assessed and diagnosed four patient cases of catheter associated bacteriuria (n= 40 total cases). We assessed the clinical cues used when diagnosing these cases to determine if the mental models were IDSA guideline compliant. In phase two, we developed a diagnostic algorithm derived from the IDSA guidelines. IDSA guideline authors and non-expert clinicians evaluated the algorithm for content and face validity. In order to determine if diagnostic accuracy improved using the algorithm, we had experts and non-experts diagnose 71 cases of bacteriuria.

Results

Only 21 (53%) diagnoses made by clinicians without the algorithm were guidelines-concordant with fair inter-rater reliability between clinicians (Fleiss’ kappa = 0.35, 95% Confidence Intervals (CIs) = 0.21 and 0.50). Evidence suggests that clinicians’ mental models are inappropriately constructed in that clinicians endorsed guidelines-discordant cues as influential in their decision-making: pyuria, systemic leukocytosis, organism type and number, weakness, and elderly or frail patient. Using the algorithm, inter-rater reliability between the expert and each non-expert was substantial (Cohen’s kappa = 0.72, 95% CIs = 0.52 and 0.93 between the expert and non-expert #1 and 0.80, 95% CIs = 0.61 and 0.99 between the expert and non-expert #2).

Conclusions

Diagnostic errors occur when clinicians’ mental models for catheter-associated bacteriuria include cues that are guidelines-discordant for CA-UTI. The understanding we gained of clinicians’ mental models, especially diagnostic errors, and the algorithm developed to address these errors will inform interventions to improve the accuracy and reliability of CA-UTI diagnoses.

Correlation of neonatal intensive care unit performance across multiple measures of quality of care

OBJECTIVES

To examine whether high performance on one measure of quality is associated with high performance on others and to develop a data-driven explanatory model of neonatal intensive care unit (NICU) performance.

DESIGN

We conducted a cross-sectional data analysis of a statewide perinatal care database. Risk-adjusted NICU ranks were computed for each of 8 measures of quality selected based on expert input. Correlations across measures were tested using the Pearson correlation coefficient. Exploratory factor analysis was used to determine whether underlying factors were driving the correlations.

SETTING

Twenty-two regional NICUs in California.

PATIENTS

In total, 5445 very low-birth-weight infants cared for between January 1, 2004, and December 31, 2007.

MAIN OUTCOMES MEASURES

Pneumothorax, growth velocity, health care-associated infection, antenatal corticosteroid use, hypothermia during the first hour of life, chronic lung disease, mortality in the NICU, and discharge on any human breast milk.

RESULTS

The NICUs varied substantially in their clinical performance across measures of quality. Of 28 unit-level correlations, 6 were significant (ρ < .05). Correlations between pairs of measures of quality of care were strong (ρ ≥ .5) for 1 pair, moderate (range, ρ ≥ .3 to ρ < .5) for 8 pairs, weak (range, ρ ≥ .1 to ρ < .3) for 5 pairs, and negligible (ρ < .1) for 14 pairs. Exploratory factor analysis revealed 4 underlying factors of quality in this sample. Pneumothorax, mortality in the NICU, and antenatal corticosteroid use loaded on factor 1; growth velocity and health care-associated infection loaded on factor 2; chronic lung disease loaded on factor 3; and discharge on any human breast milk loaded on factor 4.

CONCLUSION

In this sample, the ability of individual measures of quality to explain overall quality of neonatal intensive care was modest.

Financial incentives and physician commitment to guideline-recommended hypertension management

OBJECTIVES

To examine the impact of financial incentives on physician goal commitment to guideline-recommended hypertension care.

STUDY DESIGN

Clinic-level cluster-randomized trial with 4 arms: individual, group, or combined incentives, and control.

METHODS

A total of 83 full-time primary care physicians at 12 Veterans Affairs medical centers completed web-based surveys measuring their goal commitment to guideline-recommended hypertension care every 4 months and telephone interviews at months 8 and 16. Intervention arm participants received performance-based incentives every 4 months for 5 periods. All participants received guideline education at baseline and audit and feedback every 4 months.

RESULTS

Physician goal commitment did not vary over time or across arms. Participants reported patient nonadherence was a perceived barrier and consistent follow-up was a perceived facilitator to successful hypertension care, suggesting that providers may perceive hypertension management as more of a patient responsibility (external locus of control).

CONCLUSIONS

Financial incentives may constitute an insufficiently strong intervention to influence goal commitment when providers attribute performance to external forces beyond their control.

Improving quality of care through improved audit and feedback

Background

The Department of Veterans Affairs (VA) has led the industry in measuring facility performance as a critical element in improving quality of care, investing substantial resources to develop and maintain valid and cost-effective measures. The External Peer Review Program (EPRP) of the VA is the official data source for monitoring facility performance, used to prioritize the quality areas needing most attention. Facility performance measurement has significantly improved preventive and chronic care, as well as overall quality; however, much variability still exists in levels of performance across measures and facilities. Audit and feedback (A&F), an important component of effective performance measurement, can help reduce this variability and improve overall performance. Previous research suggests that VA Medical Centers (VAMCs) with high EPRP performance scores tend to use EPRP data as a feedback source. However, the manner in which EPRP data are used as a feedback source by individual providers as well as service line, facility, and network leadership is not well understood. An in-depth understanding of mental models, strategies, and specific feedback process characteristics adopted by high-performing facilities is thus urgently needed.

This research compares how leaders of high, low, and moderately performing VAMCs use clinical performance data from the EPRP as a feedback tool to maintain and improve quality of care.

Methods

We will conduct a qualitative, grounded theory analysis of up to 64 interviews using a novel method of sampling primary care, facility, and Veterans Integrated Service Network (VISN) leadership at high-, moderate-, and low-performing facilities. We will analyze interviews for evidence of cross-facility differences in perceptions of performance data usefulness and strategies for disseminating performance data evaluating performance, with particular attention to timeliness, individualization, and punitiveness of feedback delivery.

Discussion

Most research examining feedback to improve provider and facility performance lacks a detailed understanding of the elements of effective feedback. This research will highlight the elements most commonly used at high-performing facilities and identify additional features of their successful feedback strategies not previously identified. Armed with this information, practices can implement more effective A&F interventions to improve quality of care.

Neonatal intensive care unit safety culture varies widely

BACKGROUND

Variation in healthcare delivery and outcomes in neonatal intensive care units (NICUs) may be partly explained by differences in safety culture.

OBJECTIVE

To describe NICU care giver assessments of safety culture, explore variability within and between NICUs on safety culture domains, and test for association with care giver characteristics.

METHODS

NICU care givers in 12 hospitals were surveyed using the Safety Attitudes Questionnaire (SAQ), which has six scales: teamwork climate, safety climate, job satisfaction, stress recognition, perception of management and working conditions. Scale means, SDs and percent positives (percent agreement) were calculated for each NICU.

RESULTS

There was substantial variation in safety culture domains among NICUs. Composite mean score across the six domains ranged from 56.3 to 77.8 on a 100-point scale and NICUs in the top four NICUs were significantly different from the bottom four (p<0.001). Across the six domains, respondent assessments varied widely, but were least positive on perceptions of management (3%-80% positive; mean 33.3%) and stress recognition (18%-61% positive; mean 41.3%). Comparisons of SAQ scale scores between NICUs and a previously published adult ICU cohort generally revealed higher scores for NICUs. Composite scores for physicians were 8.2 (p=0.04) and 9.5 (p=0.02) points higher than for nurses and ancillary personnel.

CONCLUSION

There is significant variation and scope for improvement in safety culture among these NICUs. The NICU variation was similar to variation in adult ICUs, but NICU scores were generally higher. Future studies should validate whether safety culture measured with the SAQ correlates with clinical and operational outcomes in NICUs.

The Safety Attitudes Questionnaire as a tool for benchmarking safety culture in the NICU

BACKGROUND

Neonatal intensive care unit (NICU) safety culture, as measured by the Safety Attitudes Questionnaire (SAQ), varies widely. Associations with clinical outcomes in the adult intensive care unit setting make the SAQ an attractive tool for comparing clinical performance between hospitals. Little information is available on the use of the SAQ for this purpose in the NICU setting.

OBJECTIVES

To determine whether the dimensions of safety culture measured by the SAQ give consistent results when used as a NICU performance measure.

METHODS

Cross-sectional survey of caregivers in 12 NICUs, using the six scales of the SAQ: teamwork climate, safety climate, job satisfaction, stress recognition, perceptions of management and working conditions. NICUs were ranked by quantifying their contribution to overall risk-adjusted variation across the scales. Spearman rank correlation coefficients were used to test for consistency in scale performance. The authors then examined whether performance in the top four NICUs in one scale predicted top four performance in others.

RESULTS

There were 547 respondents in 12 NICUs. Of 15 NICU-level correlations in performance ranking, two were >0.7, seven were between 0.4 and 0.69, and the six remaining were <0.4. The authors found a trend towards significance in comparing the distribution of performance in the top four NICUs across domains with a binomial distribution p=0.051, indicating generally consistent performance across dimensions of safety culture.

CONCLUSION

A culture of safety permeates many aspects of patient care and organisational functioning. The SAQ may be a useful tool for comparative performance assessments among NICUs.

Design, rationale, and baseline characteristics of a cluster randomized controlled trial of pay for performance for hypertension treatment: study protocol

Background

Despite compelling evidence of the benefits of treatment and well-accepted guidelines for treatment, hypertension is controlled in less than one-half of United States citizens.

Methods/design

This randomized controlled trial tests whether explicit financial incentives promote the translation of guideline-recommended care for hypertension into clinical practice and improve blood pressure (BP) control in the primary care setting. Using constrained randomization, we assigned 12 Veterans Affairs hospital outpatient clinics to four study arms: physician-level incentive; group-level incentive; combination of physician and group incentives; and no incentives (control). All participants at the hospital (cluster) were assigned to the same study arm. We enrolled 83 full-time primary care physicians and 42 non-physician personnel. The intervention consisted of an educational session about guideline-recommended care for hypertension, five audit and feedback reports, and five disbursements of incentive payments. Incentive payments rewarded participants for chart-documented use of guideline-recommended antihypertensive medications, BP control, and appropriate responses to uncontrolled BP during a prior four-month performance period over the 20-month intervention. To identify potential unintended consequences of the incentives, the study team interviewed study participants, as well as non-participant primary care personnel and leadership at study sites. Chart reviews included data collection on quality measures not related to hypertension. To evaluate the persistence of the effect of the incentives, the study design includes a washout period.

Discussion

We briefly describe the rationale for the interventions being studied, as well as the major design choices. Rigorous research designs such as the one described here are necessary to determine whether performance-based payment arrangements such as financial incentives result in meaningful quality improvements.

Trial Registration

http://www.clinicaltrials.govNCT00302718

Towards successful coordination of electronic health record based-referrals: a qualitative analysis

BACKGROUND

Successful subspecialty referrals require considerable coordination and interactive communication among the primary care provider (PCP), the subspecialist, and the patient, which may be challenging in the outpatient setting. Even when referrals are facilitated by electronic health records (EHRs) (i.e., e-referrals), lapses in patient follow-up might occur. Although compelling reasons exist why referral coordination should be improved, little is known about which elements of the complex referral coordination process should be targeted for improvement. Using Okhuysen & Bechky's coordination framework, this paper aims to understand the barriers, facilitators, and suggestions for improving communication and coordination of EHR-based referrals in an integrated healthcare system.

METHODS

We conducted a qualitative study to understand coordination breakdowns related to e-referrals in an integrated healthcare system and examined work-system factors that affect the timely receipt of subspecialty care. We conducted interviews with seven subject matter experts and six focus groups with a total of 30 PCPs and subspecialists at two tertiary care Department of Veterans Affairs (VA) medical centers. Using techniques from grounded theory and content analysis, we identified organizational themes that affected the referral process.

RESULTS

Four themes emerged: lack of an institutional referral policy, lack of standardization in certain referral procedures, ambiguity in roles and responsibilities, and inadequate resources to adapt and respond to referral requests effectively. Marked differences in PCPs' and subspecialists' communication styles and individual mental models of the referral processes likely precluded the development of a shared mental model to facilitate coordination and successful referral completion. Notably, very few barriers related to the EHR were reported.

CONCLUSIONS

Despite facilitating information transfer between PCPs and subspecialists, e-referrals remain prone to coordination breakdowns. Clear referral policies, well-defined roles and responsibilities for key personnel, standardized procedures and communication protocols, and adequate human resources must be in place before implementing an EHR to facilitate referrals.

Understanding the management of electronic test result notifications in the outpatient setting

BACKGROUND

Notifying clinicians about abnormal test results through electronic health record (EHR) -based "alert" notifications may not always lead to timely follow-up of patients. We sought to understand barriers, facilitators, and potential interventions for safe and effective management of abnormal test result delivery via electronic alerts.

METHODS

We conducted a qualitative study consisting of six 6-8 member focus groups (N = 44) at two large, geographically dispersed Veterans Affairs facilities. Participants included full-time primary care providers, and personnel representing diagnostic services (radiology, laboratory) and information technology. We asked participants to discuss barriers, facilitators, and suggestions for improving timely management and follow-up of abnormal test result notifications and encouraged them to consider technological issues, as well as broader, human-factor-related aspects of EHR use such as organizational, personnel, and workflow.

RESULTS

Providers reported receiving a large number of alerts containing information unrelated to abnormal test results, many of which were believed to be unnecessary. Some providers also reported lacking proficiency in use of certain EHR features that would enable them to manage alerts more efficiently. Suggestions for improvement included improving display and tracking processes for critical alerts in the EHR, redesigning clinical workflow, and streamlining policies and procedures related to test result notification.

CONCLUSION

Providers perceive several challenges for fail-safe electronic communication and tracking of abnormal test results. A multi-dimensional approach that addresses technology as well as the many non-technological factors we elicited is essential to design interventions to reduce missed test results in EHRs.

Improving benchmarking by using an explicit framework for the development of composite indicators: an example using pediatric quality of care

Background

The measurement of healthcare provider performance is becoming more widespread. Physicians have been guarded about performance measurement, in part because the methodology for comparative measurement of care quality is underdeveloped. Comprehensive quality improvement will require comprehensive measurement, implying the aggregation of multiple quality metrics into composite indicators.

Objective

To present a conceptual framework to develop comprehensive, robust, and transparent composite indicators of pediatric care quality, and to highlight aspects specific to quality measurement in children.

Methods

We reviewed the scientific literature on composite indicator development, health systems, and quality measurement in the pediatric healthcare setting. Frameworks were selected for explicitness and applicability to a hospital-based measurement system.

Results

We synthesized various frameworks into a comprehensive model for the development of composite indicators of quality of care. Among its key premises, the model proposes identifying structural, process, and outcome metrics for each of the Institute of Medicine's six domains of quality (safety, effectiveness, efficiency, patient-centeredness, timeliness, and equity) and presents a step-by-step framework for embedding the quality of care measurement model into composite indicator development.

Conclusions

The framework presented offers researchers an explicit path to composite indicator development. Without a scientifically robust and comprehensive approach to measurement of the quality of healthcare, performance measurement will ultimately fail to achieve its quality improvement goals.

Provider management strategies of abnormal test result alerts: a cognitive task analysis

OBJECTIVE

Electronic medical records (EMRs) facilitate abnormal test result communication through "alert" notifications. The aim was to evaluate how primary care providers (PCPs) manage alerts related to critical diagnostic test results on their EMR screens, and compare alert-management strategies of providers with high versus low rates of timely follow-up of results.

DESIGN

28 PCPs from a large, tertiary care Veterans Affairs Medical Center (VAMC) were purposively sampled according to their rates of timely follow-up of alerts, determined in a previous study. Using techniques from cognitive task analysis, participants were interviewed about how and when they manage alerts, focusing on four alert-management features to filter, sort and reduce unnecessary alerts on their EMR screens.

RESULTS

Provider knowledge of alert-management features ranged between 4% and 75%. Almost half (46%) of providers did not use any of these features, and none used more than two. Providers with higher versus lower rates of timely follow-up used the four features similarly, except one (customizing alert notifications). Providers with low rates of timely follow-up tended to manually scan the alert list and process alerts heuristically using their clinical judgment. Additionally, 46% of providers used at least one workaround strategy to manage alerts.

CONCLUSION

Considerable heterogeneity exists in provider use of alert-management strategies; specific strategies may be associated with lower rates of timely follow-up. Standardization of alert-management strategies including improving provider knowledge of appropriate tools in the EMR to manage alerts could reduce the lack of timely follow-up of abnormal diagnostic test results.

Improving outpatient safety through effective electronic communication: a study protocol

Background

Health information technology and electronic medical records (EMRs) are potentially powerful systems-based interventions to facilitate diagnosis and treatment because they ensure the delivery of key new findings and other health related information to the practitioner. However, effective communication involves more than just information transfer; despite a state of the art EMR system, communication breakdowns can still occur. [1-3] In this project, we will adapt a model developed by the Systems Engineering Initiative for Patient Safety (SEIPS) to understand and improve the relationship between work systems and processes of care involved with electronic communication in EMRs. We plan to study three communication activities in the Veterans Health Administration's (VA) EMR: electronic communication of abnormal imaging and laboratory test results via automated notifications (i.e., alerts); electronic referral requests; and provider-to-pharmacy communication via computerized provider order entry (CPOE).

Aim

Our specific aim is to propose a protocol to evaluate the systems and processes affecting outcomes of electronic communication in the computerized patient record system (related to diagnostic test results, electronic referral requests, and CPOE prescriptions) using a human factors engineering approach, and hence guide the development of interventions for work system redesign.

Design

This research will consist of multiple qualitative methods of task analysis to identify potential sources of error related to diagnostic test result alerts, electronic referral requests, and CPOE; this will be followed by a series of focus groups to identify barriers, facilitators, and suggestions for improving the electronic communication system. Transcripts from all task analyses and focus groups will be analyzed using methods adapted from grounded theory and content analysis.