Reliability of Verbal Handoff Assessment and Handoff Quality Before and After Implementation of a Resident Handoff Bundle

Long-Term Sustainability and Adaptation of I-PASS Handovers

BACKGROUND

Inadequate communication during transitions of care is a major health care quality and safety vulnerability. In 2013 Massachusetts General Hospital (MGH) embarked on a comprehensive training program using a standardized handover system (I-PASS) that had been shown to reduce adverse events by 30% even when not completely executed on each patient. In this cross-sectional study, the authors sought to characterize handover practices six years later.

METHODS

Using a standardized interview tool, the researchers evaluated handovers between responding clinicians in 10 departments and then validated these findings through direct observations, allowing for flexibility and customization in the I-PASS elements. The study qualitatively compared I-PASS element use in verbal handovers to MGH early postintervention data, as well as verbal and written handovers with the I-PASS Study Group's postintervention results.

RESULTS

The authors observed 156 verbal and reviewed 182 written patient handovers. Ninety percent of departments adhered at least partially to the I-PASS system. Average handover duration ranged from 0.6 to 2.1 minutes per established patient. The service with best I-PASS adherence also consistently included the most information per unit of time. Acknowledging substantial differences in study technique, MGH adherence was, on average, comparable or better on all I-PASS elements in verbal handovers and on three of four elements of written handovers compared with the I-PASS Study Group's postintervention results.

CONCLUSION

Although uptake has varied across services, six years after hospitalwide implementation of I-PASS, the majority of services are performing structured and sequenced handovers, most of which include some elements of the I-PASS system. Those services with the best I-PASS adherence conducted the most efficient handovers.

It sounds like a good handover but can I trust it: the correlation between perceived quality and accuracy?

This article was migrated. The article was marked as recommended. Background Safe handover is crucial in healthcare and is taught in undergraduate and pre-vocational training curricula. It is now considered an Entrustable Professional Activity (EPA). Handover assessment tools have been developed but the correlation between the perceived quality of a handover and its accuracy has not been studied. Aims This paper aims to determine the correlation between the perceived quality and the accuracy and safety of handover. Methods This descriptive, quantitative study looked at medical students on long-term rural clinical placements who gave clinical handovers to supervisors. The supervisors scored the handovers using the Clinical Handover Assessment Tool (CHAT) and assessed the accuracy and safety of the handover, after seeing the patient. The correlation between handover scores, accuracy and safety was calculated using Cramer's V coefficient. Results 114 handovers from 25 students were assessed. The correlation coefficient for a global assessment of quality and accuracy was 0.585 and for safety was 0.583, considered large effects (>0.35). This also held using a checklist quality assessment but less strongly: 0.419, 0.363 respectively. Conclusion These findings suggest that handovers that sound 'good' are likely to be accurate: clinicians can 'trust their gut-feeling'. A high quality handover reflects more than the trainee'. clinical reasoning, communication and organisational skills: it suggests that they can provide accurate and safe handover. This supports the use of global assessments of handover as an important part of the multi-source feedback required for summative entrustment decision-making.

A Handoffs Software Led to Fewer Errors of Omission and Better Provider Satisfaction: A Randomized Control Trial

BACKGROUND

Computer-assisted communication is shown to prevent critical omissions ("errors") in the handoff process.

OBJECTIVE

The aim of the study was to study this effect and related provider satisfaction, using a standardized software.

METHODS

Fourteen internal medicine house officers staffed 6 days and 1 cross-covering teams were randomized to either the intervention group or control, employing usual handoff, so that handoff information was exchanged only between same-group subjects (daily, for 28 days).

RESULTS

In the intervention group, fewer omissions (among those studied) occurred intravenous access (17 versus 422, P < 0.001), code status (1 versus 158, P < 0.001), diet/nothing per mouth (28 versus 477, P < 0.001), and deep venous thrombosis prophylaxis (17 versus 284, P < 0.001); duration to compose handoff was similar; and physicians perceived less workload adjusted for patient census and provider characteristics (P = 0.004) as well as better handoff quality (P < 0.001) and clarity (P < 0.001).

CONCLUSIONS

The intervention was associated with fewer errors and superior provider satisfaction.

Faculty and Resident Engagement With a Workplace-Based Assessment Tool: Use of Implementation Science to Explore Enablers and Barriers

PURPOSE

Implementation of workplace-based assessment programs has encountered significant challenges. Faculty and residents alike often have a negative view of these programs as "tick-box" or "jump through the hoops" exercises. A number of recommendations have been made to address these challenges. To understand the experience with a workplace-based assessment tool that follows many of these recommendations, the authors conducted a qualitative study using the Consolidated Framework for Implementation Research (CFIR) to identify enablers and barriers to engagement with the tool.

METHOD

The Psychopharmacotherapy-Structured Clinical Observation (P-SCO) is a direct observation tool designed to assess resident performance during a psychiatric medication management visit. From August 2017 to February 2018, the P-SCO was implemented in the outpatient continuity clinics for second- and third-year residents at Zucker Hillside Hospital/Northwell Health. In February and March 2019, the authors conducted semistructured interviews of participating faculty and residents. Interview guides based on the CFIR were used to capture the enablers and barriers to engagement. Interview transcripts were independently coded. Codes were then organized into themes relevant to the domains of the CFIR.

RESULTS

Ten faculty and 10 residents were interviewed. Overall, participants had a positive experience with the P-SCO. Enabling factors for faculty and residents included the ongoing training, design features of the P-SCO, predisposing beliefs, dedicated faculty time, and the perception that the P-SCO improved verbal feedback quality. Barriers for faculty included checklist length and discomfort with feedback that threatens identity, and barriers for residents included faculty variability in timeliness and quality of feedback and minimal review of the feedback after initial receipt.

CONCLUSIONS

This study demonstrates that the negative experience of faculty and residents with workplace-based assessment tools shown in prior studies can be overcome, at least in part, when specific implementation strategies are pursued. The findings provide guidance for future research and implementation efforts.

A mobile app to capture EPA assessment data: Utilizing the consolidated framework for implementation research to identify enablers and barriers to engagement

INTRODUCTION

Mobile apps that utilize the framework of entrustable professional activities (EPAs) to capture and deliver feedback are being implemented. If EPA apps are to be successfully incorporated into programmatic assessment, a better understanding of how they are experienced by the end-users will be necessary. The authors conducted a qualitative study using the Consolidated Framework for Implementation Research (CFIR) to identify enablers and barriers to engagement with an EPA app.

METHODS

Structured interviews of faculty and residents were conducted with an interview guide based on the CFIR. Transcripts were independently coded by two study authors using directed content analysis. Differences were resolved via consensus. The study team then organized codes into themes relevant to the domains of the CFIR.

RESULTS

Eight faculty and 10 residents chose to participate in the study. Both faculty and residents found the app easy to use and effective in facilitating feedback immediately after the observed patient encounter. Faculty appreciated how the EPA app forced brief, distilled feedback. Both faculty and residents expressed positive attitudes and perceived the app as aligned with the department's philosophy. Barriers to engagement included faculty not understanding the EPA framework and scale, competing clinical demands, residents preferring more detailed feedback and both faculty and residents noting that the app's feedback should be complemented by a tool that generates more systematic, nuanced, and comprehensive feedback. Residents rarely if ever returned to the feedback after initial receipt.

DISCUSSION

This study identified key enablers and barriers to engagement with the EPA app. The findings provide guidance for future research and implementation efforts focused on the use of mobile platforms to capture direct observation feedback.

Utilizing Electronic Medical Records to Standardize Handoffs in Academic Ophthalmology

Purpose Formalized handoff procedures have been shown to increase patient safety and quality of care across multiple medical and surgical specialties,1–4 but literature regarding handoffs in ophthalmology remains sparse. We instituted a standardized handoff utilizing an electronic medical record (EMR) system to improve care for patients shared by multiple resident physicians across weekday, weeknight, and weekend duty shifts. We measured efficiency, efficacy, and resident satisfaction before and after the standardized handoff was implemented.

Methods Resident physicians surveyed were primarily responsible for patient care on consult and call services at two quaternary academic medical centers in a major metropolitan area. Patient care was performed in outpatient, emergency, and inpatient settings. Annual anonymous questionnaires consisting of 6 questions were used to collect pre- and postintervention impressions of the standardized EMR handoff process from ophthalmology resident physicians (9 per year; 3 preintervention years and 1 postintervention year). An additional anonymous postintervention questionnaire consisting of 12 questions was used to further characterize resident response to the newly implemented handoff procedure.

Results Prior to implementation of a standardized EMR-based handoff procedure, residents unanimously reported incomplete, infrequently updated handoff reports that did not include important clinical and/or psychosocial information. Following implementation, residents reported a statistically significant increase in completeness and timeliness of handoff reports. Additionally, resident perception of EMR handoff utility, efficiency, and usability were comprehensively favorable. Residents reported handoffs only added a mean of 6.5 minutes to a typical duty shift.

Conclusion Implementation of our protocol dramatically improved resident perceptions of the handoff process at our institution. Improvements included increased quality, ease-of-use, and efficiency. Our standardized EMR-based handoff procedure may be of use to other ambulatory-based services.

Advancing Our Understanding of Narrative Comments Generated by Direct Observation Tools: Lessons From the Psychopharmacotherapy-Structured Clinical Observation

BACKGROUND

While prior research has focused on the validity of quantitative ratings generated by direct observation tools, much less is known about the written comments.

OBJECTIVE

This study examines the quality of written comments and their relationship with checklist scores generated by a direct observation tool, the Psychopharmacotherapy-Structured Clinical Observation (P-SCO).

METHODS

From 2008 to 2012, faculty in a postgraduate year 3 psychiatry outpatient clinic completed 601 P-SCOs. Twenty-five percent were randomly selected from each year; the sample included 8 faculty and 57 residents. To assess quality, comments were coded for valence (reinforcing or corrective), behavioral specificity, and content. To assess the relationship between comments and scores, the authors calculated the correlation between comment and checklist score valence and examined the degree to which comments and checklist scores addressed the same content.

RESULTS

Ninety-one percent of the comments were behaviorally specific. Sixty percent were reinforcing, and 40% were corrective. Eight themes were identified, including 2 constructs not adequately represented by the checklist. Comment and checklist score valence was moderately correlated (Spearman's rho = 0.57, P < .001). Sixty-seven percent of high and low checklist scores were associated with a comment of the same valence and content. Only 50% of overall comments were associated with a checklist score of the same valence and content.

CONCLUSIONS

A direct observation tool such as the P-SCO can generate high-quality written comments. Narrative comments both explain checklist scores and convey unique content. Thematic coding of comments can improve the content validity of a checklist.

Communication at Transitions of Care

Communication errors during transitions of care are a leading source of adverse events for hospitalized patients. This article provides an overview of the role of communication errors in adverse events, describes the complexities of communication for hospitalized patients, and provides evidence regarding the positive effects of applying high-reliability principles to transitions of care and culture of safety. Elements of effective handoffs and a detailed approach for successful implementation of a handoff program are provided. The role of handoff communication in medical education at all levels, as well as for the interprofessional team, is discussed.

Evidence for the Validity of the Psychopharmacotherapy-Structured Clinical Observation Tool: Results of a Factor and Time Series Analysis

OBJECTIVE

The Psychopharmacotherapy-Structured Clinical Observation (P-SCO) is a direct observation tool designed to assess resident performance of a medication visit. This study examines two dimensions of validity for the P-SCO: internal structure and how scores correlate with another variable associated with competence (experience).

METHODS

The faculty completed 601 P-SCOs over 4 years. Multilevel exploratory factor analysis was performed with minimum thresholds for eigenvalue (≥ 1.0) and proportion of variance explained (≥ 5.0%). Internal reliability was assessed with Cronbach alpha. To examine how scores changed with experience, mean ratings (1-4 scale) were calculated for each factor by quarter of the academic year. Separate linear mixed models were also performed.

RESULTS

The analysis yielded three factors that explained 50% of the variance and demonstrated high internal reliability: affective tasks (alpha = 0.90), cognitive tasks (alpha = 0.84), and hard tasks (alpha = 0.74). Items within "hard tasks" were assessment of substance use, violence risk, and adherence, and inquiry about interactions with other providers. Monitoring adverse effects did not load on the hard task factor but also had overall low mean ratings. Compared to the first quarter, fourth quarter scores for affective tasks (b = 0.54, p < 0.01) and hard tasks (b = 0.46, p = 0.02) were significantly improved while cognitive tasks had a non-significant increase. For the hard tasks, the proportion of residents with a low mean rating improved but was still over 30% during the fourth quarter.

CONCLUSIONS

The results provide evidence for the validity of the P-SCO with respect to its internal structure and how scores correlate with experience. Curricular implications are explored, especially for the tasks that were hard to learn.

Novel Transfer of Care Sign-out Assessment Tool in a Pediatric Emergency Department

OBJECTIVE

Transfer of care sign-outs (TOCS) for admissions from a pediatric emergency department have unique challenges. Standardized and reliable assessment tools for TOCS remain elusive. We describe the development, reliability, and validity of a TOCS assessment tool.

METHODS

Video recordings of resident TOCS were assessed to capture 4 domains: completeness, synopsis, foresight, and professionalism. In phase 1, 56 TOCS were used to modify the tool and improve reliability. In phase 2, 91 TOCS were used to examine validity. Analyses included Cronbach's alpha for internal structure, intraclass correlation and Cohen's kappa for interrater reliability, Pearson's correlation for relationships between variables, and 95% confidence interval of the mean for resident group comparisons.

RESULTS

Cronbach's alpha was 0.52 for internal structure of the tool's subjective rating scale. Intraclass correlation for the subjective rating scale items ranged from 0.70 to 0.80. Cohen's kappa for most objective checklist items ranged from 0.43 to 1. Content completeness was significantly correlated with synopsis, foresight, and professionalism (Pearson's r ranged from 0.36 to 0.62, P values were <0.001). House staff senior residents scored higher (on average) than interns and rotating senior residents in synopsis and foresight. Also, house staff interns scored higher (on average) than rotating senior residents in professionalism. House staff senior residents scored higher (on average) than rotating senior residents in content completeness.

CONCLUSIONS

We provide validity evidence to support using scores from the TOCS tool to assess higher-level transfer of care comprehension and communication by pediatric emergency department residents and to test interventions to improve TOCS.

Am I getting an accurate picture: a tool to assess clinical handover in remote settings?

BACKGROUND

Good clinical handover is critical to safe medical care. Little research has investigated handover in rural settings. In a remote setting where nurses and medical students give telephone handover to an aeromedical retrieval service, we developed a tool by which the receiving clinician might assess the handover; and investigated factors impacting on the reliability and validity of that assessment.

METHODS

Researchers consulted with clinicians to develop an assessment tool, based on the ISBAR handover framework, combining validity evidence and the existing literature. The tool was applied 'live' by receiving clinicians and from recorded handovers by academic assessors. The tool's performance was analysed using generalisability theory. Receiving clinicians and assessors provided feedback.

RESULTS

Reliability for assessing a call was good (G = 0.73 with 4 assessments). The scale had a single factor structure with good internal consistency (Cronbach's alpha = 0.8). The group mean for the global score for nurses and students was 2.30 (SD 0.85) out of a maximum 3.0, with no difference between these sub-groups.

CONCLUSIONS

We have developed and evaluated a tool to assess high-stakes handover in a remote setting. It showed good reliability and was easy for working clinicians to use. Further investigation and use is warranted beyond this setting.