The use of error analysis to assess resident performance

Impact of practicing internal benchmarking on continuous improvement of cataract surgery outcomes: a retrospective observational study at Aravind Eye Hospitals, India

OBJECTIVE

We aim to assess the effectiveness of a cataract surgery outcome monitoring tool used for continuous quality improvement. The objectives are to study: (1) the quality parameters, (2) the monitoring process followed and (3) the impact on outcomes.

DESIGN AND PROCEDURES

In this retrospective observational study we evaluated a quality improvement (QI) method which has been practiced at the focal institution since 2012: internal benchmarking of cataract surgery outcomes (CATQA). We evaluated quality parameters, procedures followed and clinical outcomes. We created tables and line charts to examine trends in key outcomes.

SETTING

Aravind Eye Care System, India.

PARTICIPANTS

Phacoemulsification surgeries performed on 718 120 eyes at 10 centres (five tertiary and five secondary eye centres) from 2012 to 2020 were included.

INTERVENTIONS

An internal benchmarking of surgery outcome parameters, to assess variations among the hospitals and compare with the best hospital.

OUTCOME MEASURES

Intraoperative complications, unaided visual acuity (VA) at postoperative follow-up visit and residual postoperative refractive error (within ±0.5D).

RESULTS

Over the study period the intraoperative complication rate decreased from 1.2% to 0.6%, surgeries with uncorrected VA of 6/12 or better increased from 80.8% to 89.8%, and surgeries with postoperative refractive error within ±0.5D increased from 76.3% to 87.3%. Variability in outcome measures across hospitals declined. Additionally, benchmarking was associated with improvements in facilities, protocols and processes.

CONCLUSION

Internal benchmarking was found to be an effective QI method that enabled the practice of evidence-based management and allowed for harnessing the available information. Continuous improvement in clinical outcomes requires systematic and regular review of results, identifying gaps between hospitals, comparisons with the best hospital and implementing lessons learnt from peers.

The concept of errors in medical education: a scoping review

The purpose of this scoping review was to explore how errors are conceptualized in medical education contexts by examining different error perspectives and practices. This review used a scoping methodology with a systematic search strategy to identify relevant studies, written in English, and published before January 2021. Four medical education journals (Medical Education, Advances in Health Science Education, Medical Teacher, and Academic Medicine) and four clinical journals (Journal of the American Medical Association, Journal of General Internal Medicine, Annals of Surgery, and British Medical Journal) were purposively selected. Data extraction was charted according to a data collection form. Of 1505 screened studies, 79 studies were included. Three overarching perspectives were identified: 'understanding errors') (n = 31), 'avoiding errors' (n = 25), 'learning from errors' (n = 23). Studies that aimed at'understanding errors' used qualitative methods (19/31, 61.3%) and took place in the clinical setting (19/31, 61.3%), whereas studies that aimed at 'avoiding errors' and 'learning from errors' used quantitative methods ('avoiding errors': 20/25, 80%, and 'learning from errors': 16/23, 69.6%, p = 0.007) and took place in pre-clinical (14/25, 56%) and simulated settings (10/23, 43.5%), respectively (p < 0.001). The three perspectives differed significantly in terms of inclusion of educational theory: 'Understanding errors' studies 16.1% (5/31),'avoiding errors' studies 48% (12/25), and 'learning from errors' studies 73.9% (17/23), p < 0.001. Errors in medical education and clinical practice are defined differently, which makes comparisons difficult. A uniform understanding is not necessarily a goal but improving transparency and clarity of how errors are currently conceptualized may improve our understanding of when, why, and how to use and learn from errors in the future.

Performance assessment using sensor technology

Over the past 30 years, there have been numerous, noteworthy successes in the development, validation, and implementation of clinical skills assessments. Despite this progress, the medical profession has barely scratched the surface towards developing assessments that capture the true complexity of hands-on skills in procedural medicine. This paper highlights the development implementation and new discoveries in performance metrics when using sensor technology to assess cognitive and technical aspects of hands-on skills.

Surgical Errors Happen, but Are Learners Trained to Recover from Them? A Survey of North American Surgical Residents and Fellows

BACKGROUND

Surgical training necessitates graded supervision and supported independence in order to reach competence. In developing surgical skills, trainees can, and will, make mistakes. A key skill required for independent practice is the ability to recover from an error or unexpected complication. Error recovery includes recognizing and managing a technical error in order to ensure patient safety and may be underrepresented in current educational approaches.

OBJECTIVE

The purpose of this study is to explore residents' experiences and perceptions of error recovery training in surgical procedures.

METHOD

An online survey was sent to surgical program directors in the United States and Canada using the Accreditation Council for Graduate Medical Education and the Royal College of Physicians and Surgeons of Canada distribution lists. Participating programs distributed the survey to their residents and fellows. The survey was composed of Likert-scale items, yes/no questions as well as open-ended questions focused on perceptions, experiences, and factors that influence to error recovery training in the operating room.

RESULTS

A total of 206 surveys were completed. Overall, 99% (n = 203) agreed or strongly agreed that error recovery is an important competency for future practice. This was reflected in free-text response: "Errors can be minimized but they are inevitable, so certainly believe a surgical curriculum that addresses error recovery is of paramount importance." While 83% (n = 170) feel confident recovering from minor errors, only 34% (n = 68) feel confident that they could recover from major errors that are likely to have serious consequences on patient safety. Overall, residents do not consider that they have adequate training in error recovery, with only 37% (n = 72) felt they were adequately trained to recover from major errors. It was also mentioned "The quality of learning regarding error recovery depends entirely on the attending."

CONCLUSIONS

Opportunities to learn to recover from technical errors in the operating room are valued by surgical trainees, but they perceive their training to be both inadequate and variable. This contributes to a lack of confidence in error recovery skills throughout their surgical training. There is a need to explore how best to integrate error recovery into more formal surgical curricula in order to better support learners and, ultimately, contribute to increased surgical safety.

Distinguishing Intermediate and Novice Surgeons by Eye Movements

Surgical skill-level assessment is key to collecting the required feedback and adapting the educational programs accordingly. Currently, these assessments for the minimal invasive surgery programs are primarily based on subjective methods, and there is no consensus on skill level classifications. One of the most detailed of these classifications categorize skill levels as beginner, novice, intermediate, sub-expert, and expert. To properly integrate skill assessment into minimal invasive surgical education programs and provide skill-based training alternatives, it is necessary to classify the skill levels in as detailed a way as possible and identify the differences between all skill levels in an objective manner. Yet, despite the existence of very encouraging results in the literature, most of the studies have been conducted to better understand the differences between novice and expert surgical skill levels leaving out the other crucial skill levels between them. Additionally, there are very limited studies by considering the eye-movement behaviors of surgical residents. To this end, the present study attempted to distinguish novice- and intermediate-level surgical residents based on their eye movements. The eye-movement data was recorded from 23 volunteer surgical residents while they were performing four computer-based simulated surgical tasks under different hand conditions. The data was analyzed using logistic regression to estimate the skill levels of both groups. The best results of the estimation revealing a 91.3% recognition rate of predicting novice and intermediate surgical residents on one scenario were selected from four under the dominant hand condition. These results show that the eye-movements can be potentially used to identify surgeons with intermediate and novice skills. However, the results also indicate that the order in which the scenarios are provided, and the design of the scenario, the tasks, and their appropriateness with the skill levels of the participants are all critical factors to be considered in improving the estimation ratio, and hence require thorough assessment for future research.

Medication error encouragement training: A quasi-experimental study

BACKGROUND

Medication errors are the most common clinical errors in healthcare practice and can lead to serious consequences. Medication error encouragement training (MEET) brings students face-to-face with potential errors in the medication process, in a safe environment where they are encouraged to understand both the error and the context in which it occurred.

OBJECTIVES

The study aimed to examine the effects of a MEET intervention on medication safety confidence among nursing undergraduates.

DESIGN

This was a quasi-experimental study with a nonequivalent control group design.

PARTICIPANTS

Our sample was recruited from the nursing education department of a university, with 47 participants randomly assigned to the experimental group, and 50 to the control group.

METHODS

Both groups received theoretical training, followed by applied training. The experimental group received the MEET intervention developed specifically for this study, while the control group received traditional error avoidance training. Participants' medication administration confidence was measured pre- and post-intervention.

RESULTS

Following training, the experimental group's confidence was significantly higher than that of the control group. With regard to individual medication administration procedures, the experimental groups' medication safety confidence increased significantly after training compared to the control group in patient identification, drug information confirmation, and drug preparation.

CONCLUSIONS

Introducing MEET into nursing curricula could reduce medication errors and related complications in healthcare institutions. Further studies are needed to investigate the long-term effects of MEET interventions, as well as the generalizability of our findings.

Exploring Senior Residents' Intraoperative Error Management Strategies: A Potential Measure of Performance Improvement

OBJECTIVE

The study aim was to determine whether residents' error management strategies changed across 2 simulated laparoscopic ventral hernia (LVH) repair procedures after receiving feedback on their initial performance. We hypothesize that error detection and recovery strategies would improve during the second procedure without hands-on practice.

DESIGN

Retrospective review of participant procedural performances of simulated laparoscopic ventral herniorrhaphy. A total of 3 investigators reviewed procedure videos to identify surgical errors. Errors were deconstructed. Error management events were noted, including error identification and recovery.

SETTING

Residents performed the simulated LVH procedures during a course on advanced laparoscopy. Participants had 30 minutes to complete a LVH procedure. After verbal and simulator feedback, residents returned 24 hours later to perform a different, more difficult simulated LVH repair.

PARTICIPANTS

Senior (N = 7; postgraduate year 4-5) residents in attendance at the course participated in this study.

RESULTS

In the first LVH procedure, residents committed 121 errors (M = 17.14, standard deviation = 4.38). Although the number of errors increased to 146 (M = 20.86, standard deviation = 6.15) during the second procedure, residents progressed further in the second procedure. There was no significant difference in the number of errors committed for both procedures, but errors shifted to the late stage of the second procedure. Residents changed the error types that they attempted to recover (χ²₅=24.96, p<0.001). For the second procedure, recovery attempts increased for action and procedure errors, but decreased for strategy errors. Residents also recovered the most errors in the late stage of the second procedure (p < 0.001).

CONCLUSION

Residents' error management strategies changed between procedures following verbal feedback on their initial performance and feedback from the simulator. Errors and recovery attempts shifted to later steps during the second procedure. This may reflect residents' error management success in the earlier stages, which allowed further progression in the second simulation. Incorporating error recognition and management opportunities into surgical training could help track residents' learning curve and provide detailed, structured feedback on technical and decision-making skills.

Error tolerance: an evaluation of residents' repeated motor coordination errors

BACKGROUND

The study investigates the relationship between motor coordination errors and total errors using a human factors framework. We hypothesize motor coordination errors will correlate with total errors and provide validity evidence for error tolerance as a performance metric.

METHODS

Residents' laparoscopic skills were evaluated during a simulated laparoscopic ventral hernia repair for motor coordination errors when grasping for intra-abdominal mesh or suture. Tolerance was defined as repeated, failed attempts to correct an error and the time required to recover.

RESULTS

Residents (N = 20) committed an average of 15.45 (standard deviation [SD] = 4.61) errors and 1.70 (SD = 2.25) motor coordination errors during mesh placement. Total errors correlated with motor coordination errors (r[18] = .572, P = .008). On average, residents required 5.09 recovery attempts for 1 motor coordination error (SD = 3.15). Recovery approaches correlated to total error load (r[13] = .592, P = .02).

CONCLUSIONS

Residents' motor coordination errors and recovery approaches predict total error load. Error tolerance proved to be a valid assessment metric relating to overall performance.

Advanced Engineering Technology for Measuring Performance

The demand for competency-based assessments in surgical training is growing. Use of advanced engineering technology for clinical skills assessment allows for objective measures of hands-on performance. Clinical performance can be assessed in several ways via quantification of an assessee's hand movements (motion tracking), direction of visual attention (eye tracking), levels of stress (physiologic marker measurements), and location and pressure of palpation (force measurements). Innovations in video recording technology and qualitative analysis tools allow for a combination of observer- and technology-based assessments. Overall the goal is to create better assessments of surgical performance with robust validity evidence.