Video Assessment of Surgeons and Surgery

Spatial-temporal attention for video-based assessment of intraoperative surgical skill

Accurate, unbiased, and reproducible assessment of skill is a vital resource for surgeons throughout their career. The objective in this research is to develop and validate algorithms for video-based assessment of intraoperative surgical skill. Algorithms to classify surgical video into expert or novice categories provide a summative assessment of skill, which is useful for evaluating surgeons at discrete time points in their training or certification of surgeons. Using a spatial-temporal neural network architecture, we tested the hypothesis that explicit supervision of spatial attention supervised by instrument tip locations improves the algorithm's generalizability to unseen dataset. The best performing model had an area under the receiver operating characteristic curve (AUC) of 0.88. Augmenting the network with supervision of spatial attention improved specificity of its predictions (with small changes in sensitivity and AUC) and led to improved measures of discrimination when tested with unseen dataset. Our findings show that explicit supervision of attention learned from images using instrument tip locations can improve performance of algorithms for objective video-based assessment of surgical skill.

A Video-Based Procedure-Specific Competency Assessment Tool for Minimally Invasive Esophagectomy

Importance

Minimally invasive esophagectomy (MIE) is a complex procedure with substantial learning curves. In other complex minimally invasive procedures, suboptimal surgical performance has convincingly been associated with less favorable patient outcomes as assessed by peer review of the surgical procedure.

Objective

To develop and validate a procedure-specific competency assessment tool (CAT) for MIE.

Design, Setting, and Participants

In this international quality improvement study, a procedure-specific MIE-CAT was developed and validated. The MIE-CAT contains 8 procedural phases, and 4 quality components per phase are scored with a Likert scale ranging from 1 to 4. For evaluation of the MIE-CAT, intraoperative MIE videos performed by a single surgical team in the Esophageal Center East Netherlands were peer reviewed by 18 independent international MIE experts (with more than 120 MIEs performed). Each video was assessed by 2 or 3 blinded experts to evaluate feasibility, content validity, reliability, and construct validity. MIE-CAT version 2 was composed with refined content aimed at improving interrater reliability. A total of 32 full-length MIE videos from patients who underwent MIE between 2011 and 2020 were analyzed. Data were analyzed from January 2021 to January 2023.

Exposure

Performance assessment of transthoracic MIE with an intrathoracic anastomosis.

Main Outcomes and Measures

Feasibility, content validity, interrater and intrarater reliability, and construct validity, including correlations with both experience of the surgical team and clinical parameters, of the developed MIE-CAT.

Results

Experts found the MIE-CAT easy to understand and easy to use to grade surgical performance. The MIE-CAT demonstrated good intrarater reliability (range of intraclass correlation coefficients [ICCs], 0.807 [95% CI, 0.656 to 0.892] for quality component score to 0.898 [95% CI, 0.846 to 0.932] for phase score). Interrater reliability was moderate (range of ICCs, 0.536 [95% CI, -0.220 to 0.994] for total MIE-CAT score to 0.705 [95% CI, 0.473 to 0.846] for quality component score), and most discrepancies originated in the lymphadenectomy phases. Hypothesis testing for construct validity showed more than 75% of hypotheses correct: MIE-CAT performance scores correlated with experience of the surgical team (r = 0.288 to 0.622), blood loss (r = -0.034 to -0.545), operative time (r = -0.309 to -0.611), intraoperative complications (r = -0.052 to -0.319), and severe postoperative complications (r = -0.207 to -0.395). MIE-CAT version 2 increased usability. Interrater reliability improved but remained moderate (range of ICCs, 0.666 to 0.743), and most discrepancies between raters remained in the lymphadenectomy phases.

Conclusions and Relevance

The MIE-CAT was developed and its feasibility, content validity, reliability, and construct validity were demonstrated. By providing insight into surgical performance of MIE, the MIE-CAT might be used for clinical, training, and research purposes.

Comparative study of video recordings of non-medical devices in laparoscopic surgery: a cross-sectional study

INTRODUCTION

Laparoscopic surgery is the approach of choice for multiple procedures, being laparoscopic cholecystectomy one of the most frequently performed surgeries. Likewise, video recording of these surgeries has become widespread. Currently, the market offers medical recording devices (MRD) with an approximate cost of 2000 USD, and alternative non-medical recording devices (NMRD) with a cost ranging from 120 to 200 USD. To our knowledge, no comparative studies between the available recording devices have been done. We aim to compare the perception of the quality of videos recorded by MRD and NMRD in a group of surgeons and surgical residents.

METHODS

A cross-sectional study was conducted using an online survey to compare recordings from three NMRDs (Elgato 30 fps, AverMedia 60 fps, Hauppauge 30 fps) and one MRD (MediCap 20 fps) during a laparoscopic cholecystectomy. The survey assessed: definition of anatomical structures (DA), fluidity of movements (FM), similarity with the operating room screen (ORsim), and overall quality (OQ). Descriptive and nonparametric analytical statistics tests were applied. Results were analyzed using JMP-15 software.

RESULTS

Forty surveys were collected (80% surgeons, 20% residents). NMRDs scored significantly higher than MRD in DA (p = 0.003), FM (p < 0.001), ORsim (p < 0.001), and OQ (p < 0.001). One NMRD was chosen as the highest quality device (70%), and MRD as the poorest (78%). No significant differences were found when analyzing by surgical experience.

CONCLUSIONS

In terms of recording laparoscopic procedures, non-medical video recording devices (NMRDs) outperformed medical-grade recording device (MRD) with a higher overall score. This suggests that NMRDs could serve as a cost-effective alternative with superior video quality for recording laparoscopic surgeries.

A National Comparison of Volume and Acuity for Adult and Pediatric Trauma: A Trauma Quality Improvement Program Cohort Study

INTRODUCTION

Most injured children receive trauma care outside of a pediatric trauma center. Differences in physiology, dosing, and injury pattern limit extrapolation of adult trauma principles to pediatrics. We compare US trauma center experience with pediatric and adult trauma resuscitation.

MATERIALS AND METHODS

We queried the 2019 Trauma Quality Improvement Program to describe the experience of US trauma centers with pediatric (<15 y) and adult trauma. We quantified blunt, penetrating, burn, and unspecified traumas and compared minor, moderate, severe, and critical traumas (ISS 1-8 Minor, ISS 9-14 Moderate, ISS 15-24 Severe, ISS 25+ Critical). We estimated center-level volumes for adults and children. Institutional identifiers were generated based on unique center specific factors including hospital teaching status, hospital type, verification level, pediatric verification level, state designation, state pediatric designation, and bed size.

RESULTS

A total of 755,420 adult and 76,449 pediatric patients were treated for traumatic injuries. There were 21 times as many critical or major injuries in adults compared to children, 17 times more moderate injuries, and 6 times more minor injuries. Children and adults presented with similar rates of blunt trauma, but penetrating injuries were more common in adults and burn injuries were more common in children. Comparing center-level data, adult trauma exceeded pediatric for every severity and mechanism.

CONCLUSIONS

There is relatively limited exposure to high-acuity pediatric trauma at US centers. Investigation into pediatric trauma resuscitation education and simulation may promote pediatric readiness and lead to improved outcomes.

Measuring Quality at the Surgeon Level

Patients expect high-quality surgical care and increasingly are looking for ways to assess the quality of the surgeon they are seeing, but quality measurement is often more complicated than one might expect. Measurement of individual surgeon quality in a manner that allows for comparison among surgeons is particularly difficult. While the concept of measuring individual surgeon quality has a long history, technology now allows for new and innovative ways to measure and achieve surgical excellence. However, some recent efforts to make surgeon-level quality data publicly available have highlighted the challenges of this work. Through this chapter, the reader will be introduced to a brief history of surgical quality measurement, learn about the current state of quality measurement, and get a glimpse into what the future holds.

Intraoperative video recording in otolaryngology for surgical education: evolution and considerations

BACKGROUND

Otolaryngology is a surgical speciality well suited for the application of intraoperative video recording as an educational tool considering the number procedures within the speciality that utilize digital technology. Intraoperative recording has been utilized in endoscopic surgeries and in evaluating technique in mastoidectomy, myringotomy and grommet insertion. The impact of intra-operative video recording in otolaryngology education is vast in creating access to surgical videos for preparation outside the operating room to individualized coaching and assessment. The purpose of this project is to highlight the role of intraoperative video recording in otolaryngology training and elucidate the challenges and considerations associated with implementation.

METHODS

Related publications between 1999 to 2022 were reviewed from PubMed and Embase databases utilizing search terms "intraoperative videography," "video recording surgery," "otolaryngology," and "surgical education." 109 articles were screened independently by HB and SK, by title and abstract then full text review. 28 articles from the original search and 6 from the secondary reference review were included.

RESULTS

The application of intraoperative video recording is evident in otolaryngology surgeries including endoscopic sinus surgery, laryngeal surgery, and other endoscopic procedures. There have been significant advancements in recording tools, including devices that can capture the surgeon's perspective. The considerations and challenges identified with utilizing this educational tool were categorized into different themes including ethics/consent, regulation, liability, data, technology, and human resources.

CONCLUSION

Intra-operative video recording has been demonstrated to have significant impact within otolaryngology education. It is critical to elucidate the challenges and considerations involved to utilize this educational tool effectively. Future directives will see video-based performance analytics providing comparative metrics to encourage precise coaching of surgical residents.

AHPBA senior leaders' assessments of strengths, weaknesses, opportunities, and threats facing fellowship training in HPB surgery: "We need to standardize our training experiences"

BACKGROUND

Multiple fellowship programs in North America prepare surgeons for a career in Hepato-Pancreatico-Biliary (HPB) surgery. Inconsistent operative experiences and disease process exposures across programs and pathways produces variability in training product and therefore, lack of clarity around what trained HPB surgeons are prepared to do in early practice. Thus, a strengths, weaknesses, opportunities, and threats (SWOT) analysis of AHPBA fellowship training was conducted.

METHODS

This was a mixed-methods, cross-sectional study. Eleven AHPBA-Founding Members (FM) and 24 current or former Program Directors (PD) of programs eligible for AHPBA certificates were surveyed and interviewed. Grounded theory principles and thematic network analysis were used to analyze interview transcripts. Descriptive statistics were used to analyze survey data.

RESULTS

Three main themes were identified: (i) Concern for training rigor and consistency (ii) Desire to standardize curricula and broaden training requirements and, (iii) Need to validate both the value of training and job marketability via certification.

DISCUSSION

Based on the themes identified, the strengths of AHPBA-certified HPB programs include superior technical training and case volumes. Areas of improvement included elevating baseline competencies by increasing required case volume and breadth to ensure minimally invasive experience, operative autonomy, and multidisciplinary care coordination.

The association between video-based assessment of intraoperative technical performance and patient outcomes: a systematic review

BACKGROUND

Efforts to improve surgical safety and outcomes have traditionally placed little emphasis on intraoperative performance, partly due to difficulties in measurement. Video-based assessment (VBA) provides an opportunity for blinded and unbiased appraisal of surgeon performance. Therefore, we aimed to systematically review the existing literature on the association between intraoperative technical performance, measured using VBA, and patient outcomes.

METHODS

Major databases (Medline, Embase, Cochrane Database, and Web of Science) were systematically searched for studies assessing the association of intraoperative technical performance measured by tools supported by validity evidence with short-term (≤ 30 days) and/or long-term postoperative outcomes. Study quality was assessed using the Newcastle-Ottawa Scale. Results were appraised descriptively as study heterogeneity precluded meta-analysis.

RESULTS

A total of 11 observational studies were identified involving 8 different procedures in foregut/bariatric (n = 4), colorectal (n = 4), urologic (n = 2), and hepatobiliary surgery (n = 1). The number of surgeons assessed ranged from 1 to 34; patient sample size ranged from 47 to 10,242. High risk of bias was present in 5 of 8 studies assessing short-term outcomes and 2 of 6 studies assessing long-term outcomes. Short-term outcomes were reported in 8 studies (i.e., morbidity, mortality, and readmission), while 6 reported long-term outcomes (i.e., cancer outcomes, weight loss, and urinary continence). Better intraoperative performance was associated with fewer postoperative complications (6 of 7 studies), reoperations (3 of 4 studies), and readmissions (1 of 4 studies). Long-term outcomes were less commonly investigated, with mixed results.

CONCLUSION

Current evidence supports an association between superior intraoperative technical performance measured using surgical videos and improved short-term postoperative outcomes. Intraoperative performance analysis using video-based assessment represents a promising approach to surgical quality-improvement.

A resident robotic curriculum utilizing self-selection and a web-based feedback tool

To describe an obstetrics and gynecology residency robotic curriculum, facilitated by a web-based feedback and case-tracking tool, allowing for self-selection into advanced training. Phase I (Basic) was required for all residents and included online training modules, online assessment, and robotic bedside assistant dry lab. Phase II (Advanced) was elective console training. Before live surgery, 10 simulation drills completed to proficiency were required. A web-based tool was used for surgical feedback and case-tracking. Online assessments, drill reports, objective GEARS assessments, subjective feedback, and case-logs were reviewed (7/2018-6/2019). A satisfaction survey was reviewed. Twenty four residents completed Phase I training and 10 completed Phase II. To reach simulation proficiency, residents spent a median of 4.1 h performing required simulation drills (median of 10 (3, 26) attempts per drill) before live surgery. 128 post-surgical feedback entries were completed after performance as bedside assistant (75%, n = 96) and console surgeon (5.5%, n = 7). The most common procedure was hysterectomy 111/193 (58%). Resident console surgeons performed portions of 32 cases with a mean console time of 34.6 ± 19.5 min. Mean GEARS score 20.6 ± 3.7 (n = 28). Mean non-technical feedback results: communication (4.2 ± 0.8, n = 61), workload management (3.9 ± 0.9, n = 54), team skills (4.3 ± 0.8, n = 60). Residents completing > 50% of case assessed as "apprentice" 38.5% or "competent" 23% (n = 13). After curriculum change, 100% of surveyed attendings considered residents prepared for live surgical training, vs 17% (n = 6) prior to curriculum change [survey response rate 27/44 (61%)]. Attendings and residents were satisfied with curriculum; 95% and recommended continued use 90% (n = 19).This two-phase robotic curriculum allows residents to self-select into advanced training, alleviating many challenges of graduated robotic training.

Anterior plating technique for distal radius: comparing performance after learning through naive versus deliberate practice

BACKGROUND

Surgical teaching is most often carried out in the operating theatre through mentorship, and the performance of surgical procedures is rarely measured. The objective of this article is to compare the progression in learning curves of junior surgeons trained in the anterior plating technique for the distal radius on a nonbiological model according to three different methods.

METHODS

The materials comprised 12 junior surgeons of level 1 or 2 (as per Tang and Giddins) divided into three groups: control (G1), naive practice (G2), and deliberate practice (G3). The three groups watched a demonstration video of a level 5 expert. The four G1 surgeons (two level 1 and two level 2) saw the video only once, and each inserted five plates. The four G2 surgeons (two level 1 and two level 2) inserted five plates and watched the video before each time. The four G3 surgeons (two level 1 and two level 2) saw the video before the first plate insertion. Before posing the subsequent four plates, the four G3 surgeons watched their own video, and the expert indicated their errors and how to avoid them next time. A 12-criteria OSATS defined on the basis of the 60 videos, each graded from one (min.) to five (max.), was used to measure the objective surgical performance per plating (min. 12; max. 60) and per series of five plate fixations (min. 60, max. 300).

RESULTS

The total average objective performance of G1 was 44.73, of G2 was 50.57 and of G3 was 54.35. Change in objective performance was better for G3 (13.25) than G2 (5) or G1 (3.75). For all groups, the progression in objective performance was better amongst level 1 surgeons (9) than level 2 surgeons (5.6).

CONCLUSION

Surgical teaching is based on mentorship and experience. However, since "see one, practice many, do one" has started to replace "see one, do one, teach one", learning techniques have increasingly relied on procedure simulators. Against this background, few studies have looked at measuring the performance of surgical procedures and improved learning curves. Our results appear to suggest that deliberate practice, when used in addition to mentorship, is the best option for shortening the growth phase of the learning curve and improving performance. Deliberate practice is a learning technique for surgical procedures that is complementary to mentorship and experience, which allows the growth phase of the learning curve to be shortened and the objective performance of junior surgeons to be improved.

Intraoperative dynamics of workflow disruptions and surgeons' technical performance failures: insights from a simulated operating room

INTRODUCTION

Flow disruptions (FD) in the operating room (OR) have been found to adversely affect the levels of stress and cognitive workload of the surgical team. It has been concluded that frequent disruptions also lead to impaired technical performance and subsequently pose a risk to patient safety. However, respective studies are scarce. We therefore aimed to determine if surgical performance failures increase after disruptive events during a complete surgical intervention.

METHODS

We set up a mixed-reality-based OR simulation study within a full-team scenario. Eleven orthopaedic surgeons performed a vertebroplasty procedure from incision to closure. Simulations were audio- and videotaped and key surgical instrument movements were automatically tracked to determine performance failures, i.e. injury of critical tissue. Flow disruptions were identified through retrospective video observation and evaluated according to duration, severity, source, and initiation. We applied a multilevel binary logistic regression model to determine the relationship between FDs and technical performance failures. For this purpose, we compared FDs in one-minute intervals before performance failures with intervals without subsequent performance failures.

RESULTS

Average simulation duration was 30:02 min (SD = 10:48 min). In 11 simulated cases, 114 flow disruption events were observed with a mean hourly rate of 20.4 (SD = 5.6) and substantial variation across FD sources. Overall, 53 performance failures were recorded. We observed no relationship between FDs and likelihood of immediate performance failures: Adjusted odds ratio = 1.03 (95% CI 0.46-2.30). Likewise, no evidence could be found for different source types of FDs.

CONCLUSION

Our study advances previous methodological approaches through the utilisation of a mixed-reality simulation environment, automated surgical performance assessments, and expert-rated observations of FD events. Our data do not support the common assumption that FDs adversely affect technical performance. Yet, future studies should focus on the determining factors, mechanisms, and dynamics underlying our findings.

Minimally Trained Analysts Can Perform Fast, Objective Assessment of Orthopedic Technical Skill from Fluoroscopic Images

Skill assessment in orthopedics has traditionally relied on subjective impressions from a supervising surgeon. The feedback derived from these tools may be limited by bias and other practical issues. Objective analysis of intraoperative fluoroscopic images offers an inexpensive, repeatable, and precise assessment strategy without bias. Assessors generally refrain from using the scores of images obtained throughout the operation to evaluate skill for practical reasons. A new system was designed to facilitate rapid analysis of this fluoroscopy via minimally trained analysts. Four expert and four novice analysts independently measured one objective metric for skill using both a custom analysis software and a commercial alternative. Analysts were able to measure the objective metric three times faster when using the custom software, and without a practical difference in accuracy in comparison to the expert analysts using the commercial software. These results suggest that a well-designed fluoroscopy analysis system can facilitate inexpensive, reliable, and objective assessment of surgical skills.

Video-Based Coaching: Current Status and Role in Surgical Practice (Part 1) From the Society for Surgery of the Alimentary Tract, Health Care Quality and Outcomes Committee

Patient safety and outcomes are directly related to surgical performance. Surgical training emphasizes the importance of the surgeon in determining these outcomes. After training is complete, there is a lack of structured programs for surgeons to audit their skills and continue their individual development. There is a significant linear relationship between surgeon technical skill and surgical outcomes; however, measuring technical performance is difficult. Video-based coaching matches an individual surgeon in practice with a surgical colleague who has been trained in the core principles of coaching for individualizing instruction. It can provide objective assessment for teaching higher-level concepts, such as technical skills, cognitive skills, and decision-making. There are many benefits to video-based coaching. While the concept is gaining acceptance as a method of surgical education, it is still novel in clinical practice. As more surgeons look towards video-based coaching for quality improvement, a consistent definition of the program, goals, and metrics for assessment will be critical. This paper is a review on the status of the video-based coaching as it applies to practicing surgeons.