A validation study on the use of intra-operative video recording as an objective assessment tool for core ENT surgery

Video-based assessment (VBA) of an open, simulated orthopedic surgical procedure: a pilot study using a single-angle camera to assess surgical skill and decision making

BACKGROUND

Videos have been used in many settings including medical simulation. Limited information currently exists on video-based assessment in surgical training. Effective assessment tools have substantial impact on the future of training. The objectives of this study were as follows: to evaluate the inter-rater reliability of video-based assessment of orthopedic surgery residents performing open cadaveric simulation procedures and to explore the benefits and limitations of video-based assessment.

METHODS

A multi-method technique was used. In the quantitative portion, four residents participated in a Surgical Objective Structured Clinical Examination in 2017 at a quaternary care training center. A single camera bird's-eye view was used to videotape the procedures. Five orthopedic surgeons evaluated the surgical videos using the Ottawa Surgical Competency Operating Room Evaluation. Interclass correlation coefficient was used to calculate inter-rater reliability. In the qualitative section, semi-structured interviews were used to explore the perceived strengths and limitations of video-based assessment.

RESULTS AND DISCUSSION

The scores using video-based assessment demonstrated good inter-rater reliability (ICC = 0.832, p = 0.014) in assessing open orthopedic procedures on cadavers. Qualitatively, the strengths of video-based assessment in this study are its ability to assess global performance and/or specific skills, ability to reassess missed points during live assessment, and potential use for less common procedures. It also allows for detailed constructive feedback, flexible assessment time, anonymous assessment, multiple assessors and serves as a good coaching tool. The main limitations of video-based assessment are poor audio-video quality, and questionable feasibility for assessing readiness for practice.

CONCLUSION

Video-based assessment is a potential adjunct to live assessment in orthopedic open procedures with good inter-rater reliability. Improving audio-video quality will enhance the quality of the assessment and improve the effectiveness of using this tool in surgical training.

Development and Validation of Instrument for Operative Competency Assessment in Selective Neck Dissection

BACKGROUND

Instruments to assess surgical skills have been validated for several key indicator procedures in otolaryngology. Selective neck dissection is a core procedure for which trainees must integrate knowledge of complex head and neck anatomy with technical surgical skills. An instrument for assessment of surgical performance in selective neck dissection has not been previously developed. The objective of the current study is to develop and validate an instrument for assessing surgical competency for level II-IV selective neck dissection.

DESIGN

A Delphi working group comprised of 23 fellowship trained head and neck surgeons from 17 institutions was assembled. The modified Delphi method encompassed a 3-step process, including 2 anonymous voting rounds to successively refine individual items and establish levels of consensus. Thresholds for achieving strong consensus, at >80% agreement, were determined a priori. The resulting instrument was subsequently validated in a prospective cohort of 17 resident surgeons, spanning postgraduate year 1 to 5 training experience. Participants were asked to perform a level II-IV selective neck dissection on fresh-frozen cadaveric specimens. Performance was scored by 2 independent, blinded observers using the devised instrument and construct validity was assessed.

RESULTS

Through the modified Delphi process a final list of 30 items, considered to be the most essential items for achieving the goals of a level II-IV selective neck dissection, was developed. Construct validity was supported by a positive association between instrument scores compared to both resident postgraduate year level and number of head and neck rotations completed.

CONCLUSION

The development and validation of a novel instrument for assessment of surgical competency in level II-IV selective neck dissection, a key indicator case in otolaryngology, is described. This new instrument may be used to provide objective feedback on overall and task-specific competency to identify surgical deficiencies and offer granular feedback to enhance surgical training.

The Role of Online Videos in Teaching Procedural Skills in Postgraduate Medical Education: A Scoping Review

OBJECTIVE

The purpose of this scoping review was to outline the extent of available literature including the prevalence of video quality appraisal tools, characterize how online videos were used, and identify the gaps in the literature with implications for future research.

DESIGN

The review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for scoping reviews (PRISMA-ScR). Six databases were searched: (1) MEDLINE (Ovid), (2) EMBASE, (3) Cochrane, (4) ERIC, (5) CINAHL PLUS, and (6) Google Scholar for Medical Subject Headings terms online videos, postgraduate health education, and health professional/s.

RESULTS

A total of 6948 articles were identified, of which 78 were included in the review. The articles included were primarily either experimental or observational studies, with most being from North America. Twelve concepts were identified: (1) procedural skills teaching; (2) video assessment; (3) validation of an assessment tool; (4) video feedback; (5) coaching; (6) broadcasting; (7) learner characteristics; (8) video characteristics; (9) video quality; (10) a quality assessment tool; (11) platforms and video library; and (12) health information governance.

CONCLUSIONS

While there is a diverse and growing body of literature on this area, information is lacking about the quality appraisal of online videos.

Current Evidence for Simulation-Based Training and Assessment of Myringotomy and Ventilation Tube Insertion: A Systematic Review

OBJECTIVE

Myringotomy and ventilation tube insertion (MT) is a key procedure in otorhinolaryngology and can be trained using simulation models. We aimed to systematically review the literature on models for simulation-based training and assessment of MT and supporting educational evidence.

DATABASES REVIEWED

PubMed, Embase, Cochrane Library, Web of Science, Directory of Open Access Journals.

METHODS

Inclusion criteria were MT training and/or skills assessment using all types of training modalities and learners. Studies were divided into 1) descriptive and 2) educational interventional/observational in the analysis. For descriptive studies, we provide an overview of available models including materials and cost. Educational studies were appraised using Kirkpatrick's level of educational outcomes, Messick's framework of validity, and a structured quality assessment tool.

RESULTS

Forty-six studies were included consisting of 21 descriptive studies and 25 educational studies. Thirty-one unique physical and three virtual reality simulation models were identified. The studies report moderate to high realism of the different simulators and trainees and educators perceive them beneficial in training MT skills. Overall, simulation-based training is found to reduce procedure time and errors, and increase performance as measured using different assessment tools. None of the studies used a contemporary validity framework and the current educational evidence is limited.

CONCLUSION

Numerous simulation models and assessment tools have been described in the literature but educational evidence and systematic implementation into training curricula is scarce. There is especially a need to establish the effect of simulation-based training of MT in transfer to the operating room and on patient outcomes.

A Proposed Model for a Comprehensive Virtual Sub-Internship in Vascular Surgery

OBJECTIVE

Understand challenges, advantages, and applications of a vascular surgery virtual sub-internship (VSI) curriculum.

METHODS

Our institution hosted 25 students for 4-week VSI rotations. We held two sessions, one in July 2020 and another in August 2020. The students participated in a curriculum centered around Zoom® and phone interactions with residents and faculty. The curriculum included selected readings, surgical videos, group didactics and 1-on-1 mentorship. Anonymous pre-rotation and post-rotation self-assessments were used to ascertain achievement of learning objectives and utility of educational tools implemented during the rotation. Faculty and resident mentors were also surveyed to assess their experience.

RESULTS

With the exception of knot tying techniques (p=0.67), students reported significant improvement in their understanding vascular surgery concepts after the virtual elective (p < 0.05). The highest ranked components of the course were interpersonal including interaction with faculty, mentorship, and learning the program culture. The lowest ranked components of the course were simulation training and research opportunities. Rating of utility for aspects of the course were consistent with ranking of components, with faculty interaction having the highest average rating. The ideal number of hours for daily virtual interaction reported by students ranged from 3 to 6 with a median of 4. Overall, the majority of mentors were satisfied with virtual course. However, they did report limited ability to assess student personality and fit for the program. The number of hours per week spent by mentors on the virtual vascular surgery rotation ranged from 2 to 7 with a median of 4. Fourteen out of seventeen mentors reported that having a virtual student was significant addition to their existing workload.

CONCLUSIONS

Overall, our student and mentor feedback were positive. Several challenges inherent to the virtual environment still need to be refined, however the goals of a VSI are distinct and should be explored by training programs. With changes to healthcare in the United States in the horizon and the arrival of SARS-CoV2 pandemic constraints, implementing a virtual away rotation may be an acceptable platform for adapting our recruitment strategies.

Is 'Omega Sign' anatomical endoscopic enucleation of the prostate technique reproducible?

We aimed to evaluate the learning curve of the surgically standardised 'Omega Sign' anatomical endoscopic enucleation (AEEP) of the prostate surgery technique for junior surgeons. This study is a retrospective comparison of cases that underwent AEEP by a mentor surgeon and three junior surgeons who have completed their learning curve. A video-based laser enucleation of the prostate assessment tool (LEAT) composed of 8 steps of the technique was used to assess a senior surgeon and junior surgeons' surgical compatibility and consistency. The surgeon who defined Omega Sign technique was determined as group 1, and cases by three junior surgeons were identified as group 2. The end points were to assess the reproducibility and repeatability and operative post-operative outcomes of the technique. 55 patients' videos were rated by five experienced endourologists. There was no significant difference in LEAT scores between the groups among all steps. The most symmetry was found in the 1st and 3rd steps. Inter-rater consistency was also high for each step, with no statistically significant difference between the evaluators. The standardised anatomical 'Omega Sign' technique is reproducible for the junior surgeons. The operative steps can be performed with high consistency, and the functional and perioperative outcomes are comparable with the senior surgeon.

Using Intraoperative Recordings to Evaluate Surgical Technique and Performance in Mastoidectomy

Importance

Otolaryngology residency programs currently lack rigorous methods for assessing surgical skill and often rely on biased tools of evaluation.

Objectives

To evaluate which techniques used in mastoidectomy can serve as indicators of surgeon level (defined as the level of training) and whether these determinations of technique can be made based solely on the movement of the drill head or suction.

Design, Setting, and Participants

In this prospective, observational study conducted from January 1, 2015, to December 31, 2019, at a single tertiary care institution, 3 independent observers made blinded evaluations on 24 intraoperative recordings of surgeons (6 junior residents, 4 senior residents, and 2 attending surgeons) performing mastoidectomies.

Main Outcomes and Measures

Observers assessed drill stroke count, drilling efficiency, stroke pattern, use of suction and irrigation, and estimated surgeon level. Assessments were made on both original videos and animated videos that show only the path of the burr head or suction as dots against a white background.

Results

Among the 24 recorded mastoidectomies performed by the 12 study surgeons, intraclass correlation was excellent for original video assessment of drill stroke count (0.98 [95% CI, 0.97-1.00]), use of suction (0.75 [95% CI, 0.52-0.89]), use of irrigation (0.83 [95% CI, 0.66-0.92]), and estimated surgeon level (0.82 [95% CI, 0.64-0.92]) and fair for drilling efficiency (0.54 [95% CI, 0.09-0.79]) and stroke pattern (0.49 [95% CI, -0.02 to 0.76]). Intraclass correlation was excellent for animated video assessment of drill stroke count per unit time (0.98 [95% CI, 0.96-0.99]) and drilling efficiency (0.80 [95% CI, 0.60-0.91]), good for stroke pattern (0.68 [95% CI, 0.38-0.85]) and estimated surgeon level (based on path of drill) (0.69 [95% CI, 0.38-0.85]), and fair for use of suction (0.58 [95% CI, 0.16-0.80]) and estimated surgeon level (based on path of suction) (0.58 [95% CI, 0.17-0.80]). On evaluation of original videos, junior residents had lower drill stroke count compared with senior residents and attending surgeons (6.0 [interquartile range (IQR), 3.0-8.0] vs 9.5 [IQR, 5.0-13.0] vs 10.5 [IQR, 5.0-17.8]; η2 = 0.14 [95% CI, 0.01-0.28]). On evaluation of animated videos, junior residents also had lower drill stroke count compared with senior residents and attending surgeons (6.0 [IQR, 4.0-9.0] vs 10.5 [IQR, 10.0-13.8] vs 10.5 [IQR, 4.3-21.0]; η2 = 0.19 [95% CI, 0.04-0.33]). Compared with junior and senior residents, attending surgeons had higher median ratings of drilling efficiency (original videos: junior residents, 4.0 [IQR, 3.0-4.0]; senior residents, 4.0 [IQR, 3.0-4.8]; attending surgeons, 5.0 [IQR, 4.3-5.0]; η2 = 0.23 [95% CI, 0.06-0.37]; animated videos: junior residents, 4.0 [IQR, 3.0-4.0]; senior residents, 3.0 [IQR, 2.0-4.0]; attending surgeons, 5.0 [IQR, 4.0-5.0]; η2 = 0.25 [95% CI, 0.08-0.39]) and stroke pattern (original videos: junior residents, 4.0 [IQR, 3.0-4.0]; senior residents, 4.0 [IQR, 3.0-4.8]; attending surgeons, 5.0 [IQR, 5.0-5.0]; η2 = 0.17 [95% CI, 0.03-0.31]; animated videos: junior residents, 4.0 [IQR, 3.0-4.0]; senior residents, 4.0 [IQR, 2.0-4.0]; attending surgeons, 5.0 [IQR, 5.0-5.0]; η2 = 0.15 [95% CI, 0.02-0.29]).

Conclusions and Relevance

This study suggests that observation of intraoperative mastoidectomy recordings is a feasible method of evaluating surgeon level. Reasonable indicators of surgeon level include the drill stroke count, drilling efficiency, stroke pattern, and use of the suction irrigator. Observing the path of the drill alone is sufficient to appreciate differences in drilling technique but not sufficient to accurately determine surgeon level. Intraoperative recordings can serve as a useful addition to resident education and evaluation.

Audio and panoramic video recording in the operating room: legal and ethical perspectives

INTRODUCTION

The idea of video recording (VR) in the operating room (OR) with panoramic cameras and microphones is a new concept that is changing the approach to medical activities in the OR. However, VR in the OR has brought up many concerns regarding patient privacy and has highlighted legal and ethical issues that were never previously exposed.

AIM

To review the literature concerning these aspects and provide a better ethical and legal understanding of the new challenges concerning VR in the OR.

CONCLUSIONS

There is a disparity between the two main legal models concerning VR in the OR, namely the European legal system (General Data Protection Regulation (GDPR)) and the American legal framework (Health Insurance Portability and Accountability Act (HIPAA)). This difference mainly deals with two distinct bioethical paradigms: GDPR places a strong emphasis on protecting patients' privacy to improve the public health system, whereas HIPAA indicates the need to generate protocols to safeguard the risks connected to medical activity and patient privacy. Following from this point, we may argue that, at the ethical and bioethical level, GDPR and HIPAA depend mainly on two different ethical models: a perspective based on moral acquaintances and weak proceduralism, respectively. It is worth noting the importance of developing additional guidelines concerning different world regions to avoid the ethical problems that may emerge when simply applying a foreign paradigm to a very different culture.

Objective Assessment of Technical Skills in Otorhinolaryngology-Head and Neck Surgery Residents: A Systematic Review

Objectives The primary goal is the indexation of validated methods used to assess surgical competency in otorhinolaryngology-head and neck surgery (ORL-HNS) residents. Secondary goals include assessment of the reliability and validity of these tools, as well as the documentation of specific procedures in ORL-HNS involved. Data Sources MEDBASE, OVID, Medline, CINAHL, and EBM, as well as the printed references, available through the Université de Montréal library. Review Methods The PRISMA method was used to review digital and printed databases. Publications were reviewed by 2 independent reviewers, and selected articles were fully analyzed to classify evaluation methods and categorize them by procedure and subspecialty of ORL-HNS involved. Reliability and validity were assessed and scored for each assessment tool. Results Through the review of 30 studies, 5 evaluation methods were described and validated to assess the surgical competency of ORL-HNS residents. The evaluation method most often described was the combined Global Rating Scale and Task-Specific Checklist tool. Reliability and validity for this tool were overall high; however, considerable data were unavailable. Eleven distinctive surgical procedures were studied, encompassing many subspecialties of ORL-HNS: facial plastics, general ear-nose-throat, laryngology, otology, pediatrics, and rhinology. Conclusions Although assessment tools have been developed for an array of surgical procedures, involving most ORL-HNS subspecialties, the use of combined checklists has been repeatedly validated in the literature and shown to be easily applicable in practice. It has been applied to many ORL-HNS procedures but not in oncologic surgery to date.

What is Known About the Attributes of a Successful Surgical Trainer? A Systematic Review

BACKGROUND

Surgical training has been subject to significant upheaval in recent years with an increasingly rigorous assessment regimen for trainees. The assessment of surgical trainers is less well evolved by comparison. Recent proposals from the Royal College of Surgeons of England recommend "professionalising the trainers." However, they do not suggest any accepted or validated methods of trainer assessment, nor do they indicate how these might be implemented and monitored in a real-world training program to determine their effect on trainee outcomes.

AIM

To determine what is known about the attributes of successful surgical trainers.

METHODS

We conducted a systematic review of the scientific literature using the preferred reporting items for systematic reviews and meta-analyses and Cochrane guidelines of the Medline database using specific search criteria. The qualitative analysis involved grouping trainer attributes together into "themes" within 4 "super-themes." Each theme needed to be mentioned by a minimum of 5 studies.

RESULTS

After review of the full study texts a total of 14 studies met the inclusion criteria. Thirteen studies involved the views of trainees, whereas only 1 study solely assessed the views of trainers. There was a wide variety of study designs and types of participants. The attribute themes are listed in brackets after each super-theme: "character" (approachability, patience, enthusiasm, encouraging/supportiveness), "procedural" (willingness to let trainee operate, balance between supervision and independence), "teamwork and communication" (sets educational aims and objectives, ability to use appropriate feedback, communication skills, and time availability to train) and "clinical" (capable, good relationships with patients, and the health care team).

CONCLUSIONS

This detailed review describes several perceived important themes for the positive attributes of surgical trainers. The identification of these key attributes is only of value if their presence is confirmed by effective and feasible evaluation, and if the possession of such attributes in trainers is proven to have a positive effect on training. In times of reduced training opportunities, exploring the topics raised by this review through future education research is warranted.

From stereoscopic recording to virtual reality headsets: Designing a new way to learn surgery

OBJECTIVE

To improve surgical practice, there are several different approaches to simulation. Due to wearable technologies, recording 3D movies is now easy. The development of a virtual reality headset allows imagining a different way of watching these videos: using dedicated software to increase interactivity in a 3D immersive experience. The objective was to record 3D movies via a main surgeon's perspective, to watch files using virtual reality headsets and to validate pedagogic interest.

MATERIAL AND METHODS

Surgical procedures were recorded using a system combining two side-by-side cameras placed on a helmet. We added two LEDs just below the cameras to enhance luminosity. Two files were obtained in mp4 format and edited using dedicated software to create 3D movies. Files obtained were then played using a virtual reality headset. Surgeons who tried the immersive experience completed a questionnaire to evaluate the interest of this procedure for surgical learning.

RESULTS

Twenty surgical procedures were recorded. The movies capture a scene which is extended 180° horizontally and 90° vertically. The immersive experience created by the device conveys a genuine feeling of being in the operating room and seeing the procedure first-hand through the eyes of the main surgeon. All surgeons indicated that they believe in pedagogical interest of this method.

CONCLUSIONS

We succeeded in recording the main surgeon's point of view in 3D and watch it on a virtual reality headset. This new approach enhances the understanding of surgery; most of the surgeons appreciated its pedagogic value. This method could be an effective learning tool in the future.

An effective method for videorecording the nasal-dorsal part of a rhinoplasty - a multiple case study

Rhinoplasty is a difficult procedure to record for several reasons which include the narrow, dark and deep surgical field cluttered with many instruments, and the heads and hands of the surgical team often blocking the view. This study discusses 13 rhinoplasty cases that were recorded using the authors' technique. A three-minute, unedited segment was extracted from the mid-portion of each surgical case, and that footage was evaluated by three specialist examiners. Results indicate that the authors' technique can offer a effective solution in the video recording of the nasal-dorsal part of a rhinoplasty.

The Surgeon's View: Comparison of Two Digital Video Recording Systems in Veterinary Surgery

Video recording and photography during surgical procedures are useful in veterinary medicine for several reasons, including legal, educational, and archival purposes. Many systems are available, such as hand cameras, light-mounted cameras, and head cameras. We chose a reasonably priced head camera that is among the smallest video cameras available. To best describe its possible uses and advantages, we recorded video and images of eight different surgical cases and procedures, both in hospital and field settings. All procedures were recorded both with a head-mounted camera and a commercial hand-held photo camera. Then sixteen volunteers (eight senior clinicians and eight final-year students) completed an evaluation questionnaire. Both cameras produced high-quality photographs and videos, but observers rated the head camera significantly better regarding point of view and their understanding of the surgical operation. The head camera was considered significantly more useful in teaching surgical procedures. Interestingly, senior clinicians tended to assign generally lower scores compared to students. The head camera we tested is an effective, easy-to-use tool for recording surgeries and various veterinary procedures in all situations, with no need for assistance from a dedicated operator. It can be a valuable aid for veterinarians working in all fields of the profession and a useful tool for veterinary surgical education.

Construct validity of the ovine model in endoscopic sinus surgery training

OBJECTIVES/HYPOTHESIS

To demonstrate construct validity of the ovine model as a tool for training in endoscopic sinus surgery (ESS).

STUDY DESIGN

Prospective, cross-sectional evaluation study.

METHODS

Over 18 consecutive months, trainees and experts were evaluated in their ability to perform a range of tasks (based on previous face validation and descriptive studies conducted by the same group) relating to ESS on the sheep-head model. Anonymized randomized video recordings of the above were assessed by two independent and blinded assessors. A validated assessment tool utilizing a five-point Likert scale was employed. Construct validity was calculated by comparing scores across training levels and experts using mean and interquartile range of global and task-specific scores. Subgroup analysis of the intermediate group ascertained previous experience. Nonparametric descriptive statistics were used, and analysis was carried out using SPSS version 21 (IBM, Armonk, NY).

RESULTS

Reliability of the assessment tool was confirmed. The model discriminated well between different levels of expertise in global and task-specific scores. A positive correlation was noted between year in training and both global and task-specific scores (P < .001). Experience of the intermediate group was variable, and the number of ESS procedures performed under supervision had the highest impact on performance.

CONCLUSIONS

This study describes an alternative model for ESS training and assessment. It is also the first to demonstrate construct validity of the sheep-head model for ESS training.