Convolutional Neural Network-Based Deep Learning Engine for Mastoidectomy Instrument Recognition and Movement Tracking

OBJECTIVE

To develop a convolutional neural network-based computer vision model to recognize and track 2 mastoidectomy surgical instruments-the drill and the suction-irrigator-from intraoperative video recordings of mastoidectomies.

STUDY DESIGN

Technological development and model validation.

SETTING

Academic center.

METHODS

Ten 1-minute videos of mastoidectomies done for cochlear implantation by varying levels of resident surgeons were collected. For each video, containing 900 frames, an open-access computer vision annotation tool was used to annotate the drill and suction-irrigator class images with bounding boxes. A mastoidectomy instrument tracking module, which extracts the center coordinates of bounding boxes, was developed using a feature pyramid network and layered with DETECTRON, an open-access faster-region-based convolutional neural network. Eight videos were used to train the model, and 2 videos were used for testing. Outcome measures included Intersection over Union (IoU) ratio, accuracy, and average precision.

RESULTS

For an IoU of 0.5, the mean average precision for the drill was 99% and 86% for the suction-irrigator. The model proved capable of generating maps of drill and suction-irrigator stroke direction and distance for the entirety of each video.

CONCLUSIONS

This computer vision model can identify and track the drill and suction-irrigator from videos of intraoperative mastoidectomies performed by residents with excellent precision. It can now be employed to retrospectively study objective mastoidectomy measures of expert and resident surgeons, such as drill and suction-irrigator stroke concentration, economy of motion, speed, and coordination, setting the stage for characterization of objective expectations for safe and efficient mastoidectomies.

Fully immersive virtual reality for skull-base surgery: surgical training and beyond

PURPOSE

A virtual reality (VR) system, where surgeons can practice procedures on virtual anatomies, is a scalable and cost-effective alternative to cadaveric training. The fully digitized virtual surgeries can also be used to assess the surgeon's skills using measurements that are otherwise hard to collect in reality. Thus, we present the Fully Immersive Virtual Reality System (FIVRS) for skull-base surgery, which combines surgical simulation software with a high-fidelity hardware setup.

METHODS

FIVRS allows surgeons to follow normal clinical workflows inside the VR environment. FIVRS uses advanced rendering designs and drilling algorithms for realistic bone ablation. A head-mounted display with ergonomics similar to that of surgical microscopes is used to improve immersiveness. Extensive multi-modal data are recorded for post-analysis, including eye gaze, motion, force, and video of the surgery. A user-friendly interface is also designed to ease the learning curve of using FIVRS.

RESULTS

We present results from a user study involving surgeons with various levels of expertise. The preliminary data recorded by FIVRS differentiate between participants with different levels of expertise, promising future research on automatic skill assessment. Furthermore, informal feedback from the study participants about the system's intuitiveness and immersiveness was positive.

CONCLUSION

We present FIVRS, a fully immersive VR system for skull-base surgery. FIVRS features a realistic software simulation coupled with modern hardware for improved realism. The system is completely open source and provides feature-rich data in an industry-standard format.

Content Validity of a High-Fidelity Surgical Middle Ear Simulator: A Randomized Prospective International Multicenter Trial

OBJECTIVE

After demonstration of face validity of a surgical middle ear simulator (SMS) previously, we assessed the content validity of the simulator with otolaryngology residents.

STUDY DESIGN

Multicenter randomized prospective international study.

SETTING

Four academic institutions.

METHODS

Novice participants were randomized into control, low-fidelity (LF), and high-fidelity (HF) groups. Control and LF produced 2 recordings from 2 attempts, and HF produced 4 recordings from 10 attempts, with trials 1, 4, 7, and 10 used for scoring. Three blinded experts graded videos of the simulated stapedectomy operation using an objective skills assessment test format consisting of global and stapedotomy-specific scales.

RESULTS

A total of 152 recordings from 61 participants were included. Baseline characteristics did not differ significantly between groups. Depending on the step of the operation, inter-rater reliability ranged from 24 to 90%. For LF and HF, years of training was significantly associated with improved scores in certain objective skills assessment test subparts. HF outperformed the control group on stapes and global scores ( p < 0.05). The HF group demonstrated improvement in global score over trials, but plateaued after four trials. Scores varied greatly for participants from different institutions in certain operative steps, such as transecting incudostapedial joints, likely due to differences in instrumentation and time elapsed since manufacture.

CONCLUSION

Practice with SMS led to better performance in both global and stapes-specific scores. Further studies are needed to examine construct validity and to create otology-appropriate grading systems. Variables like instrumentation and decline in flexibility of the simulator after 12 months greatly affect performance on the simulator.

Are Video Recordings Reliable for Assessing Surgical Performance? A Prospective Reliability Study Using Generalizability Theory

INTRODUCTION

Reliability is pivotal in surgical skills assessment. Video-based assessment can be used for objective assessment without physical presence of assessors. However, its reliability for surgical assessments remains largely unexplored. In this study, we evaluated the reliability of video-based versus physical assessments of novices' surgical performances on human cadavers and 3D-printed models-an emerging simulation modality.

METHODS

Eighteen otorhinolaryngology residents performed 2 to 3 mastoidectomies on a 3D-printed model and 1 procedure on a human cadaver. Performances were rated by 3 experts evaluating the final surgical result using a well-known assessment tool. Performances were rated both hands-on/physically and by video recordings. Interrater reliability and intrarater reliability were explored using κ statistics and the optimal number of raters and performances required in either assessment modality was determined using generalizability theory.

RESULTS

Interrater reliability was moderate with a mean κ score of 0.58 (range 0.53-0.62) for video-based assessment and 0.60 (range, 0.55-0.69) for physical assessment. Video-based and physical assessments were equally reliable (G coefficient 0.85 vs. 0.80 for 3D-printed models and 0.86 vs 0.87 for cadaver dissections). The interaction between rater and assessment modality contributed to 8.1% to 9.1% of the estimated variance. For the 3D-printed models, 2 raters evaluating 2 video-recorded performances or 3 raters physically assessing 2 performances yielded sufficient reliability for high-stakes assessment (G coefficient >0.8).

CONCLUSIONS

Video-based and physical assessments were equally reliable. Some raters were affected by changing from physical to video-based assessment; consequently, assessment should be either physical or video based, not a combination.

Supplementing Intraoperative Mastoidectomy Teaching With Video-Based Coaching

OBJECTIVES

Video-based coaching might complement general surgery education, but little is known of its applicability for otologic microsurgical teaching. Our purpose was thus to evaluate the content and resident-perceived benefit of video-based coaching for mastoidectomy education.

METHODS

In this mixed-methods pilot design, mastoidectomies were recorded from operative microscopes and reviewed during 30-minute video-based coaching sessions at 2 tertiary care centers. Eight residents and 3 attendings participated. Ten-point Likert-type questionnaires on the extent to which attendings taught 12 topics through 8 techniques were completed by residents after surgical and coaching sessions. Coaching sessions and structured interviews with residents were audio-recorded, transcribed and iteratively coded.

RESULTS

Seven audio-recordings were available for coaching sessions, during which a mean of 2.22 ± 0.5 topics per minute were discussed. Of the 12 teaching topics, technique was discussed most frequently (32%, 0.71 ± 0.2 topics/min), followed by anatomy (16%, 0.31 ± 0.16 topics/min). Of all 8 ratings between coaching and operative sessions, residents indicated a greater extent of discussion of anatomy (median difference, [95% confidence interval (CI)] of 3 [1-4]), progress (2.25 [95% CI, 0.5-4]), technique (3.5 [95% CI, 1.5-5.5]), pitfalls (2.5 [95% CI, 1-3.5]), and summarizing (3 [95% CI, 1-5]). In structured interviews, residents reported improved self-confidence and global perspective.

CONCLUSIONS

Video-based coaching is educationally dense and characterized by perceived richer teaching and promotion of a deeper surgical understanding. It requires no additional resources, can be completed in a short period of time and can be implemented programmatically for any otolaryngologic subspecialty utilizing video-recording capable equipment.

Development and Validation of Instrument for Operative Competency Assessment in Selective Neck Dissection

BACKGROUND

Instruments to assess surgical skills have been validated for several key indicator procedures in otolaryngology. Selective neck dissection is a core procedure for which trainees must integrate knowledge of complex head and neck anatomy with technical surgical skills. An instrument for assessment of surgical performance in selective neck dissection has not been previously developed. The objective of the current study is to develop and validate an instrument for assessing surgical competency for level II-IV selective neck dissection.

DESIGN

A Delphi working group comprised of 23 fellowship trained head and neck surgeons from 17 institutions was assembled. The modified Delphi method encompassed a 3-step process, including 2 anonymous voting rounds to successively refine individual items and establish levels of consensus. Thresholds for achieving strong consensus, at >80% agreement, were determined a priori. The resulting instrument was subsequently validated in a prospective cohort of 17 resident surgeons, spanning postgraduate year 1 to 5 training experience. Participants were asked to perform a level II-IV selective neck dissection on fresh-frozen cadaveric specimens. Performance was scored by 2 independent, blinded observers using the devised instrument and construct validity was assessed.

RESULTS

Through the modified Delphi process a final list of 30 items, considered to be the most essential items for achieving the goals of a level II-IV selective neck dissection, was developed. Construct validity was supported by a positive association between instrument scores compared to both resident postgraduate year level and number of head and neck rotations completed.

CONCLUSION

The development and validation of a novel instrument for assessment of surgical competency in level II-IV selective neck dissection, a key indicator case in otolaryngology, is described. This new instrument may be used to provide objective feedback on overall and task-specific competency to identify surgical deficiencies and offer granular feedback to enhance surgical training.

Video Analysis of Otologic Instrument Movement During Resident Mastoidectomies

OBJECTIVE

To measure surgical instrument movement during resident mastoidectomies and identify metrics that correlate with experience.

STUDY DESIGN

Retrospective case series.

SETTING

Tertiary care center.

SUBJECTS

Ten postgraduate year (PGY) 2, 6 PGY3, 7 PGY4, and 19 PGY5 recordings of mastoidectomy performed by otolaryngology residents.

INTERVENTIONS

One-minute intraoperative recordings of mastoidectomies performed during cochlear implantation were collected. Drill and suction-irrigator motion were analyzed with sports motion tracking software.

MAIN OUTCOME MEASURES

Mean instrument speed, angle, and angular velocity were calculated. Mann-Whitney U tests compared mean instrument metrics between PGY levels. Change in drill speed for seven residents between their PGY2 to PGY5 years was individually analyzed.

RESULTS

Mean drill speed was significantly greater for PGY5 residents compared with PGY2s (2.9 versus 1.8 cm/s, p = 0.001). Compared with PGY2 residents, suction speed was greater as a PGY5 (1.2 versus 0.9 cm/s; p = 0.201) and significantly greater as a PGY4 (1.5 versus 0.9 cm/s, p = 0.039). Of the seven residents individually analyzed, group mean drill speed increased by 0.4 cm/s, yearly.

CONCLUSIONS

Drill and suction-irrigator movement during the second minute of drilling of a cortical mastoidectomy seems to increase with resident level. Objective video analysis is a potential adjunct for differentiating novices from more experienced surgeons and monitoring surgical skills progress.

Comparison of Summative Temporal Bone Dissection Scales Demonstrate Equivalence

Introduction  Temporal bone surgery is a unique and complicated surgical skill that requires extensive training. There is an educational requirement to maximize trainee experience and provide effective feedback.

Objective  We evaluate three temporal bone dissection scales for efficacy, reliability, and accuracy in identifying resident skill during temporal bone surgery.

Methods  Residents of various skill levels performed a mastoidectomy with posterior tympanotomy on identic 3D-printed temporal bone models. Four blinded otologic surgeons evaluated each specimen at two separate intervals using three separate dissection scales: the Welling Scale (WS), the Iowa Temporal Bone Assessment Tool (ITBAT), and the CanadaWest Scale (CWS). Scores from each scale were compared in their ability to accurately separate residents by skill level, inter- and intrarater reliability, and efficiency in application.

Results  Nineteen residents from 9 postgraduate programs participated. Assessment was clustered into junior (postgraduate year or PGY 1, 2), intermediate (PGY 3) and senior resident (PGY 4, 5) cohorts. Analysis of variance (ANOVA) found significant differences between cohort performance ( p  < 0.05) for all 3 scales considering the PGY level and the subjective account of temporal bone surgical experience. The inter-rater reliability was consistent across each scale. The intrarater reliability was comparable between the CWS (0.711) and the WS (0.713), but not the ITBAT (0.289). Time (in seconds) to complete scoring for each scale was also comparable between the CWS (42.7 ± 16.8), the WS (76.6 ± 14.5), and the ITBAT (105.6 ± 38.9).

Conclusion  All three scales demonstrated construct validity and consistency in performance, and consideration should be given to judicious use in training.

[Multimodal training concept for temporal bone surgery]

BACKGROUND

Surgical training is increasingly supported by the use of simulators. For temporal bone surgery, shown here by means of mastoidectomy, there are other training models besides cadaver specimens, such as artificial temporal bones or computer-based simulators.

OBJECTIVES

A structured training concept was created which integrates different training methods of mastoidectomy with regard to effectiveness and current learning theory in education.

METHOD

A selective literature research was conducted to compare learning-theoretical findings and the availability and effectiveness of currently existing training models.

RESULTS

To acquire surgical skills, a stepwise approach is suggested. Depending on the progress with computer-based simulation, plastic or native temporal bones should be used. To achieve a plateau of the learning curve, approximately 25 semi-autonomous preparations are recommended. Different 'Objective Structured Assessments of Technical Skills' (OSATS) are implemented to assess the learning progress at different levels.

DISCUSSION

Simulation-based training is recommended until an adequate learning curve plateau is achieved. This is reasonable for patient safety, based on limited accessibility of human cadaveric temporal bones but also by findings of the learning theory.

CONCLUSION

The curriculum integrates different training models of mastoidectomy and OSATS into an overall concept. The training plan has to be continuously adapted to new findings and technical developments.

Automated assessment of cortical mastoidectomy performance in virtual reality

INTRODUCTION

Cortical mastoidectomy is a core skill that Otolaryngology trainees must gain competency in. Automated competency assessments have the potential to reduce assessment subjectivity and bias, as well as reducing the workload for surgical trainers.

OBJECTIVES

This study aimed to develop and validate an automated competency assessment system for cortical mastoidectomy.

PARTICIPANTS

Data from 60 participants (Group 1) were used to develop and validate an automated competency assessment system for cortical mastoidectomy. Data from 14 other participants (Group 2) were used to test the generalisability of the automated assessment.

DESIGN

Participants drilled cortical mastoidectomies on a virtual reality temporal bone simulator. Procedures were graded by a blinded expert using the previously validated Melbourne Mastoidectomy Scale: a different expert assessed procedures by Groups 1 and 2. Using data from Group 1, simulator metrics were developed to map directly to the individual items of this scale. Metric value thresholds were calculated by comparing automated simulator metric values to expert scores. Binary scores per item were allocated using these thresholds. Validation was performed using random sub-sampling. The generalisability of the method was investigated by performing the automated assessment on mastoidectomies performed by Group 2, and correlating these with scores of a second blinded expert.

RESULTS

The automated binary score compared with the expert score per item had an accuracy, sensitivity and specificity of 0.9450, 0.9547 and 0.9343, respectively, for Group 1; and 0.8614, 0.8579 and 0.8654, respectively, for Group 2. There was a strong correlation between the total scores per participant assigned by the expert and calculated by the automatic assessment method for both Group 1 (r = .9144, P < .0001) and Group 2 (r = .7224, P < .0001).

CONCLUSION

This study outlines a virtual reality-based method of automated assessment of competency in cortical mastoidectomy, which proved comparable to the assessment provided by human experts.

A time-sensitive rubric for assessing mastoidectomy proficiency

OBJECTIVE

To develop a time-sensitive, standardized rubric for cadaveric temporal bone dissection for otolaryngology resident education.

METHODS

This is a five-year prospective cohort study that evaluated otolaryngology resident performance during sequential cadaveric temporal bone dissection courses at a single otolaryngology residency training program. A canal-wall-up mastoidectomy with a facial recess approach was performed adhering to a 30-minute time limit and graded according to a standardized rubric. Main outcome measures included: (1) correct structure identification and (2) injuries sustained to structures as compared by resident post-graduate year (PGY) level.

RESULTS

Thirteen residents were evaluated from October 2012 to March 2017. This included 57 individual graded exercises performed over ten dissection courses. The average score for PGY-2 residents was lowest (68.9), and PGY-5 residents achieved the highest average score (87.7). Junior residents correctly identified fewer structures (77.5%) when compared to senior residents (91.3%), p < 0.0001. Correct performance of a facial recess approach was achieved by 100% of senior residents, but only 59.3% of junior residents (p = 0.0003). The percentage of major injuries, which included the facial nerve, tegmen, labyrinth, and ossicular chain, decreased each PGY-level from a maximum of 17% by PGY-2 residents to a minimum of 5% by PGY-5 residents.

CONCLUSION

Senior residents correctly identify more structures and are able to complete a facial recess approach with higher fidelity when subjected to a time-sensitive graded mastoidectomy rubric.

Assessment of operative competency for thyroidectomy: Comparison of resident self-assessment vs attending surgeon assessment

BACKGROUND

Self-awareness of skill, essential for progression as a surgeon, has direct bearing on postresidency practice; however, studies have supported that residents achieve self-perceived competence later than believed by program directors. This study compares residents' self-perception of operative competency to attending surgeon's evaluation using Computer Enhanced Visual Learning, a validated online hemithyroidectomy-specific assessment tool.

METHODS

Eleven otolaryngology-Head and Neck Surgery (HNS) residents completed a preoperative module and postoperative survey, later reviewed by an attending surgeon. Eighty-three performances were assessed for inter-rater reliabilities of key surgical steps.

RESULTS

Almost perfect agreement (Kappa = 0.81-1.00) was shown in 11 of 18 parameters. Substantial agreement (Kappa = 0.61-0.80) was demonstrated in the remaining seven parameters.

CONCLUSIONS

Otolaryngology-HNS trainees have high self-awareness of their performance at each step in a hemithyroidectomy. Standardized assessment tools can allow for documentation of procedural performance and serve as guides for improvement. This is the only study to examine otolaryngology-HNS trainees' self-perceived skill compared to an attending surgeon's assessment for hemithyroidectomy.

Subjective vs Computerized Assessment of Surgeon Skill Level During Mastoidectomy

This pilot study examines the use of surgical instrument tracking and motion analysis in objectively measuring surgical performance. Accuracy of objective measures in distinguishing between surgeons of different levels was compared to that of subjective assessments. Twenty-four intraoperative video clips of mastoidectomies performed by junior residents (n = 12), senior residents (n = 8), and faculty (n = 4) were sent to otolaryngology programs via survey, yielding 708 subjective ratings of surgical experience level. Tracking software captured the total distance traveled by the drill, suction irrigator, and patient's head. Measurements were used to predict surgeon level of training, and accuracy was estimated via area under the curve (AUC) of receiver operating characteristic curves. Key objective metrics proved more accurate than subjective evaluations in determining both faculty vs resident level and senior vs junior resident level. The findings of this study suggest that objective analysis using computer software has the potential to improve the accuracy of surgical skill assessment.

Validation of an Objective Assessment Tool for Tonsillectomy in Otolaryngology Resident Training

OBJECTIVE

Create and validate an objective structured assessment of technical skills (OSATS) for otolaryngology residents learning how to perform a tonsillectomy.

STUDY DESIGN

Multicenter prospective longitudinal validation study.

METHODS

A multi-institutional study at six tertiary academic otolaryngology residency programs from July 2009 to May 2012. Using the modified Delphi technique, a panel of pediatric otolaryngologists created a tonsillectomy task-based checklist (TBC) for a tonsil OSATS using a 5-point Likert-type scale. Residents were assessed by pediatric otolaryngology staff at the time of surgery with the TBC and a global rating scale. Procedure time, patient age, number of previously performed tonsillectomies, and surgical technique were also collected.

RESULTS

One hundred sixty-seven tonsil OSATS were completed for 38 residents, and competency was recorded for 99 (59.2%). Residents scored as competent had performed significantly more previous tonsillectomies than those deemed noncompetent, 44.4 ± 35.6 and 13.5 ± 11.6, respectively (P < .001). The mean overall score on the tonsil TBC was 4.0 ± 0.8 and 2.6 ± 1.0 for competent and noncompetent, respectively (P < .001). Higher number of tonsillectomies performed and mean tonsil TBC score significantly increased the likelihood of competency (P < .001). Each additional tonsillectomy performed increased the likelihood of achieving competency by 6.3% (P = .006, 95% confidence interval (CI): 1.330-1.110), and each 1.0 point increase in mean tonsil TBC score increased the likelihood of competency by a factor of 2.71 (P = .006, 95% CI:1.330-5.513). There is a 95% likelihood of competency at 48 tonsillectomies or a tonsil TBC score of 4.91.

CONCLUSION

The tonsil OSATS is a valid and feasible instrument to assess resident competency with tonsillectomy and provides timely objective feedback.

LEVEL OF EVIDENCE

4. Laryngoscope, 131:E359-E366, 2021.

Construct Validation of a Printed Bone Substitute in Otologic Education

: Patient safety demands enhancements in training. Graduated cadaveric bone exposure is fundamental to otologic training. Printed bone models (PBM) provide a low-cost, anatomically consistent adjunct to cadaveric materials in trainee skill acquisition.The purpose of this study is to determine if resident training level can be distinguished on the basis of performance employing a printed temporal bone model, graded by a previous validated scale.

METHODS

Nineteen residents (11 male, 8 female) from 9 graduate programs, attending a National Otolaryngology Conference, completed a mastoidectomy with posterior tympanotomy on identic 3D PBMs and a Likert scale (1-7) survey on subjective appreciation of the simulation. Four experts graded participant performance using the previously validated Welling Scale.

RESULTS

ANOVA revealed significant performance differences between the junior/intermediate and junior/senior PGY cohorts. No difference was observed between intermediate/senior cohorts on the basis of PGY or subjective temporal bone dissection experience. Clustering aspects of the scale with specific focus on thinning tasks found a similar outcome to the composite scale scores.Subjective experience judged printed bone to be similar to cadaveric in drill-bone interaction. Participants believed the simulation would improve surgical performance, comfort with actual patients, and operative speed.

CONCLUSION

Subjectively, printed bone compared favorably to cadaveric.The simulation demonstrated construct validity but was challenged in differentiating senior from intermediate trainee performance. This may be a function of the PBM inherent character, limitations in grading instrument fidelity or sample size. It is also possible that the dominant period of skill acquisition for mastoidectomy and posterior tympanotomy are primarily acquired during the junior training.

Reliability of Observational Assessment Methods for Outcome-based Assessment of Surgical Skill: Systematic Review and Meta-analyses

BACKGROUND

Reliable performance assessment is a necessary prerequisite for outcome-based assessment of surgical technical skill. Numerous observational instruments for technical skill assessment have been developed in recent years. However, methodological shortcomings of reported studies might negatively impinge on the interpretation of inter-rater reliability.

OBJECTIVE

To synthesize the evidence about the inter-rater reliability of observational instruments for technical skill assessment for high-stakes decisions.

DESIGN

A systematic review and meta-analysis were performed. We searched Scopus (including MEDLINE) and Pubmed, and key publications through December, 2016. This included original studies that evaluated reliability of instruments for the observational assessment of technical skills. Two reviewers independently extracted information on the primary outcome (the reliability statistic), secondary outcomes, and general information. We calculated pooled estimates using multilevel random effects meta-analyses where appropriate.

RESULTS

A total of 247 documents met our inclusion criteria and provided 491 inter-rater reliability estimates. Inappropriate inter-rater reliability indices were reported for 40% of the checklists estimates, 50% of the rating scales estimates and 41% of the other types of assessment instruments estimates. Only 14 documents provided sufficient information to be included in the meta-analyses. The pooled Cohen's kappa was .78 (95% CI 0.69-0.89, p < 0.001) and pooled proportion agreement was 0.84 (95% CI 0.71-0.96, p < 0.001). A moderator analysis was performed to explore the influence of type of assessment instrument as a possible source of heterogeneity.

CONCLUSIONS AND RELEVANCE

For high-stakes decisions, there was often insufficient information available on which to base conclusions. The use of suboptimal statistical methods and incomplete reporting of reliability estimates does not support the use of observational assessment instruments for technical skill for high-stakes decisions. Interpretations of inter-rater reliability should consider the reliability index and assessment instrument used. Reporting of inter-rater reliability needs to be improved by detailed descriptions of the assessment process.

Transfer of Automated Performance Feedback Models to Different Specimens in Virtual Reality Temporal Bone Surgery

Virtual reality has gained popularity as an effective training platform in many fields including surgery. However, it has been shown that the availability of a simulator alone is not sufficient to promote practice. Therefore, simulator-based surgical curricula need to be developed and integrated into existing surgical training programs. As practice variation is an important aspect of a surgical curriculum, surgical simulators should support practice on multiple specimens. Furthermore, to ensure that surgical skills are acquired, and to support self-guided learning, automated feedback on performance needs to be provided during practice. Automated feedback is typically provided by comparing real-time performance with expert models generated from pre-collected data. Since collecting data on multiple specimens for the purpose of developing feedback models is costly and time-consuming, methods of transferring feedback from one specimen to another should be investigated. In this paper, we discuss a simple method of feedback transfer between specimens in virtual reality temporal bone surgery and validate the accuracy and effectiveness of the transfer through a user study.

Standard Setting of Competency in Mastoidectomy for the Cross-Institutional Mastoidectomy Assessment Tool

OBJECTIVE

Competency-based surgical training involves progressive autonomy given to the trainee. This requires systematic and evidence-based assessment with well-defined standards of proficiency. The objective of this study is to develop standards for the cross-institutional mastoidectomy assessment tool to inform decisions regarding whether a resident demonstrates sufficient skill to perform a mastoidectomy with or without supervision.

METHODS

A panel of fellowship-trained content experts in mastoidectomy was surveyed in relation to the 16 items of the assessment tool to determine the skills needed for supervised and unsupervised surgery. We examined the consensus score to investigate the degree of agreement among respondents for each survey item as well as additional analyses to determine whether the reported skill level required for each survey item was significantly different for the supervised versus unsupervised level.

RESULTS

Ten panelists representing different US training programs responded. There was considerable consensus on cut-off scores for each item and trainee level between panelists, with moderate (0.62) to very high (0.95) consensus scores depending on assessment item. Further analyses demonstrated that the difference between supervised and unsupervised skill levels was significantly meaningful for all items. Finally, minimum-passing scores for each item was established.

CONCLUSION

We defined performance standards for the cross-institutional mastoidectomy assessment tool using the Angoff method. These cut-off scores that can be used to determine when trainees can progress from performance under supervision to performance without supervision. This can be used to guide training in a competency-based training curriculum.

A review of simulation applications in temporal bone surgery

Background

Temporal bone surgery is a technically challenging and high-risk procedure in an anatomically complex area. Safe temporal bone surgery emphasizes a consummate anatomic understanding and technique development that requires the guidance of an experienced otologic surgeon and years of practice. Temporal bone simulation can augment otologic surgical training and enable rehearsal of surgical procedures.

Objectives

The purpose of this article is to provide an updated review of temporal bone simulation platforms and their uses.

Data Sources

PubMed literature search. Search terms included temporal bone, temporal bone simulation, virtual reality (VR), and presurgical planning and rehearsal.

Discussion

Various simulation platforms such as cadaveric bone, three-dimensional (3D) printed models, and VR simulation have been used for temporal bone surgery training. However, each simulation method has its drawbacks. There is a need to improve upon current simulation platforms to enhance surgical training and skills assessment, as well as a need to explore other clinically significant applications of simulation, such as preoperative planning and rehearsal, in otologic surgery.

Conclusions

There is no replacement for actual surgical experience, but high-fidelity temporal bone models such as those produced with 3D printing and computer simulation have emerged as promising tools in otolaryngologic surgery. Improvements in the fidelity of both 3D printed and VR simulators as well as integration of a standardized assessment format would allow for an expansion in the use of these simulation platforms in training and assessment.

Level of Evidence

5.

Assessment of Surgical Skills and Competency

Evaluation of surgical skills and competency are important aspects of the medical education process. Measurable and reproducible methods of assessment with objective feedback are essential components of surgical training. Objective Structured Assessment of Technical Skills (OSATS) is widely used across the medical specialties and otolaryngology-specific tools have been developed and validated for sinus and mastoid surgery. Although assessment of surgical skills can be time-consuming and requires human and financial resources, new evaluation methods and emerging technology may alleviate these barriers while also improving data collection practices.

Cross-Institutional Evaluation of a Mastoidectomy Assessment Instrument

OBJECTIVE

The objective of this work is to obtain validity evidence for an evaluation instrument used to assess the performance level of a mastoidectomy. The instrument has been previously described and had been formulated by a multi-institutional consortium.

DESIGN

Mastoidectomies were performed on a virtual temporal bone system and then rated by experts using a previously described 15 element task-based checklist. Based on the results, a second, similar checklist was created and a second round of rating was performed.

SETTING

Twelve otolaryngological surgical training programs in the United States.

PARTICIPANTS

In all, 65 mastoidectomy performances were evaluated coming from 37 individuals with a variety of temporal bone dissection experience, from medical students to attending physicians. Raters were attending surgeons from 12 different institutions.

RESULTS

Intraclass correlation scores varied greatly between items in the checklist with some being low and some being high. Percentage agreement scores were similar to previous rating instruments. There is strong evidence that a high score on the task-based checklist is necessary for a rater to consider a mastoidectomy to be performed at the level of an expert but a high score is not a sufficient condition.

CONCLUSIONS

Rewording of the instrument items to focus on safety does not result in increased reliability of the instrument. The strong result of the Necessary Condition Analysis suggests that going beyond simple correlation measures can give extra insight into grading results. Additionally, we suggest using a multiple point scale instead of a binary pass/fail question combined with descriptive mastery levels.

Randomized Controlled Pilot Study of Video Self-assessment for Resident Mastoidectomy Training

A prospective randomized controlled pilot study was performed to determine if video self-assessment improves competency in mastoidectomy and to assess interrater agreement between expert and resident evaluations of recorded mastoidectomy. Sixteen otolaryngology residents were recorded while performing cadaveric mastoidectomy and randomized into video self-assessment and control groups. All residents performed a second recorded mastoidectomy. Performance was evaluated by blinded experts with a validated assessment scale. Video self-assessment did not lead to greater skill improvement between the first and second mastoidectomy. Interrater agreement was fair to substantial between the expert evaluators and between resident self-evaluations by recall and video review. Agreement between experts and residents was only slight to fair; residents consistently rated their performance higher than experts (P < .05). In conclusion, 1 session of video self-review did not lead to improved competence in mastoidectomy over standard practice. While experts agree on assessments, residents may overestimate their competency in performing cadaveric mastoidectomy.

Multi-Institutional Development of a Mastoidectomy Performance Evaluation Instrument

OBJECTIVE

A method for rating surgical performance of a mastoidectomy procedure that is shown to apply universally across teaching institutions has not yet been devised. This work describes the development of a rating instrument created from a multi-institutional consortium.

DESIGN

Using a participatory design and a modified Delphi approach, a multi-institutional group of expert otologists constructed a 15-element task-based checklist for evaluating mastoidectomy performance. This instrument was further refined into a 14-element checklist focusing on the concept of safety after using it to rate a large and varied population of performances.

SETTING

Twelve otolaryngological surgical training programs in the United States.

PARTICIPANTS

A total of 14 surgeons from 12 different institutions took part in the construction of the instrument.

RESULTS

By using 14 experts from 12 different institutions and a literature review, individual metrics were identified, rated as to the level of importance and operationally defined to create a rating scale for mastoidectomy performance. Initial use of the rating scale showed modest rater agreement. The operational definitions of individual metrics were modified to emphasize "safe" as opposed to "proper" technique. A second rating instrument was developed based on this feedback.

CONCLUSIONS

Using a consensus-building approach with multiple rounds of communication between experts is a feasible way to construct a rating instrument for mastoidectomy. Expert opinion alone using a Delphi method provides face and content validity evidence, however, this is not sufficient to develop a universally acceptable rating instrument. A continued process of development and experimentation to demonstrate evidence for reliability and validity making use of a large population of raters and performances is necessary to achieve universal acceptance.

Objective Assessment of Technical Skills in Otorhinolaryngology-Head and Neck Surgery Residents: A Systematic Review

Objectives The primary goal is the indexation of validated methods used to assess surgical competency in otorhinolaryngology-head and neck surgery (ORL-HNS) residents. Secondary goals include assessment of the reliability and validity of these tools, as well as the documentation of specific procedures in ORL-HNS involved. Data Sources MEDBASE, OVID, Medline, CINAHL, and EBM, as well as the printed references, available through the Université de Montréal library. Review Methods The PRISMA method was used to review digital and printed databases. Publications were reviewed by 2 independent reviewers, and selected articles were fully analyzed to classify evaluation methods and categorize them by procedure and subspecialty of ORL-HNS involved. Reliability and validity were assessed and scored for each assessment tool. Results Through the review of 30 studies, 5 evaluation methods were described and validated to assess the surgical competency of ORL-HNS residents. The evaluation method most often described was the combined Global Rating Scale and Task-Specific Checklist tool. Reliability and validity for this tool were overall high; however, considerable data were unavailable. Eleven distinctive surgical procedures were studied, encompassing many subspecialties of ORL-HNS: facial plastics, general ear-nose-throat, laryngology, otology, pediatrics, and rhinology. Conclusions Although assessment tools have been developed for an array of surgical procedures, involving most ORL-HNS subspecialties, the use of combined checklists has been repeatedly validated in the literature and shown to be easily applicable in practice. It has been applied to many ORL-HNS procedures but not in oncologic surgery to date.

Development and Validation of the Cochlear Implant Surgical Competency Assessment Instrument

OBJECTIVE

We present a new instrument for evaluation of cochlear implant (CI) surgical skills and review its validation process.

METHODS

An instrument to assess CI surgical competency incorporated results of structured surveys of comprehensiveness sent to 30 international CI experts and US trainees. One-hundred evaluations of 28 residents, fellows, and practicing CI surgeons were completed. Surgical skills were evaluated by four experienced neurotologists (two raters per subject) using two temporal bones per subject. A training session was completed by 24 subjects between the first and second procedure. Comparison of two blinded rater's scores per subject provided information on interrater reliability. Correlation of competency scores with degree of training and with improvement after a training session provided information on construct validity.

RESULTS

High levels of interrater reliability were confirmed by using the intraclass correlation coefficient. Construct validity was demonstrated by correlation of higher performance scores with increasing years of training, board certification, and fellowship training. Construct validity is also supported by improvement in scores after a CI training session as well as by acceptability surveys.

DISCUSSION

Data indicate that this instrument is an objective, accurate, and dependable procedure-specific instrument for evaluating CI surgical competency.

CONCLUSION

The cochlear implant surgical competency assessment (CI-SCA) can be used to establish CI surgical competency, identify surgical skills that require remediation and demonstrate progress during training.

Development and Pilot Testing of an Objective Structured Clinical Examination for Obstructive Sleep Apnea

Objective

Our aim was (1) to develop an objective structured clinical examination (OSCE) for obstructive sleep apnea (OSA) and (2) to test the reliability and evaluate the feasibility of this OSCE while assessing residents’ clinical skills in multiple core competencies via the standardized patient methodology.

Study Design

Development of assessment tool.

Setting

Johns Hopkins Medicine Simulation Center.

Subjects and Methods

Residents of the Department of Otolaryngology–Head and Neck Surgery at The Johns Hopkins University School of Medicine were invited to participate. A 2-station OSCE was developed. The first station used a standardized patient, and the encounter was videotaped for later evaluation by medical faculty not familiar with the participants being tested. The second was a computer-based station developed per a modified Delphi technique, based on feedback from otolaryngology and sleep medicine faculty involved in the care of patients with OSA. Checklists were developed for each station to standardize the evaluation of each trainee’s performance.

Results

The OSCE that we developed is a feasible tool for assessing residents’ performance and skills to diagnose and manage a patient with OSA. Internal consistency, as assessed by Krippendorff’s alpha, was 0.699 for station 1 and 0.95 for station 2.

Conclusion

This OSCE was found to be feasible for assessment of clinical competency in OSA. Our model provides targeted assessment of multiple competencies and opportunity to improve clinical knowledge and skills.

Educational Cadaveric Module for Teaching Percutaneous and Intranasal Osteotomies in Rhinoplasty

Lateral osteotomies are essential to rhinoplasty and are performed through percutaneous or intranasal approaches. Both techniques are difficult to teach as they rely on tactile feedback. Thus, it is critical to understand trainee learning curves to minimize complications. Herein, we aim to (1) demonstrate an educational module for teaching lateral osteotomies and (2) examine potential differences in outcomes between the 2 surgical approaches when performed by trainees. After a hands-on cadaveric laboratory, trainees (n = 24) reported increased confidence in performing both types of osteotomies ( P < .0001). Completion of the bony cut was similar between intranasal and percutaneous osteotomies (96% vs 75%, P = .097), as was correct placement of the osteotomy (75% vs 67%, P = .53). Intranasal osteotomies were more likely to cause periosteal disruption ( P = .02). This pilot study demonstrates that cadaveric laboratories are an effective way to teach lateral osteotomies and that percutaneous osteotomies may be less likely to cause periosteal disruption in trainees' hands.

Patient Safety and Quality Improvement in Otolaryngology Education: A Systematic Review

Objective

The breadth and depth of patient safety/quality improvement (PS/QI) research dedicated to otolaryngology–head and neck surgery (OHNS) education remains unknown. This systematic review aims to define this scope and to identify knowledge gaps as well as potential areas of future study to improved PS/QI education and training in OHNS.

Data Sources

A computerized Ovid/Medline database search was conducted (January 1, 1965, to May 15, 2015). Similar computerized searches were conducted using Cochrane Database, PubMed, and Google Scholar.

Review Methods

The study protocol was developed a priori using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). Articles were classified by year, subspecialty, Institute of Medicine (IOM) Crossing the Chasm categories, and World Health Organization (WHO) subclass.

Results

Computerized searches yielded 8743 eligible articles, 267 (3.4%) of which met otolaryngology PS/QI inclusion criteria; 51 (19%) were dedicated to resident/fellow education and training. Simulation studies (39%) and performance/competency evaluation (23.5%) were the most common focus. Most projects involved general otolaryngology (47%), rhinology (18%), and otology (16%). Classification by the IOM included effective care (45%), safety/effective care (41%), and effective and efficient care (7.8%). Most research fell into the WHO category of “identifying solutions” (61%).

Conclusion

Nineteen percent of OHNS PS/QI articles are dedicated to education, the majority of which are simulation and focus on effective care. Knowledges gaps for future research include facial plastics PS/QI and the WHO category of “studies translating evidence into safer care.”

Performance Assessment for Mastoidectomy

Objective The aim of this report is to provide a review of the current literature for assessment of performance for mastoidectomy, to identify the current assessment tools available in the literature, and to summarize the evidence for their validity. Data Sources The MEDLINE database was accessed via PubMed. Review Methods Inclusion criteria consisted of English-language published articles that reported use of a mastoidectomy performance assessment tool. Studies ranged from 2007 to November 2015 and were divided into 2 groups: intraoperative assessments and those performed with simulation (cadaveric laboratory or virtual reality). Studies that contained specific reliability analyses were also highlighted. For each publication, validity evidence data were analyzed and interpreted according to conceptual definitions provided in a recent systematic review on the modern framework of validity evidence. Conclusions Twenty-three studies were identified that met our inclusion criteria for review, including 4 intraoperative objective assessment studies, 5 cadaveric studies, 10 virtual reality simulation studies, and 4 that used both cadaveric assessment and virtual reality. Implications for Practice A review of the literature revealed a wide variety of mastoidectomy assessment tools and varying levels of reliability and validity evidence. The assessment tool developed at Johns Hopkins possesses the most validity evidence of those reviewed. However, a number of agreed-on specific metrics could be integrated into a standardized assessment instrument to be used nationally. A universally agreed-on assessment tool will provide a means for developing standardized benchmarks for performing mastoid surgery.

Mental practice in postgraduate training: a randomized controlled trial in mastoidectomy skills

Background

Mental practice, the cognitive rehearsal of a task in the absence of overt physical movement, has been successfully used in teaching complex psychomotor tasks including sports and music, and recently, surgical skills.

The objectives of this study were, 1) To develop and evaluate a mental practice protocol for mastoidectomy 2) To assess the immediate impact of mental practice on a mastoidectomy surgical task among senior Otolaryngology─Head & Neck Surgery (OHNS) residents.

Method

Three expert surgeons were interviewed using verbal protocol analysis to develop a mastoidectomy mental practice script. Twelve senior Residents from Canadian training programs were randomized into two groups. All Residents were video-recorded performing a baseline mastoidectomy in a temporal bone lab. The intervention group received mental practice training, while the control group undertook self-directed textbook study. All subjects were then video-recorded performing a second mastoidectomy. Changes in pre- and post-test scores using validated expert ratings, the Task Specific Evaluation of Mastoidectomy and the Global Evaluation of Mastoidectomy, were statistically analyzed.

Results

A mental practice script was successfully developed based on interviews of three expert surgeon-educators. Task Specific Evaluation and Global Evaluation scores increased in both the mental practice and textbook study groups; there was no significant difference between the two groups in the change in scores post-intervention. There was a high and statistically signficant correlation between evaluators on the outcome measures.

Conclusions

We were not able to demonstrate a significant difference for the benefits of mental practice in mastoidectomy, possibly due to the sample size. However, mental practice is a surgical education tool which is portable, accessible, inexpensive and safe.

Electronic supplementary material

The online version of this article (doi:10.1186/s40463-016-0162-2) contains supplementary material, which is available to authorized users.

Integration of high-resolution data for temporal bone surgical simulations

PURPOSE

To report on the state of the art in obtaining high-resolution 3D data of the microanatomy of the temporal bone and to process that data for integration into a surgical simulator. Specifically, we report on our experience in this area and discuss the issues involved to further the field.

DATA SOURCES

Current temporal bone image acquisition and image processing established in the literature as well as in house methodological development.

REVIEW METHODS

We reviewed the current English literature for the techniques used in computer-based temporal bone simulation systems to obtain and process anatomical data for use within the simulation. Search terms included "temporal bone simulation, surgical simulation, temporal bone." Articles were chosen and reviewed that directly addressed data acquisition and processing/segmentation and enhancement with emphasis given to computer-based systems. We present the results from this review in relationship to our approach.

CONCLUSIONS

High-resolution CT imaging ([Formula: see text] voxel resolution), along with unique image processing and rendering algorithms, and structure-specific enhancement are needed for high-level training and assessment using temporal bone surgical simulators. Higher-resolution clinical scanning and automated processes that run in efficient time frames are needed before these systems can routinely support pre-surgical planning. Additionally, protocols such as that provided in this manuscript need to be disseminated to increase the number and variety of virtual temporal bones available for training and performance assessment.

Evaluating the reliability of surgical assessment methods in an orthopedic residency program

BACKGROUND

Orthopedic surgical education in Canada has seen major change in the last 15 years. Work hour restrictions and external influence have led to new approaches for surgical training. With a change toward competency-based educational models under the CanMEDS headings there is a need to ensure the validity of modern assessment methods. Our objective was to evaluate the reliability of a currently used surgical skill assessment tool within an orthopedic residency program, as measured by the Surgical Encounters Form.

METHODS

A surgical assessment tool has previously been created at our institution that comprises 15 items spanning 4 of the CanMEDS competencies. Results were blinded to the primary investigator and coded by a third party. The assessments were collected, and we measured percent agreement using Cronbach's α and Fleiss κ.

RESULTS

Over a 5-month period 11 staff members assessed 10 residents. Eighty-eight assessments were completed in total. Weighted percent agreement was 90.9%. Cronbach's α averaged 0.865 for the medical expert role, 0.920 for technical skills, 0.934 for the communicator role, 1.00 for the collaborator role and 1.00 for the health advocate role. The mean Fleiss κ score was 0.147 (95% confidence interval -0.071 to 0.364), demonstrating low interrater reliability.

CONCLUSION

Despite the development of a validated assessment tool to evaluate surgical skills acquisition, interrater reliability results suggest low levels of agreement among assessors.

Peak and ceiling effects in final-product analysis of mastoidectomy performance

BACKGROUND

Virtual reality surgical simulation of mastoidectomy is a promising training tool for novices. Final-product analysis for assessing novice mastoidectomy performance could be limited by a peak or ceiling effect. These may be countered by simulator-integrated tutoring.

METHODS

Twenty-two participants completed a single session of self-directed practice of the mastoidectomy procedure in a virtual reality simulator. Participants were randomised for additional simulator-integrated tutoring. Performances were assessed at 10-minute intervals using final-product analysis.

RESULTS

In all, 45.5 per cent of participants peaked before the 60-minute time limit. None of the participants achieved the maximum score, suggesting a ceiling effect. The tutored group performed better than the non-tutored group but tutoring did not eliminate the peak or ceiling effects.

CONCLUSION

Timing and adequate instruction is important when using final-product analysis to assess novice mastoidectomy performance. Improved real-time feedback and tutoring could address the limitations of final product based assessment.

Feasibility of a synthetic temporal bone for training in mastoidectomy: face, content, and concurrent validity

OBJECTIVE

To investigate the face, content, and concurrent validity of the synthetic Pettigrew temporal bone (PTB) for mastoidectomy training as compared with cadaveric temporal bone (CTB).

STUDY DESIGN

A prospective evaluation study.

METHODS

Participants were invited to perform a step-by-step modified radical mastoidectomy using both bones and complete a 22-item, 5-point Likert scale questionnaire. The questionnaire is divided into 4 domains: face validity (FV), global content (GC), task-specific content (TSC), and curriculum recommendation (CR).

RESULTS

Thirty-six experts and 89 trainees completed all tasks, 63 using CTB and 62 using PTB. The PTB median FV was 4 (IQR: 4-5), GC of 4 (IQR: 4-5), TSC of 4 (IQR: 3-4), and CR of 4 (IQR: 4-5). The CTB was rated significantly higher than PTB by both groups in all domains; CTB FV: 5 (IQR: 4-5), GC: 5 (IQR: 4-5), TSC: 5 (IQR: 4-5), and CR: 5 (IQR: 5-5), p < 0.001 for each. Trainees rated PTB significantly higher than experts in all domains. There was no statistically significant difference between experts and trainees in rating the CTB in all domains. PTB gives similar haptic feedback to CTB, allows the use of suction and irrigation, has the important landmarks painted for identification, and contains articulating ossicles. The facial nerve anatomy was found to be inaccurate around the region of the second genu.

CONCLUSION

Participants found PTB to be valid for teaching some, yet not all, aspects of mastoid surgery, and experts agreed that it could improve global transferrable otologic skills. It is essential that the facial nerve anatomy is addressed before recommending this model.

Resident participation in cadaveric temporal bone dissection correlates with improved performance on a standardized skill assessment instrument

HYPOTHESIS

Increasing numbers of cadaveric temporal bone (CTB) dissection translates to improved scores on a timed microdissection of a CTB.

BACKGROUND

Recent literature regarding resident education has focused on virtual learning. However, advocates for virtual temporal bone drilling admit that there is not yet a substitute for drilling a CTB.

METHODS

Retrospective review of resident performance on a standardized instrument during a timed microdissection of CTBs. Resident performance on the graded dissection was compared with the number of CTBs drilled during the year. Graded performance was also compared with the total number of CTBs dissected over 4 years of residency. Faculty assessed intraoperative skill of the senior residents. These rankings were compared with the number of CTBs drilled. Comparisons were made using Pearson's and Spearman's correlations.

RESULTS

Comparison of test scores from the most recent resident year to the number of CTBs drilled during the corresponding year correlated well (r = 0.41, p = 0.002). The correlation between the score during the highest year of training and the cumulative number of CTB drilled during residency was even stronger (r = 0.604, p = 0.005). Faculty rankings correlated well comparing general surgical skills with TB surgical skills (r = 0.655, p = 0.008). Comparing faculty rankings of TB surgical skill with the number of CTB drilled during the final year of residency yielded a negative correlation (r = -0.8) but was not significant (p = 0.1).

CONCLUSION

Greater exposure to CTB dissection correlates with improved scoring on a standardized instrument. Residents who struggle with temporal bone surgery tend to use CTB dissection more than those who are more facile.