Predicting Resident Competence for Otolaryngology Key Indicator Procedures

OBJECTIVE

Competency-based surgical education requires practical assessments and meaningful benchmarks. In otolaryngology, key indicator procedure (KIP) minima are indicators of surgical exposure during training, yet it remains unknown how many times trainees must be evaluated on KIPs to ensure operative competence. Herein, we used Bayesian mixed effects models to compute predicted performance expectations for KIPs.

METHODS

From November 2017 to September 2021, a smartphone application (SIMPL OR) was used by attendings at five otolaryngology training programs to rate resident operative performance after each case on a five-level scale. Bayesian mixed effects models were used to estimate the probability that postgraduate year (PGY) 3, 4, or 5 trainees would earn a "practice-ready" (PR) rating on a subsequent evaluation based on their previously earned PR ratings for each KIP. Probabilities of earning a subsequent PR rating were examined for interpretability, and cross-validation was used to assess predictive validity.

RESULTS

A total of 842 assessments of KIPs were submitted by 72 attendings for 92 residents PGY 2-5. The predictive model had an average Area Under the Receiver Operating Curve of 0.77. The number of prior PR ratings that senior residents needed to attain a 95% probability of earning a PR rating on a subsequent evaluation was estimated for each KIP. For example, for mastoidectomies, PGY4 residents needed to earn 10 PR ratings whereas PGY5 residents needed 4 PR ratings on average to have a 95% probability of attaining a PR rating on a subsequent evaluation.

CONCLUSION

Predictive modeling can inform assessment benchmarks for competency-based surgical education.

LEVEL OF EVIDENCE

NA Laryngoscope, 133:3341-3345, 2023.

Changes in functional neuroimaging measures as novices gain proficiency on the fundamentals of laparoscopic surgery suturing task

Significance

As trainees practice fundamental surgical skills, they typically rely on performance measures such as time and errors, which are limited in their sensitivity.

Aim

The goal of our study was to evaluate the use of portable neuroimaging measures to map the neural processes associated with learning basic surgical skills.

Approach

Twenty-one subjects completed 15 sessions of training on the fundamentals of laparoscopic surgery (FLS) suture with intracorporeal knot-tying task in a box trainer. Functional near infrared spectroscopy data were recorded using an optode montage that covered the prefrontal and sensorimotor brain areas throughout the task. Average oxy-hemoglobin (HbO) changes were determined for repetitions performed during the first week of training compared with the third week of training. Statistical differences between the time periods were evaluated using a general linear model of the HbO changes.

Results

Average performance scores across task repetitions increased significantly from the first day to the last day of training ( p < 0.01 ). During the first day of training, there was significant lateral prefrontal cortex (PFC) activation. On the final day, significant activation was observed in the PFC, as well as the sensorimotor areas. When comparing the two periods, significant differences in activation ( p < 0.05 ) were found for the right medial PFC and the right inferior parietal gyrus. While gaining proficiency, trainees activated the perception-action cycle to build a perceptual model and then apply the model to improve task execution.

Conclusions

Learners engaged the sensorimotor areas more substantially as they developed skill on the FLS suturing task. These findings are consistent with findings for the FLS pattern cutting task and contribute to the development of objective metrics for skill evaluation.

Multi-Modal Deep Learning for Assessing Surgeon Technical Skill

This paper introduces a new dataset of a surgical knot-tying task, and a multi-modal deep learning model that achieves comparable performance to expert human raters on this skill assessment task. Seventy-two surgical trainees and faculty were recruited for the knot-tying task, and were recorded using video, kinematic, and image data. Three expert human raters conducted the skills assessment using the Objective Structured Assessment of Technical Skill (OSATS) Global Rating Scale (GRS). We also designed and developed three deep learning models: a ResNet-based image model, a ResNet-LSTM kinematic model, and a multi-modal model leveraging the image and time-series kinematic data. All three models demonstrate performance comparable to the expert human raters on most GRS domains. The multi-modal model demonstrates the best overall performance, as measured using the mean squared error (MSE) and intraclass correlation coefficient (ICC). This work is significant since it demonstrates that multi-modal deep learning has the potential to replicate human raters on a challenging human-performed knot-tying task. The study demonstrates an algorithm with state-of-the-art performance in surgical skill assessment. As objective assessment of technical skill continues to be a growing, but resource-heavy, element of surgical education, this study is an important step towards automated surgical skill assessment, ultimately leading to reduced burden on training faculty and institutes.

Reconstructing the nasal septum from instrument motion during septoplasty surgery

Purpose: Surgery involves modifying anatomy to achieve a goal. Reconstructing anatomy can facilitate surgical care through surgical planning, real-time decision support, or anticipating outcomes. Tool motion is a rich source of data that can be used to quantify anatomy. Our work develops and validates a method for reconstructing the nasal septum from unstructured motion of the Cottle elevator during the elevation phase of septoplasty surgery, without need to explicitly delineate the surface of the septum. Approach: The proposed method uses iterative closest point registration to initially register a template septum to the tool motion. Subsequently, statistical shape modeling with iterative most likely oriented point registration is used to fit the reconstructed septum to Cottle tip position and orientation during flap elevation. Regularization of the shape model and transformation is incorporated. The proposed methods were validated on 10 septoplasty surgeries performed on cadavers by operators of varying experience level. Preoperative CT images of the cadaver septums were segmented as ground truth. Results: We estimated reconstruction error as the difference between the projections of the Cottle tip onto the surface of the reconstructed septum and the ground-truth septum segmented from the CT image. We found translational differences of 2.74 ( 2.06 - 2.81 )    mm and a rotational differences of 8.95 ( 7.11 - 10.55 )    deg between the reconstructed septum and the ground-truth septum [median (interquartile range)], given the optimal regularization parameters. Conclusions: Accurate reconstruction of the nasal septum can be achieved from tool tracking data during septoplasty surgery on cadavers. This enables understanding of the septal anatomy without need for traditional medical imaging. This result may be used to facilitate surgical planning, intraoperative care, or skills assessment.

Developing and Assessing the Feasibility of Implementing a Surgical Objective Structured Clinical Skills Examination (S-OSCE)

OBJECTIVE

To1 describe the development and evaluate the feasibility of a surgical objective structured clinical examination (OSCE) for the purpose of competency assessment based on the Royal College of Canada's CanMEDS framework.

DESIGN

A unique surgical OSCE was developed to evaluate the clinical and surgical management of common orthopaedic problems using simulated patients and cadaveric specimens. Cases were graded by degree of difficulty (less complex, complex, more complex) Developing an assessment tool with significant resource utilization and good correlation with traditional methods is challenging. The feasibility of an OSCE that evaluates independent clinical and surgical decision making was evaluated. In addition, as part of establishing construct validity, correlation of OSCE scores with previously validated O-scores was performed.

SETTING

A tertiary level academic teaching hospital.

PARTICIPANTS

Thirty-four Postgraduate year 3-5 trainees of a 5-year Canadian orthopedic residency program creating 96 operative case performances available for final review.

RESULTS

The development of the OSCE cases involved a multistep process with attending surgeons, residents and a surgical education consultant. There were 4 different OSCE days, over a 3 year period (2016-2018) encompassing a variety of less complex and more complex procedures. Performance on the OSCE correlated strongly with the (O-SCORE, 0.89) and a linear regression analysis correlated moderately with year of training (r² = 0.5737). The feasibility analysis demonstrated good financial practicality with solid programmatic integration.

CONCLUSIONS

The unique surgical OSCE scores correlate strongly with an established entrustability scale. Administering this OSCE to evaluate preoperative and intraoperative decision making to complement other forms of assessment is feasible. The financial burden to training programs is modest in comparison to the insight gained by both residents and faculty.

Intraoperative Point of View Video Capture and Surgical Segmentation in Carpal Tunnel Release: A Feasibility Analysis

OBJECTIVE

The purpose of this study was to (1) examine the feasibility of intraoperative point of view video while performing open and endoscopic carpal tunnel release (CTR), (2) define surgical segments of CTR, and (3) describe the duration of various surgical steps of open versus endoscopic CTR in a teaching setting.

DESIGN

Fellowship trained hand surgeons reached consensus on surgical segments for CTR. Adult patients 18 and older previously indicated for CTR in clinic were eligible. Head-mounted point-of-view cameras were worn during endoscopic and open CTR by resident surgeons. Video was reviewed to determine segment duration. Independent sample t tests were used for comparison of duration by technique with statistical significance set as p < 0.05.

SETTING

University of Iowa Hospitals and Clinics; 200 Hawkins Dr, Iowa City, IA 52242; Tertiary Academic Medical Center.

PARTICIPANTS

Orthopedic Surgery Residents and Orthopedic Surgery Faculty.

RESULTS

Surgical segments were defined as incision, dissection of superficial soft tissue structures, transection of the carpal ligament, and surgical incision closure. Twelve of 14 video capture events yielded data. In the teaching setting, the average duration of endoscopic CTR was 609.5 seconds (±111.07) versus 547.75 seconds (±82.06) for open with p value = 0.406. No surgical segments were significantly different. Transition time from dissection to ligament transection differed significantly (p = 0.004) between endoscopic (46.88 seconds ± 19.19) and open (9.0 seconds ± 7.90) CTR. Transition time between ligament transection and closure was significantly different (p = 0.029) among endoscopic (50.5 seconds ± 15.0) and open (26.25 seconds ± 2.99) CTR.

CONCLUSIONS

Point-of-view video capture is feasible for the capture of video during a common hand surgery procedure. A method for managing device battery power is necessary for future applications. CTR can be defined as, and described in, individual procedure segments potentially useful for surgical education as well as efficiency improvements. Identification of surgical segments may aid the development of better objective tools for the assessment of surgeon skill and competency for common orthopedic procedures.

Use of a Novel Measure of Nontechnical Skills in Surgical Trainees: Is There an Association With Technical Skills Performance?

OBJECTIVE

To investigate whether scores on a psychological measure of concentration and interpersonal characteristics, The Attentional and Interpersonal Style Inventory (TAIS), are associated with performance of surgical skills.

DESIGN

Postgraduate surgical trainees completed an operative surgical skills assessment in the simulation laboratory and the psychological measure (TAIS). The surgical skills assessment consisted of 6 tasks (3 per trainee): laceration suturing; lipoma excision; incision and closure of a laparotomy wound; bowel anastomosis; saphenofemoral junction ligation and basic laparoscopic skills. The association between operative surgical skill performance and TAIS factors was investigated.

SETTING

The TAIS assessments and surgical skills assessments were conducted at the National Surgical Training Centre at the Royal College of Surgeons in Ireland (RCSI).

PARTICIPANTS

One hundred and two surgical trainees in years one and two (PGY 2-3 equivalent) participated in the study.

RESULTS

Performance on 2 of the 6 tasks assessed (bowel anastomosis and lipoma excision) were positively associated with multiple TAIS factors (energy, confidence, competitiveness, extroversion, self-criticism and performing under pressure). Another factor, focus over time, was significantly associated with scores on the lipoma excision task.

CONCLUSIONS

Trainees with high levels of energy, confidence, competitiveness, extroversion, and focus over time and low levels of self-criticism demonstrated better performance on specific technical skills tasks.

Energy-Based Metrics for Arthroscopic Skills Assessment

Minimally invasive skills assessment methods are essential in developing efficient surgical simulators and implementing consistent skills evaluation. Although numerous methods have been investigated in the literature, there is still a need to further improve the accuracy of surgical skills assessment. Energy expenditure can be an indication of motor skills proficiency. The goals of this study are to develop objective metrics based on energy expenditure, normalize these metrics, and investigate classifying trainees using these metrics. To this end, different forms of energy consisting of mechanical energy and work were considered and their values were divided by the related value of an ideal performance to develop normalized metrics. These metrics were used as inputs for various machine learning algorithms including support vector machines (SVM) and neural networks (NNs) for classification. The accuracy of the combination of the normalized energy-based metrics with these classifiers was evaluated through a leave-one-subject-out cross-validation. The proposed method was validated using 26 subjects at two experience levels (novices and experts) in three arthroscopic tasks. The results showed that there are statistically significant differences between novices and experts for almost all of the normalized energy-based metrics. The accuracy of classification using SVM and NN methods was between 70% and 95% for the various tasks. The results show that the normalized energy-based metrics and their combination with SVM and NN classifiers are capable of providing accurate classification of trainees. The assessment method proposed in this study can enhance surgical training by providing appropriate feedback to trainees about their level of expertise and can be used in the evaluation of proficiency.

Comparison of Canadian and Swiss Surgical Training Curricula: Moving on Toward Competency-Based Surgical Education

OBJECTIVE

Quality of surgical training in the era of resident duty-hour restrictions (RDHR) is part of an ongoing debate. Most training elements are provided during surgical service. As exposure to surgical procedures is important but time-consuming, RDHR may affect quality of surgical training. Providing structured training elements may help to compensate for this shortcoming.

DESIGN

This binational anonymous questionnaire-based study evaluates frequency, time, and structure of surgical training programs at 2 typical academic teaching hospitals with different RDHR.

SETTING

Departments of Surgery of University of Basel (Basel, Switzerland) and the Queen's University (Kingston, Ontario, Canada).

PARTICIPANTS

Surgical consultants and residents of the Queen's University Hospital (Kingston, Ontario, Canada) and the University Hospital Basel (Basel, Switzerland) were eligible for this study.

RESULTS

Questionnaire response rate was 37% (105/284). Queen's residents work 80 hours per week, receiving 7 hours of formal training (8.8% of workweek). Basel residents work 60 hours per week, including 1 hour of formal training (1.7% of working time). Queen's faculty and residents rated their program as "structured" or "rather structured" in contrast to Basel faculty and residents who rated their programs as "neutral" in structure or "unstructured." Respondents identified specific structured training elements more frequently at Queen's than in Basel. Two-thirds of residents responded that they seek out additional surgical experiences through voluntary extra work. Basel participants articulated a stronger need for improvement of current surgical training. Although Basel residents and consultants in both institutions fear negative influence of RDHR on the training program, this was not the case in Queen's residents.

CONCLUSIONS

Providing more structured surgical training elements may be advantageous in providing optimal-quality surgical education in an era of work-hour restrictions.

Robotic microsurgical training and evaluation

Robotic surgery has expanded rapidly over the past two decades and is in widespread use among the surgical subspecialties. Clinical applications in plastic surgery have emerged gradually over the last few years. One of the promising applications is robotic-assisted microvascular anastomosis. Here the authors first describe a process by which an assessment instrument they developed called the Structured Assessment of Robotic Microsurgical Skills (SARMS) was validated. The instrument combines the previously validated Structured Assessment of Microsurgical Skills (SAMS) with other skill domains in robotic surgery. Interrater reliability for the SARMS instrument was excellent for all skill areas among four expert, blinded evaluators. They then present a process by which the learning curve for robotic-assisted microvascular anastomoses was measured and plotted. Ten study participants performed five robotic microanastomoses each that were recorded, deidentified and scored. Trends in SARMS scores were plotted. All skill areas and overall performance improved significantly for each participant over the five microanastomotic sessions, and operative time decreased for all participants. The results showed an initial steep ascent in technical skill acquisition followed by more gradual improvement, and a steady decrease in operative times for the cohort. Participants at all levels of training, ranging from minimal microsurgical experience to expert microsurgeons gained proficiency over the course of five robotic sessions.