Proficiency Levels and Validity Evidence for Scoring Metrics for a Virtual Reality and Inanimate Robotic Surgery Simulation Curriculum

OBJECTIVE

Our institution recently implemented a virtual reality (VR) skills curriculum for general surgery residents using the SimNow simulator. Based on a content alignment study, we revised the curriculum to include only 20 of 33 VR tasks and we added 3 previously validated inanimate tasks. The purpose of this study was to establish expert-derived proficiency levels for all tasks and to evaluate the validity of the scoring for the VR tasks.

DESIGN

Two expert robotic surgeons performed 5 repetitions of each VR and inanimate task. The trimmed mean (lowest scoring attempt and outliers [>2 standard deviations] were eliminated) was defined as the expert level for each task. For the VR tasks, expert levels were compared to resident performance to evaluate validity.

SETTING

This study was conducted at the University of Texas Southwestern Medical Center (Dallas, TX), a tertiary care academic teaching hospital.

PARTICIPANTS

Two expert robotic surgeons participated in this study. The data from 42 residents (PGY2-4) who completed the original curriculum was used to represent novice performance.

RESULTS

Comparison of expert levels and resident performance was statistically significant for 15 VR tasks (supporting validity) and approached significance (p = 0.06, 0.09) for 2 VR tasks; expert levels were designated as proficiency levels for these 17 tasks. Group comparisons were clearly not significant (p = 0.2-0.8) for 3 VR tasks; 2 of these 3 tasks were retained as introductory exercises (with 3 repetitions required) and 1 was excluded. For the 3 inanimate tasks, expert levels minus 2 standard deviations were designated as proficiency levels.

CONCLUSIONS

This analysis generated validity evidence for 15 VR tasks and established expert-derived proficiency levels for 17 VR tasks and 3 inanimate tasks. Our proposed curriculum now consists of 19 VR and 3 inanimate tasks using the selected proficiency levels. We anticipate that this design will maximize curriculum efficiency and effectiveness.

Assessing North Texas Regional Trauma Handoffs: A Multicenter Mixed-Methods Needs Assessment

INTRODUCTION

Trauma video review of Emergency Medical Services (EMS) handoffs demonstrates frequent problems including interruptions and incomplete information transfer. This study aimed to perform a regional needs assessment of handoff perceptions and expectations to guide future standardization efforts.

METHODS

A multidisciplinary team of trauma providers through consensus building created an anonymous survey which was then distributed through the North Central Texas Trauma Regional Advisory Council and four regional level-1 trauma institutions. Qualitative data underwent content analysis; quantitative data are presented with descriptive statistics.

RESULTS

Survey responses (n = 249) were submitted by trauma nurses (38%), EMS (24%), emergency physicians (14%), and trauma physicians (13%). Median overall handoff quality was rated well (4, scale 1-5) despite some variability between hospitals (3, scale 1-5). The top five most important handoff details were the same for both stable and unstable patients: primary mechanism, blood pressure, heart rate, Glasgow Coma Scale, and location of injuries. While providers felt neutral about the data order, the vast majority supported immediate bed transfer and primary survey in unstable patients. The majority of receiving providers report interrupting handoff at least once (78%); and 66% of EMS clinicians found interruptions disruptive. Content analysis revealed top priority categories for improvement: environment, communication, information relayed, team dynamics, and flow of care.

CONCLUSION

Although our data demonstrated satisfaction and concordance with respect to the EMS handoff, 84% of EMS clinicians reported some to high amounts of variability across institutions. Gaps in the development of standardized handoffs identified include exposure, education, and enforcement of these protocols.

Evaluating emergency department tube thoracostomy: A single-center use of trauma video review to assess efficiency and technique

BACKGROUND

Emergency department tube thoracostomy is a common procedure; however, assessing procedural skills is difficult. We sought to describe procedural variability and technical complications of emergency department tube thoracostomy using trauma video review. We hypothesized that factors such as hemodynamic abnormality lead to increased technical difficulty and malpositioning.

METHODS

Using trauma video review, we reviewed all emergency department tube thoracostomy from 2020 to 2022. Patients were stratified into hemodynamically abnormal (systolic blood pressure <90 or heart rate >120) and hemodynamically normal (systolic blood pressure ≥90 or heart rate ≤120). Emergency department tube thoracostomies outside of video-capable rooms, with incomplete visualization, or in patients undergoing cardiopulmonary resuscitation or resuscitative thoracotomy were excluded. The primary outcome was a procedure score modified from the validated tool ranging from 0 to 11 (higher score indicating better performance). Also measured were procedural times to (1) decision to place, (2) pleural entry, and (3) procedure completion. Postprocedure x-ray and chart review were used to determine accuracy.

RESULTS

In total, 51 videos met the inclusion criteria. The median age was 34 [interquartile range 24-40] years, body mass index 25.8 [interquartile range 21.8-30.7], predominately male (75%), blunt injury (57%), with Injury Severity Score of 22 [14.5-41]. The median procedure score was 9 [7-10]. Emergency department tube thoracostomies in patients with abnormal hemodynamics had significantly lower procedure scores (8 vs 10, P < .05). Hemodynamically abnormal patients had significantly shorter times from decision to proceed to pleural entry (4.05 vs 8.25 minutes, P < .001), and to completion (6.31 vs 14.23 minutes, P < .001). The most common complication was malpositioning (35.1%), with no significant difference noted when comparing hemodynamically normal and abnormal patients (P = .41).

CONCLUSION

Using trauma video review we identified significant procedural variability in emergency department tube thoracostomy, mainly that hemodynamic abnormality led to lower proficiency scores and increased malpositioning. Efforts are needed to define procedural benchmarks and evaluation in the context of patient outcomes. Using this technology and methodology can help establish procedural norms.

Developing a Robotic Surgery Curriculum: Selection of Virtual Reality Drills for Content Alignment

INTRODUCTION

Despite the importance of simulation-based training for robotic surgery, there is no consensus about its training curricula. Recently, a virtual reality (VR) platform (SimNow, Intuitive, Inc) was introduced with 33 VR drills but without evidence of their validity. As part of our creating a new robotic VR curriculum, we assessed the drills' validity through content mapping and the alignment between learning goals and drill content.

METHODS

Three robotically trained surgeons content-mapped all 33 drills for how well the drills incorporated 15 surgery skills and also rated the drills' difficulty, usefulness, relevance, and uniqueness. Drills were added to the new curriculum based on consensus about ratings and historic learner data. The drills were grouped according to similar skill sets and arranged in order of complexity.

RESULTS

The 33 drills were judged to have 12/15 surgery skills as primary goals and 13/15 as secondary goals. Twenty of the 33 drills were selected for inclusion in the new curriculum; these had 11/15 skills as primary goals and 11/15 as secondary goals. However, skills regarding energy sources, atraumatic handling, blunt dissection, fine dissection, and running suturing were poorly represented in the drills. Three previously validated inanimate drills were added to the curriculum to address lacking skill domains.

CONCLUSIONS

We identified 20 of the 33 SimNow drills as a foundation for a robotic surgery curriculum based on content-oriented evidence. We added 3 other drills to address identified gaps in drill content.

Using Trauma Video Review to Assess EMS Handoff and Trauma Team Non-Technical Skills

OBJECTIVE

Handoffs by emergency medical services (EMS) personnel suffer from poor structure, inattention, and interruptions. The relationship between the quality of EMS communication and the non-technical performance of trauma teams remains unknown.

METHODS

We analyzed 3 months of trauma resuscitation videos (highest acuity activations or patients with an Injury Severity Score [ISS] of ≥15). Handoffs were scored using the mechanism-injury-signs-treatment (MIST) framework for completeness (0-20), efficiency (category jumps), interruptions, and timeliness. Trauma team non-technical performance was scored using the Trauma Non-Technical Skills (T-NOTECHS) scale (5-15).

RESULTS

We analyzed 99 videos. Handoffs lasted a median of 62 seconds [IQR: 43-74], scored 11 [10-13] for completeness, and had 2 [1-3] interruptions. Most interruptions were verbal (85.2%) and caused by the trauma team (64.9%). Most handoffs (92%) were efficient with 2 or fewer jumps. Patient transfer during handoff occurred in 53.5% of the videos; EMS providers giving handoff helped transfer in 69.8% of the Primary surveys began during handoff in 42.4% of the videos. Resuscitation teams who scored in the top-quartile on the T-NOTECHS (>11) had higher MIST scores than teams in lower quartiles (13 [11.25-14.75] vs. 11 [10-13]; p < .01). There were no significant differences in ISS, efficiency, timeliness, or interruptions between top- and lower-quartile groups.

CONCLUSIONS

There is a relationship between EMS MIST completeness and high performance of non-technical skill by trauma teams. Trauma video review (TVR) can help identify modifiable behaviors to improve EMS handoff and resuscitation efforts and therefore trauma team performance.

Educational Experience Impacts Wellness More than Hours Worked

OBJECTIVE

The Accreditation Council for Graduate Medical Education has focused its interests on resident wellbeing and the clinical work environment in recent years. Concerns regarding both duty hours as well as service obligations versus education resulted in programs nationwide receiving citations, including ours. This study aimed to evaluate the impact of those 2 factors on surgical residents' general wellbeing, hypothesizing that service obligations would be a stronger predictor.

DESIGN

We have previously reported on the use of a "Fuel Gauge" tool developed at our institution for monitoring resident wellbeing. We performed a retrospective comparison of prospectively collected cross-sectional survey data comparing the Fuel Gauge to a bimonthly "Service Versus Education" (SVE) report. This report used similar scaling and allowed residents to provide feedback on the balance of the educational quality of their current rotation in comparison to their perception of service obligation. Pearson's correlation was then used to compare those scores with duty hour logs to determine if a relationship could be identified between the 3 measurements.

SETTING

Academic institution of the University of Texas Southwestern in Dallas, Texas.

PARTICIPANTS

Active general surgery residents (n = 73).

RESULTS

During the study period, 73 residents filled out both a Fuel Gauge assessment and a SVE assessment at least once, with 273 complete data points available for analysis. Our program's Fuel Gauge median was 4, and our program's median SVE score was 4. Fuel Gauge assessment scores demonstrated a moderate positive correlation with SVE (r = 0.65, p < 0.001), while only a weakly negative association with increasing hours worked (r = -0.15, p = 0.015). SVE also demonstrated a weak negative correlation with hours logged (r = -0.225, p = 0.001).

CONCLUSIONS

While the Accreditation Council for Graduate Medical Education recognizes that multiple factors contribute to resident wellbeing issues, early efforts were focused on limiting excessive duty hours. Examining our institutional data regarding the previously understudied factor of SVE, we indeed found a stronger correlation with resident perception of low educational value rather than excessive work hours contributing to lower Fuel Gauge scores. These data, if verified, should guide program directors in identifying other institutional factors that may more strongly contribute to their own culture of resident wellness.

Survey of the 2020 Fellowship Council application and match process and the impact of COVID-19

BACKGROUND

The interview process represents a necessary but potentially resource intensive process from applicant and program perspectives. This study aimed to identify opinions of the 2020 Fellowship Council (FC) application and match process and in-cycle transition to virtual interviews due to the COVID-19 pandemic.

METHODS

Surveys were developed to assess the interview process and were distributed by the FC to all applicants and fellowship programs. Completion was voluntary and data (median [IQR] reported) were anonymous.

RESULTS

Applicant response was 53%. Applicants submitted 27.5 (13.25-40) applications, were offered 10 (4-17) interviews, and ranked 10 (5-15) programs. Due to COVID-19, 74% of interview plans changed. Applicants completed 30% of their planned in-person interviews. For decision-making, 90% felt that in-person and 81% virtual interviews were sufficiently informative. Expected cost was $4750 ($2000-$6000) vs. actual cost $1000 ($250-$2250), (p < 0.05). Expected missed work-days were 10 (5-16) versus actual 3 (0-6.25) (p < 0.05). For future interviews, 44% of applicants preferred in-person after virtual pre-interviews, 29% preferred virtual only, and 18% preferred in-person only. Program response was 38%. Programs received 60 (43-85.5) applications, offered 20 (15-26) interviews, completed 16 (12.5-21) interviews, and ranked 14 (10-18) candidates. For decision-making, 92% of programs felt in-person versus 71% virtual interviews were sufficiently informative. Person-hours were greater for in-person 48 (27.5-80) versus virtual 24 (9-40) interviews (p < 0.05). For future interviews, 38% of programs preferred in-person after virtual pre-interviews, 31% preferred in-person only, and 21% preferred virtual only.

CONCLUSION

Despite pandemic changes, 81% of applicants and 71% of programs felt they gained sufficient information from virtual sessions to create rank lists. Virtual interviews had lower costs and fewer missed work-days for applicants and decreased resource usage for programs. The majority of both groups preferred either solely virtual or virtual pre-interview followed by in-person interview formats. Virtual interviews should be incorporated into future fellowship application cycles.

Overcoming Health Literacy Barriers by Developing Standardized Surgical Discharge Instructions

INTRODUCTION

Approximately one-third of surgical patients exhibit low health literacy, and 39% of our patients are primary Spanish speakers. We first evaluated the current content of our arteriovenous fistula/graft discharge instruction (DCI) templates. Using the Plan-Do-Study-Act cycle quality improvement methodology, we then aimed to optimize the readability and formally translate new DCI and evaluate usage and inappropriate bouncebacks following implementation.

METHODS

Current arteriovenous fistula/graft template content was reviewed by the literacy department for readability and vascular faculty for completeness and accuracy. The literacy department edits were categorized by word choice, added/removed content, format change, and grammatical errors. Two vascular surgeons rated completeness and accuracy on a Likert scale (1-5). Retrospective chart review was performed for telephone calls and emergency department bouncebacks for 3 mo flanking new DCI implementation.

RESULTS

Of the 10 templates, all were in English and word count ranged from 192 to 990 words. Despite each template including all necessary subcategories, the median number of edits per 100 words was 9.2 [7.0-9.5]. Approximately half of the edits (5.4 [5.1-5.5]) were word choice edits. Overall, experts rated completeness at 3.9 [3.2-4.2] and accuracy at 4.0 [3.7-4.1]. Highest template utilization occurred during post-implementation months 1 (90%) and 3 (100%) with orientation sessions. There was a significant increase in concordant Spanish DCI use (P < 0.01) and no inappropriate bouncebacks after implementation.

CONCLUSIONS

Our study demonstrated notable variability in the content and readability of our vascular access instruction templates. New DCI had strong usage and language concordance; continued use may decrease bouncebacks.

At-home medical student simulation: achieving knot-tying proficiency using video-based assessment

Purpose

Due to the pandemic, we restructured our medical student knot-tying simulation to a virtual format. This study evaluated curriculum feasibility and effectiveness.

Methods

Over 4 weeks, second-year medical students (n = 229) viewed a video tutorial (task demonstration, errors, scoring) and self-practiced to proficiency (no critical errors, < 2 min) using at-home suture kits (simple interrupted suture, instrument tie, penrose drain model). Optional virtual tutoring sessions were offered. Students submitted video performance for proficiency verification. Two sets of 14 videos were viewed by two surgeons until inter-rater reliability (IRR) was established. Students scoring "needs remediation" attended virtual remediation sessions. Non-parametric statistics were performed using RStudio.

Results

All 229 medical students completed the curriculum within 1-4 h; 1.3% attended an optional tutorial. All videos were assessed: 4.8% "exceeds expectations", 60.7% "meets expectations", and 34.5% "needs remediation." All 79 needing remediation due to critical errors achieved proficiency during 1-h group sessions. IRR Cohen's κ was 0.69 (initial) and 1.0 (ultimate). Task completion time was 56 (47-68) s (median [IQR]); p < 0.01 between all pairs. Students rated the overall curriculum (79.2%) and overall curriculum and video tutorial effectiveness (92.7%) as "agree" or "strongly agree". No definitive preference emerged regarding virtual versus in-person formats; however, 80.2% affirmed wanting other at-home skills curricula. Comments supported home practice as lower stress; remediation students valued direct formative feedback.

Conclusions

A completely virtual 1-month knot-tying simulation is feasible and effective in achieving proficiency using video-based assessment and as-needed remediation strategies for a large student class.

Creating a Proficiency-based Remote Laparoscopic Skills Curriculum for the COVID-19 Era

OBJECTIVE

Social distancing restrictions due to COVID-19 challenged our ability to educate incoming surgery interns who depend on early simulation training for basic skill acquisition. This study aimed to create a proficiency-based laparoscopic skills curriculum using remote learning.

DESIGN

Content experts designed 5 surgical tasks to address hand-eye coordination, depth perception, and precision cutting. A scoring formula was used to measure performance: cutoff time - completion time - (K × errors) = score; the constant K was determined for each task. As a benchmark for proficiency, a fellowship-trained laparoscopic surgeon performed 3 consecutive repetitions of each task; proficiency was defined as the surgeon's mean score minus 2 standard deviations. To train remotely, PGY1 surgery residents (n = 29) were each issued a donated portable laparoscopic training box, task explanations, and score sheets. Remote training included submitting a pre-test video, self-training to proficiency, and submitting a post-test video. Construct validity (expert vs. trainee pre-tests) and skill acquisition (trainee pre-tests vs. post-tests) were compared using a Wilcoxon test (median [IQR] reported).

SETTING

The University of Texas Southwestern Medical Center in Dallas, Texas PARTICIPANTS: Surgery interns RESULTS: Expert and trainee pre-test performance was significantly different for all tasks, supporting construct validity. One trainee was proficient at pre-test. After 1 month of self-training, 7 additional residents achieved proficiency on all 5 tasks after 2-18 repetitions; trainee post-test scores were significantly improved versus pre-test on all tasks (p = 0.01).

CONCLUSIONS

This proficiency-based curriculum demonstrated construct validity, was feasible as a remote teaching option, and resulted in significant skill acquisition. The remote format, including video-based performance assessment, facilitates effective at-home learning and may allow additional innovations such as video-based coaching for more advanced curricula.

The Impact of COVID-19 on Surgical Education

BACKGROUND

At the onset of social distancing, our general surgery residency transitioned its educational curriculum to an entirely virtual format with no gaps in conference offerings. The aim of this study is to examine the feasibility of our evolution to a virtual format and report program attitudes toward the changes.

METHODS

On March 15, 2020, due to the coronavirus disease (COVID-19) our institution restricted mass gatherings. We immediately transitioned all lectures to a virtual platform. The cancellation of elective surgeries in April 2020 then created the need for augmented resident education opportunities. We responded by creating additional lectures and implementing a daily conference itinerary. To evaluate the success of the changes and inform the development of future curriculum, we surveyed residents and faculty regarding the changes. Classes and faculty answers were compared for perception of value of the online format.

RESULTS

Pre-COVID-19, residency-wide educational offerings were concentrated to one half-day per week. Once restrictions were in place, our educational opportunities were expanded to a daily schedule and averaged 16.5 hours/week during April. Overall, 41/63 residents and 25/94 faculty completed the survey. The majority of residents reported an increased ability (56%) or similar ability (34.1%) to attend virtual conferences while 9.9% indicated a decrease. Faculty responses indicated similar effects (64% increased, 32% similar, 4% decreased). PGY-1 residents rated the changes negatively compared to other trainees and faculty. PGY-2 residents reported neutral views and all other trainees and faculty believed the changes positively affected educational value. Comments from PGY1 and 2 residents revealed they could not focus on virtual conferences as it was not "protected time" in a classroom and that they felt responsible for patient care during virtual lectures. A majority of both residents (61%) and faculty (84%) reported they would prefer to continue virtual conferences in the future.

CONCLUSIONS

The necessity for adapting our academic offerings during the COVID-19 era has afforded our program the opportunity to recognize the feasibility of virtual platforms and expand our educational offerings. The majority of participants report stable to improved attendance and educational value. Virtual lectures should still be considered protected time in order to maximize the experience for junior residents.

Thriving or surviving? A critical examination of funding models for fellowship council fellowships

BACKGROUND

Since 1997, the Fellowship Council (FC) has evolved into a robust organization responsible for the advanced training of nearly half of the US residency graduates entering general surgery practice. While FC fellowships are competitive (55% match rate) and offer outstanding educational experiences, funding is arguably vulnerable. This study aimed to investigate the current funding models of FC fellowships.

METHODS

Under an IRB-approved protocol, an electronic survey was administered to 167 FC programs with subsequent phone interviews to collect data on total cost and funding sources. De-identified data were also obtained via 2020-2021 Foundation for Surgical Fellowships (FSF) grant applications. Means and ranges are reported.

RESULTS

Data were obtained from 59 programs (35% response rate) via the FC survey and 116 programs via FSF applications; the average cost to train one fellow per year was $107,957 and $110,816, respectively. Most programs utilized departmental and grants funds. Additionally, 36% (FC data) to 39% (FSF data) of programs indicated billing for their fellow, generating on average $74,824 ($15,000-200,000) and $33,281 ($11,500-66,259), respectively. FC data documented that 14% of programs generated net positive revenue, whereas FSF data documented that all programs were budget-neutral.

CONCLUSION

Both data sets yielded similar overall results, supporting the accuracy of our findings. Expenses varied widely, which may, in part, be due to regional cost differences. Most programs relied on multiple funding sources. A minority were able to generate a positive revenue stream. Although fewer than half of programs billed for their fellow, this source accounted for substantial revenue. Institutional support and external grant funding have continued to be important sources for the majority of programs as well. Given the value of these fellowships and inherent vulnerabilities associated with graduate medical education funding, alternative grant funding models and standardization of annual financial reporting are encouraged.

Antifungal Therapy in Fungal Necrotizing Soft Tissue Infections

BACKGROUND

Necrotizing soft tissue infections (NSTIs) are life-threatening surgical emergencies associated with high morbidity and mortality. Fungal NSTIs are considered rare and have been largely understudied. The purpose of this study was to study the impact of fungal NSTIs and antifungal therapy on mortality after NSTIs.

METHODS

A retrospective chart review was performed on patients with NSTIs from 2012 to 2018. Patient baseline characteristics, microbiologic data, antimicrobial therapy, and clinical outcomes were collected. Patients were excluded if they had comfort care before excision. The primary outcome measured was in-hospital mortality.

RESULTS

A total of 215 patients met study criteria with a fungal species identified in 29 patients (13.5%). The most prevalent fungal organism was Candida tropicalis (n = 11). Fungal NSTIs were more prevalent in patients taking immunosuppressive medications (17.2% versus 3.2%, P = 0.01). A fungal NSTI was significantly associated with in-hospital mortality (odds ratio, 3.13; 95% confidence interval, 1.16-8.40; P = 0.02). Furthermore, fungal NSTI patients had longer lengths of stay (32 d [interquartile range, 16-53] versus 19 d [interquartile range, 11-31], P < 0.01), more likely to require initiation of renal replacement therapy (24.1% versus 8.6%, P = 0.02), and more likely to require mechanical ventilation (64.5% versus 42.0%, P = 0.02). Initiation of antifungals was associated with a significantly lower rate of in-hospital mortality (6.7% versus 57.1%, P = 0.01).

CONCLUSIONS

Fungal NSTIs are more common in patients taking immunosuppressive medications and are significantly associated with in-hospital mortality. Antifungal therapy is associated with decreased in-hospital mortality in those with fungal NSTIs. Consideration should be given to adding antifungals in empiric treatment regimens, especially in those taking immunosuppressive medications.