A Scoping Review of the Literature on Entrustable Professional Activities in Surgery Residency Programs

OBJECTIVE

Entrustable professional activities (EPAs) are a crucial component of contemporary postgraduate medical education with many surgery residency programs having implemented EPAs as a competency assessment framework to assess and provide feedback on the performance of their residents. Despite broad implementation of EPAs, there is a paucity of evidence regarding the impact of EPAs on the learners and learning environments. A first step in improving understanding of the use and impact of EPAs is by mapping the rising number of EPA-related publications from the field of surgery. The primary objective of this scoping review is to examine the nature, extent, and range of articles on the development, implementation, and assessment of EPAs. The second objective is to identify the experiences and factors that influence EPA implementation and use in practice in surgical specialties.

DESIGN

Scoping review. Four electronic databases (Medline, Embase, Education Source, and ERIC) were searched on January 20, 2022, and then again on July 19, 2023. A quasi-statistical content analysis was employed to quantify and draw meaning from the information related to the development, implementation, assessment, validity, reliability, and experiences with EPAs in the workplace.

PARTICIPANTS

A total of 42 empirical and nonempirical articles were included.

RESULTS

Four thematic categories describe the topic areas in included articles related to: 1) the development and refinement of EPAs, including the multiple steps taken to develop and refine unique EPAs for surgery residency programs; 2) the methods for implementing EPAs; 3) outcomes of EPA use in practice; 4) barriers, facilitators, and areas for improvement for the implementation and use of EPAs in surgical education.

CONCLUSIONS

This scoping review highlights the key trends and gaps from the rapidly increasing number of publications on EPAs in surgery residency, from development to their use in the workplace. Existing EPA studies lack a theoretical and/or conceptual basis; future development and implementation studies should adopt implementation science frameworks to better structure and operationalize EPAs within surgery residency programs.

Surgery Clerkship Directors' Perceptions of the COVID-19 Pandemic's Impact on Medical Student Education

BACKGROUND

This study assessed the national impact of the coronavirus disease 2019 (COVID-19) pandemic on the education of medical students assigned to surgery clerkship rotations, as reported by surgery clerkship directors (CDs).

STUDY DESIGN

In the spring of 2020 and 2021, the authors surveyed 164 CDs from 144 Liaison Committee on Medical Education-accredited US medical schools about their views of the pandemic's impact on the surgery clerkship curriculum, students' experiences, outcomes, and institutional responses.

RESULTS

Overall survey response rates, calculated as number of respondents/number of surveyed, were 44.5% (73 of 164) and 50.6% (83 of 164) for the spring 2020 and 2021 surveys, respectively. Nearly all CDs (more than 95%) pivoted to virtual platforms and solutions. Most returned to some form of in-person learning by winter 2020, and prepandemic status by spring 2021 (46%, 38 of 83). Students' progression to the next year was delayed by 12% (9 of 73), and preparation was negatively impacted by 45% (37 of 83). Despite these data, CDs perceived students' interest in surgical careers was not significantly affected (89% vs 77.0%, p = 0.09). During the 1-year study, the proportion of CDs reporting a severe negative impact on the curriculum dropped significantly (p < 0.0001) for most parameters assessed except summative evaluations (40.3% vs 45.7%, p = 0.53). CDs (n = 83) also noted the pandemic's positive impact with respect to virtual patient encounters (21.7%), didactics (16.9%), student test performance (16.9%), continuous personal learning (14.5%), engagement in the clerkship (9.6%), and student interest in surgery as a career (7.2%).

CONCLUSIONS

During the pandemic, the severe negative impact on student educational programs lessened, and novel virtual curricular solutions emerged. Student interest in surgery as a career was sustained. Measures of student competency and effectiveness of new curriculum, including telehealth, remain areas for future investigation.

Surgical skills training for practicing surgeons founded on established educational theories and frameworks

Following completion of structured training in residency or fellowship programs, surgeons need to acquire new skills throughout their careers to address changes in their surgical practices, and to perform new procedures and use new technologies as these are developed. The opportunities available to surgeons to acquire the new skills, safely introduce these skills into surgical practice, and then maintain the skills, vary greatly and may not fully support a surgeon's needs. The article shares background information relating to the vexing problems, and describes established educational theories and frameworks as well as specific frameworks relating to surgical skills training and verification that may be used in designing and implementing comprehensive and impactful surgical skills training programs aimed at practicing surgeons. A number of practical strategies based on these theories and frameworks are highlighted. National professional organizations and academic medical centers need to play a critical role in pursuing such efforts, which will remain pivotal in supporting the professional work of surgeons and in providing optimal surgical care well into the future.

Piloting a National Curricular Resource for the Transition to Surgical Residency: Characteristics of Participating Schools and Their Students

OBJECTIVE

To determine the extent to which U.S. Liaison Committee of Medical Education (LCME)-accredited medical schools chose to participate in piloting a national curricular resource, the American College of Surgeons/Association of Program Directors in Surgery/Association of Surgical Education Resident Prep Curriculum ("ACS-surgery-prep curriculum"), and implications of such participation for student access nationally to this resource.

DESIGN

We examined the significance of school-level differences in ACS-surgery-prep curriculum pilot participation and student-level differences in curriculum access based on medical school attended in bivariate analysis.

SETTING

U.S. medical schools choosing to participate in the ACS-surgery-prep curriculum through 2021. Students graduating from U.S. LCME-accredited medical schools in 2020-2021 were invited to complete the Association of American Medical Colleges 2021 Graduation Questionnaire (GQ).

PARTICIPANTS

Our study included data for 2569 students intending surgery specialties (16% of 16,353 2021 GQ respondents) from ACS-surgery-prep curriculum pilot and non-pilot schools.

RESULTS

Of 148 medical schools attended by 2021 GQ respondents, 93 (63%) were identified as ACS-surgery-prep curriculum pilot schools. Pilot participation varied by school region, community-based designation, and research intensity (each p < 0.05) but not by ownership or transition to residency (TTR) course requirements (each p > 0.05). Of 2569 GQ respondents nationally intending surgery specialties, 1697 (66%) attended an ACS-surgery-prep curriculum pilot school; this proportion did not vary by gender or race/ethnicity (each p > 0.05) but varied by students' school TTR course requirements (p < 0.001). Findings were similar among the 1059 students intending general surgery specialties specifically (41% of all 2569 students intending surgery specialties).

CONCLUSIONS

Many U.S. LCME-accredited medical schools piloted this national TTR surgery curriculum. School-level characteristics associated with pilot participation can inform outreach efforts to encourage the participation of interested schools in piloting this TTR resource. With this curriculum distribution model, we observed no gender or racial/ethnic disparities in curriculum access nationally among students intending surgery specialties.

A review of the American College of Surgeons accredited education institutes' assessment practices for learners, faculty, and continuous program improvement

Purpose

Simulation is an instructional modality that offers opportunities for assessment across many domains. The American College of Surgeons created the Accredited Education Institutes (AEIs) to build a community of high-quality simulation centers focused around improving surgical education and training. The goals of this project were to identify assessment methods used by AEIs, discuss how these methods align with established assessment frameworks, identify best practices, and provide guidance on best practice implementation.

Methods

The authors analyzed responses regarding learner assessment, faculty assessment, and continuous program improvement from AEI accreditations surveys using deductive qualitative analysis.

Results

Data from ninety-six centers were reviewed. Codes for each category were organized into formal and informal themes. For learner assessment, examinations and checklists identified as the most common types of formal assessment used and oral feedback as the most common type of informal assessment. For faculty assessment, written evaluations were the most common formal type and debriefs were the most common informal type. For continuous program improvement, written evaluations were the most common formal type and oral feedback was the most frequent informal type.

Discussion

The goal of assessment should be to encourage learning through feedback and to ensure the attainment of educational competencies. The data revealed a variety of assessment modalities used to accomplish this goal with AEIs frequently utilizing some of the most reliable forms of assessment. We discuss how these forms of assessment can be integrated with best practices to develop assessment portfolios for learners and faculty, performance improvement reports for faculty, and assessments of clinical impact.

Supplementary Information

The online version contains supplementary material available at 10.1007/s44186-023-00132-6.

The American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA): Validity Evidence From a Three-Year National Study

OBJECTIVE

To gather validity evidence supporting the use and interpretation of scores from the American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA) Program.

SUMMARY AND BACKGROUND DATA

ACS ERRA is an online formative assessment program developed to assess entering surgery residents' ability to make critical clinical decisions, and includes 12 clinical areas and 20 topics identified by a national panel of surgeon educators and residency program directors.

METHODS

Data from 3 national testing administrations of ACS ERRA (2018-2020) were used to gather validity evidence regarding content, response process, internal structure (reliability), relations to other variables, and consequences.

RESULTS

Over the 3 administrations, 1975 surgery residents participated from 125 distinct residency programs. Overall scores [Mean = 64% (SD = 7%)] remained consistent across the 3 years ( P = 0.670). There were no significant differences among resident characteristics (gender, age, international medical graduate status). The mean case discrimination index was 0.54 [SD = 0.15]. Kappa inter-rater reliability for scoring was 0.87; the overall test score reliability (G-coefficient) was 0.86 (Ф-coefficient = 0.83). Residents who completed residency readiness programs had higher ACS ERRA scores (66% versus 63%, Cohen's d = 0.23, P < 0.001). On average, 15% of decisions made (21/140 per test) involved potentially harmful actions. Variability in scores from graduating medical schools (7%) carried over twice as much weight than from matched residency programs (3%).

CONCLUSIONS

ACS ERRA scores provide valuable information to entering surgery residents and surgery program directors to aid in development of individual and group learning plans.

American College of Surgeons Objective Assessment of Skills in Surgery (ACS OASIS): A Formative Assessment of Junior Residents' Technical Skills

OBJECTIVE

The objective assessment of technical skills of junior residents is essential in implementing competency-based training and providing specific feedback regarding areas for improvement. An innovative assessment that can be easily implemented by training programs nationwide has been developed by expert surgeon educators under the aegis of the American College of Surgeons (ACS) Division of Education. This assessment, ACS Objective Assessment of Skills in Surgery (ACS OASIS) uses eight stations to address technical skills important for junior residents within the domains of laparoscopic appendectomy, excision of lipoma, central line placement, laparoscopic cholecystectomy, trocar placement, exploratory laparotomy, repair of enterotomy, and tube thoracostomy. The purpose of this study was to implement ACS OASIS at a number of sites to study its psychometric rigor.

DESIGN

The ACS OASIS was pre-piloted at two programs to establish feasibility and to gather information regarding implementation. Each skills station was 12 minutes long, and the faculty completed a checklist with 5 to 15 items, and a global assessment scale. The study was then repeated at three pilot sites and included 29 junior residents who were assessed by a total of 44 faculty. Psychometric data for the stations and checklists were collected and analyzed.

SETTING

The pre-pilot sites were Geisinger and University of Tennessee Knoxville.Data were gathered from pilot sites that included Wellspan Health, Duke University, and University of California Los Angeles.

RESULTS

The mean checklist score for all learners was 76% (IQR of 66%-85%). The average global rating was 3.36 on a 5-point scale with a standard deviation of 0.56. The overall cut score derived using the borderline group method was at 68% with 34% of performances requiring remediation. Using this criterion, the average number of stations that were completed by each learner without need for remediation was five.The station discrimination index ranged from 0.27 to 0.65 (all above the threshold of 0.25), demonstrating solid psychometric characteristics at the station level. The internal-consistency reliability was 0.76 with SEM of 5.8%. The inter-rater reliability (intraclass correlation) was high at 0.73 with general agreement of 79% between the two raters. The station discrimination was at 0.45 (range of 0.27 to 0.65) indicating a high level of differentiation between high and low performers. Using the generalizability theory, the G-coefficient reliability was at 0.72 with the reliability projection flattening after 8 stations. Overall, 75% to 82% the faculty and learners rated ACS OASIS as realistic and beneficial.

CONCLUSIONS

ACS OASIS is a psychometrically sound technical skills assessment tool that can provide useful information for feedback to junior residents and support efforts to remediate gaps in performance.

Resident Readiness for Senior Level Decision Making: Identifying the Domains for Formative Assessment and Feedback

OBJECTIVE

To establish expert consensus regarding the domains and topics for senior surgery residents (PGY-4) to make critical decisions and assume senior-level responsibilities, and to develop the formative American College of Surgeons Senior Resident Readiness Assessment (ACS SRRA) Program.

DESIGN

The American College of Surgeons (ACS) education leadership team conducted a focus group with surgical experts to identify the content for an assessment tool to evaluate senior residents' readiness for their increased levels of responsibility. After the focus group, national experts were recruited to develop consensus on the topics through three rounds of surveys using Delphi methodology. The Delphi participants rated topics using Likert-type scales and their comments were incorporated into subsequent rounds. Consensus was defined as ≥ 80% agreement with internal-consistency reliability (Cronbach's alpha) ≥ 0.8. In a stepwise fashion, topics that did not achieve consensus for inclusion were removed from subsequent survey rounds.

SETTING

The surveys were administered via an online questionnaire.

PARTICIPANTS

Twelve program directors and assistant program directors made up the focus group. The 39 Delphi participants represented seven different surgical subspecialties and were from diverse practice settings. The median length of experience in general surgery resident education was 20 years (IQR 14.3-30.0) with 64% of the experts being either current or past general surgery residency program directors.

RESULTS

The response rate was 100% and Cronbach's alpha was ≥ 0.9 for each round. The Delphi participants contributed a large number of comments. Of the 201 topics that were evaluated initially, 120 topics in 25 core clinical areas were included to create the final domains of ACS SRRA.

CONCLUSIONS

National consensus on the domain of the ACS SRRA has been achieved via the modified Delphi method among expert surgeon educators. ACS SRRA will identify clinical topics and areas in which each senior resident needs improvement and provide data to residents and residency programs to develop individualized learning plans. This would help in preparing the senior residents to assume their responsibilities and support their readiness for future fellowship training or surgical practice.

Artificial Intelligence Methods and Artificial Intelligence-Enabled Metrics for Surgical Education: A Multidisciplinary Consensus

BACKGROUND

Artificial intelligence (AI) methods and AI-enabled metrics hold tremendous potential to advance surgical education. Our objective was to generate consensus guidance on specific needs for AI methods and AI-enabled metrics for surgical education.

STUDY DESIGN

The study included a systematic literature search, a virtual conference, and a 3-round Delphi survey of 40 representative multidisciplinary stakeholders with domain expertise selected through purposeful sampling. The accelerated Delphi process was completed within 10 days. The survey covered overall utility, anticipated future (10-year time horizon), and applications for surgical training, assessment, and feedback. Consensus was agreement among 80% or more respondents. We coded survey questions into 11 themes and descriptively analyzed the responses.

RESULTS

The respondents included surgeons (40%), engineers (15%), affiliates of industry (27.5%), professional societies (7.5%), regulatory agencies (7.5%), and a lawyer (2.5%). The survey included 155 questions; consensus was achieved on 136 (87.7%). The panel listed 6 deliverables each for AI-enhanced learning curve analytics and surgical skill assessment. For feedback, the panel identified 10 priority deliverables spanning 2-year (n = 2), 5-year (n = 4), and 10-year (n = 4) timeframes. Within 2 years, the panel expects development of methods to recognize anatomy in images of the surgical field and to provide surgeons with performance feedback immediately after an operation. The panel also identified 5 essential that should be included in operative performance reports for surgeons.

CONCLUSIONS

The Delphi panel consensus provides a specific, bold, and forward-looking roadmap for AI methods and AI-enabled metrics for surgical education.

Coaching in Surgical Education: A Systematic Review

OBJECTIVE

The objectives of this study were to review the coaching literature to (1) characterize the criteria integral to the coaching process, specifically in surgery, and (2) describe how these criteria have been variably implemented in published studies.

BACKGROUND

Coaching is a distinct educational intervention, but within surgery the term is frequently used interchangeably with other more established terms such as teaching and mentoring.

METHODS

A systematic search was performed of the MEDLINE and Cochrane databases to identify studies that used coach/coaching as an intervention for surgeons for either technical or nontechnical skills. Study quality was evaluated using the Medical Education Research Study Quality Instrument (MERSQI).

RESULTS

A total of 2280 articles were identified and after screening by title, abstract and full text, 35 remained. Thirteen coaching criteria (a-m) were identified in 4 general categories: 1. overarching goal (a. refine performance of an existing skill set), 2. the coach (b. trusting partnership, c. avoids assessment, d. 2-way communication), the coachee (e. voluntary participation, f. self-reflection, g. goal setting, h. action plan, i. outcome evaluation), and the coach-coachee rapport (j. coaching training, k. structured coaching model, l. non-directive, m. open ended questions). Adherence to these criteria ranged from as high of 73% of studies (voluntary participation of coach and coachee) to as low as 7% (use of open-ended questions).

CONCLUSIONS

Coaching is being used inconsistently within the surgical education literature. Our hope is that with establishing criteria for coaching, future studies will implement this intervention more consistently and allow for better comparison and generalization of results.

Lived Experiences of Surgical Residents During the COVID-19 Pandemic: A Qualitative Assessment

OBJECTIVE

As the COVID-19 pandemic dynamically changes our society, it is important to consider how the pandemic has affected the training and wellness of surgical residents. Using a qualitative study of national focus groups with general surgery residents, we aim to identify common themes surrounding their personal, clinical, and educational experiences that could be used to inform practice and policy for future pandemics and disasters.

DESIGN

Six 90-minute focus groups were conducted by a trained qualitative researcher who elicited responses on six predetermined topics. De-identified transcripts and audio recordings were later analyzed by two independent researchers who organized responses to each topic into themes.

SETTING

Focus groups were conducted virtually and anonymously.

PARTICIPANTS

General surgery residents were recruited from across the country. Demographic information of potential participants was coded, and subjects were randomly selected to ensure a diverse group of participants.

RESULTS

The impact of the COVID-19 pandemic on residents' clinical, educational, and personal experiences varied depending on the institutional response of the program and the burden of COVID-19 cases geographically. Many successes were identified: the use of telehealth and virtual didactics, an increased sense of camaraderie amongst residents, and flexibility in scheduling. Many challenges were also identified: uncertainty at work regarding personal protective equipment and scheduling, decreased case volume and educational opportunities, and emotional trauma and burnout associated with the pandemic.

CONCLUSIONS

These data gathered from our qualitative study highlight a clear, urgent need for thoughtful institutional planning and policies for the remainder of this and future pandemics. Residency programs must ensure a balanced training program for surgical residents as they attempt to master the skills of their craft while also serving as employed health care providers in a pandemic. Furthermore, a focus on wellness, in addition to clinical competency and education, is vital to resident resilience and success in a pandemic setting.

Preceptoring, proctoring, mentoring, and coaching in surgery

Contemporary models of surgical education that are founded on sound educational theories and constructs need to be used across the continuum of professional development of surgeons starting with the first day of medical school, through surgery residency and fellowship training, to the last day of surgical practice. The highly learner-centered and individually-focused special interventions of preceptoring, proctoring, mentoring, and coaching should be linked to innovative competency-based education models to address the educational needs of learners at all levels, and especially of surgeons in practice to continually improve their knowledge, skills, and performance, with the aspirational goal of achieving expertise and mastery. Each of these interventions is distinct with its own unique characteristics, applications, and anticipated impact, which must be clearly recognized for the interventions to be used most effectively. Broad acceptance and adoption of the aforementioned special interventions require recognition of the value each brings to the learner and the educational program. Professional organizations should play a key role in designing innovative educational programs that include these interventions, supporting their integration into surgical education and surgical practice, and influencing changes in the cultures in surgery to facilitate broad adoption of these interventions.

Defining the need for faculty development in assessment

BACKGROUND

High-quality workplace-based assessments are essential for competency-based surgical education. We explored education leaders' perceptions regarding faculty competence in assessment.

METHODS

Surgical education leaders were surveyed regarding which areas faculty needed improvement, and knowledge of assessment tools. Respondents were queried on specific skills regarding (a)importance in resident/medical student education (b)competence of faculty in assessment and feedback.

RESULTS

Surveys (n = 636) were emailed, 103 responded most faculty needed improvement in: verbal (86%) and written (83%) feedback, assessing operative skill (49%) and preparation for procedures (50%). Cholecystectomy, trauma laparotomy, inguinal herniorrhaphy were "very-extremely important" in resident education (99%), but 21-24% thought faculty "moderately to not-at-all" competent in assessment. This gap was larger for non-technical skills. Regarding assessment tools, 56% used OSATS, 49% Zwisch; most were unfamiliar with all non-technical tools.

SUMMARY

These data demonstrate a significant perceived gap in competence of faculty in assessment and feedback, and unfamiliarity with assessment tools. This can inform faculty development to support competency-based surgical education.

Impact of the COVID-19 pandemic on surgical trainee education and well-being spring 2020-winter 2020: A path forward

Background

The time course and longitudinal impact of the COVID -19 pandemic on surgical education(SE) and learner well-being (LWB)is unknown.

Material and methods

Check-in surveys were distributed to Surgery Program Directors and Department Chairs, including general surgery and surgical specialties, in the summer and winter of 2020 and compared to a survey from spring 2020. Statistical associations for items with self-reported ACGME Stage and the survey period were assessed using categorical analysis.

Results

Stage 3 institutions were reported in spring (30%), summer (4%) [p < 0.0001] and increased in the winter (18%). Severe disruption (SD) was stage dependent (Stage 3; 45% (83/184) vs. Stages 1 and 2; 26% (206/801)[p < 0.0001]). This lessened in the winter (23%) vs. spring (32%) p = 0.02. LWB severe disruption was similar in spring 27%, summer 22%, winter 25% and was associated with Stage 3.

Conclusions

Steps taken during the pandemic reduced SD but did not improve LWB. Systemic efforts are needed to protect learners and combat isolation pervasive in a pandemic.

Gathering Validity Evidence to Adapt the Non-technical Skills for Surgeons (NOTSS) Assessment Tool to the United States Context

BACKGROUND

Nontechnical skills are of increasing focus for safe and effective performance in the operating room. Assessment tools have been developed in Europe, Africa, and Asia but not adapted to the unique aspects of surgical delivery in the United States. Our objective was to use the Non-Technical Skills for Surgeons (NOTSS) assessment tool as a basis to establish consensus on essential nontechnical skills for surgical trainees and practicing surgeons in the U.S surgical context.

STUDY DESIGN

A mixed-methods research design was used in the form of a modified Delphi process to build consensus on essential NOTSS. A panel of surgical experts from hospitals across the U.S used this iterative process in 4 rounds to generate, rate, and classify behaviors. The primary outcome was consensus on behaviors as being essential for surgeons to achieve the best patient outcomes in the operating room, with a median rating of ≥6 on a 7-point scale for inclusion.

RESULTS

A total of 10 surgical experts participated. One hundred and thirty eight behaviors were generated in Round 1, and reduced to 100 behaviors in Rounds 2 and 3 based on application of inclusion criteria. The final skill list consisted of behaviors in Situation Awareness (n = 26), Decision Making (n = 18), Teamwork (n = 25), and Leadership (n = 31). No additional NOTSS categories or elements emerged from the analysis. In Round 4, all 100 behaviors were successfully grouped into 12 nontechnical skills elements. Labels and definitions were reworded to reflect the U.S. context, and an appropriate assessment scale was selected.

CONCLUSIONS

A panel of surgical experts from across the U.S. reached consensus on the essential NOTSS to achieve the best patient outcomes in the operating room. These behaviors form an empirical basis for the first context-specific nontechnical skills assessment and training tool for practicing surgeons in the U.S.

Impact of the COVID-19 Pandemic on Surgical Training and Learner Well-Being: Report of a Survey of General Surgery and Other Surgical Specialty Educators

BACKGROUND

The COVID-19 pandemic disrupted the delivery of surgical services. The purpose of this communication was to report the impact of the pandemic on surgical training and learner well-being and to document adaptations made by surgery departments.

STUDY DESIGN

A 37-item survey was distributed to educational leaders in general surgery and other surgical specialty training programs. It included both closed- and open-ended questions and the self-reported stages of GME during the COVID-19 pandemic, as defined by the ACGME. Statistical associations for items with stage were assessed using categorical analysis.

RESULTS

The response rate was 21% (472 of 2,196). US stage distribution (n = 447) was as follows: stage 1, 22%; stage 2, 48%; and stage 3, 30%. Impact on clinical education significantly increased by stage, with severe reductions in nonemergency operations (73% and 86% vs 98%) and emergency operations (8% and 16% vs 34%). Variable effects were reported on minimal expected case numbers across all stages. Reductions were reported in outpatient experience (83%), in-hospital experience (70%), and outside rotations (57%). Increases in ICU rotations were reported with advancing stage (7% and 13% vs 37%). Severity of impact on didactic education increased with stage (14% and 30% vs 46%). Virtual conferences were adopted by 97% across all stages. Severity of impact on learner well-being increased by stage-physical safety (6% and 9% vs 31%), physical health (0% and 7% vs 17%), and emotional health (11% and 24% vs 42%). Regardless of stage, most but not all made adaptations to support trainees' well-being.

CONCLUSIONS

The pandemic adversely impacted surgical training and the well-being of learners across all surgical specialties proportional to increasing ACGME stage. There is a need to develop education disaster plans to support technical competency and learner well-being. Careful assessment for program advancement will also be necessary. The experience during this pandemic shows that virtual learning and telemedicine will have a considerable impact on the future of surgical education.

Overarching themes from ACS-AEI accreditation survey best practices 2011-2019

The American College of Surgeons launched the Accredited Education Institutes (ACS-AEIs) in 2005 to create a network of simulation centers to offer the highest-quality surgical education and training and to pursue cutting-edge scholarship. Specific standards and criteria were developed to accredit simulation centers, and decisions made based on the application, site surveyor visit, and ACS-AEI Accreditation Committee review. In addition to granting accreditation and providing recommendations for improvement, site surveyors and the Accreditation Committee identified best practices defined as areas far exceeding the accreditation standards or novel methods of advancing high-quality, impactful education. Best practices were compiled into a list starting in 2011 for dissemination to all members of the ACS-AEI Consortium through on-line videos, newsletters, and workshops at the Annual ACS Surgical Simulation Summit. Using thematic analysis, the authors classified common themes from the 337 best practices identified during 247 accreditation reviews between June 2011 and June 2019. As best practices were extracted, the authors compiled them into a single database. Eight overarching themes were identified, including approaches to faculty development, scholarly activity, development of curricula, use of resources, delivery of educational content, assessment of learners, and collaboration between centers. The authors discuss themes in the context of their practical relevance and highlight strategies used at different centers, with an emphasis on the importance of each theme in developing a successful simulation program.

Assessment of the Non-Technical Skills for Surgeons (NOTSS) framework in the USA

Background

Surgeons' non-technical skills are important for patient safety. The Non-Technical Skills for Surgeons assessment tool was developed in the UK and recently adapted to the US surgical context (NOTSS-US). The aim of this study was to evaluate the reliability and distribution of non-technical skill ratings given by attending (consultant) surgeons who underwent brief online training.

Methods

Attending surgeons across six specialties at a large US academic medical centre underwent a 10-min online training, then rated 60-s standardized videos of simulated operations. Intraclass correlation coefficient (ICC), and mean(s.d.) values for NOTSS-US ratings were determined for each non-technical skill category (score range 1–5, where 1 indicates poor, 3 average and 5 excellent) and for total NOTSS-US score (range 4–20; sum of 4 category scores). Outcomes were adjusted for rater characteristics including sex, specialty and clinical experience.

Results

A total of 8889 ratings were submitted by 81 surgeon raters on 30 simulated intraoperative videos. The mean(s.d.) total NOTSS-US score for all videos was 9·5(4·8) of 20. The within-video ICC for total NOTSS-US score was 0·64 (95 per cent c.i. 0·57 to 0·70). For individual non-technical skill categories, the ICC was highest for social skills (communication/teamwork: 0·63, 95 per cent c.i. 0·56 to 0·71; leadership: 0·64, 0·55 to 0·72) and lowest for cognitive skills (situation awareness: 0·54, 0·45 to 0·62; decision-making: 0·50, 0·41 to 0·59). Women gave higher total NOTSS-US scores than men (adjusted mean difference 0·93, 95 per cent c.i. 0·44 to 1·43; P = 0·001).

Conclusion

After brief online training, the inter-rater reliability of the NOTSS-US assessment tool achieved moderate strength among trained surgeons rating simulated intraoperative videos.

Development and evaluation of the General Surgery Objective Structured Assessment of Technical Skill (GOSATS)

Background

Technical skill acquisition is important in surgery specialty training. Despite an emphasis on competency-based training, few tools are currently available for direct technical skills assessment at the completion of training. The aim of this study was to develop and validate a simulated technical skill examination for graduating (postgraduate year (PGY)5) general surgery trainees.

Methods

A simulated eight-station, procedure-based general surgery technical skills examination was developed. Board-certified general surgeons blinded to the level of training rated performance of PGY3 and PGY5 trainees by means of validated scoring. Cronbach's α was used to calculate reliability indices, and a conjunctive model to set a pass score with borderline regression methodology. Subkoviak methodology was employed to assess the reliability of the pass–fail decision. The relationship between passing the examination and PGY level was evaluated using χ2 analysis.

Results

Ten PGY3 and nine PGY5 trainees were included. Interstation reliability was 0·66, and inter-rater reliability for three stations was 0·92, 0·97 and 0·76. A pass score of 176·8 of 280 (63·1 per cent) was set. The pass rate for PGY5 trainees was 78 per cent (7 of 9), compared with 30 per cent (3 of 10) for PGY3 trainees. Reliability of the pass–fail decision had an agreement coefficient of 0·88. Graduating trainees were significantly more likely to pass the examination than PGY3 trainees (χ2 = 4·34, P = 0·037).

Conclusion

A summative general surgery technical skills examination was developed with reliability indices within the range needed for high-stakes assessments. Further evaluation is required before the examination can be used in decisions regarding certification.

Needs Assessment for an American College of Surgeons Certificate in Applied Surgical Education Leadership (CASEL)

OBJECTIVE

Surgeon educators in departments of surgery play key roles in leading and advancing surgical education. Their activities include ensuring sound curricula and evaluation systems, monitoring education resources, overseeing faculty development, and providing mentorship. For more than 25 years, the American College of Surgeons (ACS) has offered a comprehensive "Surgeons as Educators" (SAE) course to address fundamental topics in surgical education. This study aims to identify future career needs of SAE graduates to inform the development of an American College of Surgeons Certificate in Applied Surgical Education Leadership program.

DESIGN

An IRB exempt, anonymous electronic survey was developed to determine educational roles, career aspirations, and needs of SAE Graduates.

SETTING AND PARTICIPANTS

Participants included all 763 1993-2016 SAE graduates.

RESULTS

One hundred and thirty-five responses were received from 600 (22.5%) graduates with valid email addresses. Sixty (45%) respondents completed the SAE Course > 5 years prior to the study (M5YRS) and 75 (55%) within the last 5 years (L5YRS). L5YRS respondents were less likely to be full professors (8% vs. 44%) or to serve as program directors (32% vs. 57%), and more likely to be associate program directors (25% vs. 17%) or clerkship directors (40% vs. 18%). High percentages of both L5YRS and M5YRS reported not pursuing additional educational opportunities post-SAE due to time and fiscal constraints. One-fifth of respondents were unaware of additional opportunities and 19% of M5YRS versus 6% of L5YRS stated that existing programs did not meet their needs. Overall improving skills as educational leaders, developing faculty development programs, and conducting educational research were noted as priorities for future development. Differences were observed between the L5YRS and M5YRS groups. The dominant preferences for course format were full-time face-to-face (41%) or a combination of full-time face-to-face with online modules (24%). The most important considerations in deciding to pursue a certificate course were course content, and interest in advancing career and time constraints.

CONCLUSIONS

An SAE graduate survey has confirmed the need for additional formal training in surgical education leadership in order to permit surgeon educators meet the demands of the changing landscape of surgical education. The needs of early career faculty may differ from those of more senior surgeon educators.

Educational interventions aimed at the transition from surgical training to surgical practice

The transition from surgical training to surgical practice is a critical juncture in the career progression of surgeons. This period is associated with myriad challenges that need to be addressed through specific educational interventions to ensure delivery of safe care to patients and to support the career aspirations of junior surgeons entering the practice environment. These interventions should be based on principles of contemporary surgical education and training, and focus on the needs of surgery residents and junior surgeons entering practice. The specific systems of patient care in which the junior surgeons will work should also be considered while planning and implementing such educational interventions. Senior surgeon colleagues within these systems should play key roles in supporting the junior surgeons entering practice, and may require special training to serve as effective mentors, preceptors, and coaches. Professional societies should play a key role in establishing national standards regarding the educational programs aimed at this transition and develop programs to complement local efforts to address various needs. The American College of Surgeons Division of Education has developed a spectrum of innovative programs that are aimed at this important transition.

The value proposition of simulation-based education

Simulation has become an integral part of physician education, and abundant evidence confirms that simulation-based education improves learners' skills and behaviors and is associated with improved patient outcomes. The resources required to implement simulation-based education, however, have led some stakeholders to question the overall value proposition of simulation-based education. This paper summarizes the information from a special panel on this topic and defines research priorities for the field. Future work should focus on both outcomes and costs, with robust measurement of resource investments, provider performance (in both simulation and real settings), patient outcomes, and impact on the health care organization. Increased attention to training practicing clinicians and health care teams is also essential. Clarifying the value proposition of simulation-based education will require a major national effort with funding from multiple sponsors and active engagement of a variety of stakeholders.

"Taking Training to the Next Level": The American College of Surgeons Committee on Residency Training Survey

OBJECTIVE

The American College of Surgeons (ACS) appointed a committee of leaders from the ACS, Association of Program Directors in Surgery, Accreditation Council for Graduate Medical Education, and American Board of Surgery to define key challenges facing surgery resident training programs and to explore solutions. The committee wanted to solicit the perspectives of surgery resident program directors (PDs) given their pivotal role in residency training.

DESIGN

Two surveys were developed, pilot tested, and administered to PDs following Institutional Review Board approval. PDs from 247 Accreditation Council for Graduate Medical Education-accredited general surgery programs were randomized to receive 1 of the 2 surveys. Bias analyses were conducted, and adjusted Pearson χ² tests were used to test for differences in response patterns by program type and size.

SETTING

All accredited general surgery programs in the United States were included in the sampling frame of the survey; 10 programs with initial or withdrawn accreditation were excluded from the sampling frame.

PARTICIPANTS

A total of 135 PDs responded, resulting in a 54.7% response rate (Survey A: n = 67 and Survey B: n = 68). The respondent sample was determined to be representative of program type and size.

RESULTS

Nearly 52% of PD responses were from university-based programs, and 41% had over 6 residents per graduating cohort. More than 61% of PDs reported that, compared to 10 years ago, both entering and graduating residents are less prepared in technical skills. PDs expressed significant concerns regarding the effect of duty-hour restrictions on the overall preparation of graduating residents (61%) and quality of patient care (57%). The current 5-year training structure was viewed as needing a significant or extensive increase in opportunities for resident autonomy (63%), and the greatest barriers to resident autonomy were viewed to be patient preferences not to be cared for by residents (68%), liability concerns (68%), and Centers for Medicare and Medicaid Services regulations (65%). Although 64% of PDs believe that moderate or significant changes are needed in the current structure of residency training, 35% believe that no changes in the structure are needed. When asked for their 1 best recommendation regarding the structure of surgical residency, only 22% of PDs selected retaining the current 5-year structure. The greatest percentage of PDs (28%) selected the "4 + 2" model as their 1 best recommendation for the structure to be used. In the area of faculty development, 56% of PDs supported a significant or extensive increase in Train the Teacher programs, and 41% supported a significant or extensive increase in faculty certification in education.

CONCLUSIONS

Information regarding the valuable perspectives of PDs gathered through these surveys should help in implementing important changes in residency training and faculty development. These efforts will need to be pursued collaboratively with involvement of key stakeholders, including the organizations represented on this ACS committee.

See More, Do More, Teach More: Surgical Resident Autonomy and the Transition to Independent Practice

The graduate medical education system is tasked with training competent and autonomous health care providers while also improving patient safety, delivering more efficient care, and cutting costs. Concerns about resident autonomy and preparation for independent and safe practice appear to be growing, and the field of surgery faces unique challenges in preparing graduates for independent practice. Multiple factors are contributing to an erosion of resident autonomy and decreased operative experience, including differing views of autonomy, financial forces, duty hours regulations, and diverse community health care needs. Identifying these barriers and developing solutions to overcome them are vital first steps in reversing the trend of diminishing autonomy in surgical residency training. This Commentary highlights the problem of decreasing autonomy, outlines specific threats to resident autonomy, and discusses potential solutions to mitigate their impact on the successful transition to independent practice.

The value proposition of simulation

BACKGROUND

Simulation has been shown to improve trainee performance at the bedside and in the operating room. As the use of simulation-based training is expanded to address a host of health care challenges, its added value needs to be clearly demonstrated. Demonstrable improvements will support the expansion of infrastructure, staff, and programs within existing simulation facilities as well as the establishment of new facilities to meet growing needs and demands. Thus, organizational and institutional leaders, faculty members, and other stakeholders can be assured of the best use of existing resources and can be persuaded to make greater investments in simulation-based training for the future.

METHODS

A multidisciplinary panel was convened during the 8th Annual Meeting of the Consortium of the American College of Surgeons-Accredited Education Institutes (Simulation Centers) in March 2015 to discuss the added value of simulation-based training. Panelists shared the ways in which the value of simulation was demonstrated at their institutions.

CONCLUSION

The value of simulation-based training was considered and described in terms of educational impact, patient care outcomes, and costs.

Education and Training to Address Specific Needs During the Career Progression of Surgeons

Surgeons have specific education and training needs as they enter practice, progress through the core period of active practice, and then as they wind down their clinical work before retirement. These transitions and the career progression process, combined with the dynamic health care environment, present specific opportunities for innovative education and training based on practice-based learning and improvement, and continuous professional development methods. Cutting-edge technologies, blended models, simulation, mentoring, preceptoring, and integrated approaches can play critical roles in supporting surgeons as they provide the best surgical care throughout various phases of their careers.

Continuing Professional Development in the Twenty-First Century

The critical role of continuing professional development (CPD) in supporting delivery of patient care of the highest quality and safety is receiving significant attention in the current era of monumental change. CPD is essential in efforts to ensure effectiveness of new models of health care delivery, improve outcomes and value in health care, address external regulations, and foster patient engagement. The unique features of CPD; the use of special mastery-based teaching, learning, and assessment methods, and other special interventions to promote excellence; and direct involvement of a variety of key stakeholders differentiate CPD from undergraduate medical education and graduate medical education. The needs of procedural specialties relating to CPD are different from those of primary care disciplines and require special attention for the greatest impact. Simulation-based education and training can be very useful in CPD aimed at improving outcomes and promoting patient safety. Preceptoring, proctoring, mentoring, and coaching should be used routinely to address specific needs in CPD. Distinct CPD strategies are necessary for retraining, reentry, and remediation. Participation in CPD programs can be encouraged by leveraging the joy of learning, which should drive physicians and surgeons to strive continually to be the best in their professional work.

Transition to surgical residency: a multi-institutional study of perceived intern preparedness and the effect of a formal residency preparatory course in the fourth year of medical school

PURPOSE

To evaluate interns' perceived preparedness for defined surgical residency responsibilities and to determine whether fourth-year medical school (M4) preparatory courses ("bootcamps") facilitate transition to internship.

METHOD

The authors conducted a multi-institutional, mixed-methods study (June 2009) evaluating interns from 11 U.S. and Canadian surgery residency programs. Interns completed structured surveys and answered open-ended reflective questions about their preparedness for their surgery internship. Analyses include t tests comparing ratings of interns who had and had not participated in formal internship preparation programs. The authors calculated Cohen d for effect size and used grounded theory to identify themes in the interns' reflections.

RESULTS

Of 221 eligible interns, 158 (71.5%) participated. Interns self-reported only moderate preparation for most defined care responsibilities in the medical knowledge and patient care domains but, overall, felt well prepared in the professionalism, interpersonal communication, practice-based learning, and systems-based practice domains. Interns who participated in M4 preparatory curricula had higher self-assessed ratings of surgical technical skills, professionalism, interpersonal communication skills, and overall preparation, at statistically significant levels (P < .05) with medium effect sizes. Themes identified in interns' characterizations of their greatest internship challenges included anxiety or lack of preparation related to performance of technical skills or procedures, managing simultaneous demands, being first responders for critically ill patients, clinical management of predictable postoperative conditions, and difficult communications.

CONCLUSIONS

Entering surgical residency, interns report not feeling prepared to fulfill common clinical and professional responsibilities. As M4 curricula may enhance preparation, programs facilitating transition to residency should be developed and evaluated.

Emerging Techniques in Minimally Invasive Surgery. Pros and Cons

New trends have emerged regarding the best minimally invasive access approaches to perform gastrointestinal surgery. However, these newer approaches are seen critically by those who demand a more strict assessment of outcomes and safety. An international panel of expert gathered at the 2014 American College of Surgeons Meeting with the goal of providing an evidence-based understanding of the real value of these approaches in gastrointestinal surgery. The panel has compared the efficacy and safety of most established approaches to gastrointestinal diseases to those of new treatment modalities: peroral esophageal myotomy vs. laparoscopic myotomy for achalasia, transgastric vs. transvaginal approach, and single-incision vs. multi-port access minimally invasive surgery. The panel found that (1) the outcome of these new approaches was not superior to that of established surgical procedures; (2) the new approaches are generally performed in few highly specialized centers; and (3) transgastric and transvaginal approaches might be safe and feasible in very experienced hands, but cost, training, operative time, and tools seem to limit their application for the treatment of common procedures such as cholecystectomy and appendectomy. Because the expected advantages of new approaches have yet to be proven in controlled trials, new approaches should be considered for adoption into practice only after thorough analyses of their efficacy and effectiveness and appropriate training.

Expanding simulation-based education through institution-wide initiatives: A blueprint for success

BACKGROUND

The Consortium of American College of Surgeons Accredited Education Institutes (ACS-AEIs) was created to promote patient safety through the use of simulation, develop innovative education and training, advance technologies, identify best practices, and encourage research and collaboration.

METHODS

During the seventh annual meeting of the consortium, leaders from across the consortium who have developed institution-wide simulation centers were invited to participate in a panel to discuss their experiences and the lessons learned.

CONCLUSION

These discussions resulted in definition of 5 key areas that need to be addressed effectively to support efforts of the ACS-AEIs.

Gearing up for milestones in surgery: Will simulation play a role?

BACKGROUND

The Consortium of American College of Surgeons-Accredited Education Institutes was created to promote patient safety through the use of simulation, develop new education and technologies, identify best practices, and encourage research and collaboration.

METHODS

During the 7th Annual Meeting of the Consortium, leaders from a variety of specialties discussed how simulation is playing a role in the assessment of resident performance within the context of the Milestones of the Accreditation Council for Graduate Medical Education as part of the Next Accreditation System.

CONCLUSION

This report presents experiences from several viewpoints and supports the utility of simulation for this purpose.