The Science of Selection: Using Best Practices From Industry to Improve Success in Surgery Training

Performance in Irish Selection and Future Performance in Surgical Training

Importance

Selection processes for surgical training should aim to identify candidates who will become competent independent practitioners and should aspire to high standards of reliability and validity.

Objective

To determine the association between measured candidate factors at the time of an Irish selection and assessment outcomes in surgical training, examined via rate of progression to Higher Specialist Training (HST), attrition rates, and performance as assessed through a multimodal framework of workplace-based and simulation-based assessments.

Design, Setting, and Participants

This retrospective observational cohort study included data from all successful applicants to the Royal College of Surgeons in Ireland (RCSI) national Core Surgical Training (CST) program. Participants included all trainees recruited to dedicated postgraduate surgical training from 2016 to 2020. These data were analyzed from July 11, 2016, through July 10, 2022.

Exposures

Selection decisions were based on a composite score that was derived from technical aptitude assessments, undergraduate academic performance, and a 4-station multiple mini-interview.

Main outcomes and measures

Assessment data, attrition rates, and rates of progression to HST were recorded for each trainee. CST performance was assessed using workplace-based and simulation-based technical and nontechnical skill assessments. Potential associations between selection and assessment measures were explored using Pearson correlation, logistic regression, and multiple linear-regression analyses.

Results

Data were available for 303 trainees. Composite scores were positively associated with progression to HST (odds ratio [OR], 1.09; 95% CI, 1.05-1.13). There was a weak positive correlation, ranging from 0.23 to 0.34, between scores and performance across all CST assessments. Multivariable linear regression analysis showed technical aptitude scores at application were associated with future operative performance assessment scores, both in the workplace (β = 0.31; 95% CI, 0.14-0.48) and simulated environments (β = 0.57; 95% CI, 0.33-0.81). There was evidence that the interpersonal skills interview station was associated with future performance in simulated communication skill assessments (β = 0.55; 95% CI, 0.22-0.87).

Conclusions and Relevance

In this study, performance at the time of Irish national selection, measured across technical and nontechnical domains in a multimodal fashion, was associated with future performance in the workplace and in simulated environments. Future studies will be required to explore the consequential validity of selection, including potential unintended effects of selection and ranking on candidate performance.

Residency Selection in Otolaryngology: Past, Present, & Future

OBJECTIVES

To examine the otolaryngology residency selection process, including past experiences based on the medical literature and survey analysis of our present practices to generate recommendations for future selection system design.

METHODS

A mixed-methods study, including a scoping review and a cross-sectional survey, was completed. Four databases were assessed for articles on otolaryngology residency selection published from January 1, 2016 through December 31, 2020. A 36-question survey was developed and distributed to 114 otolaryngology program directors. Descriptive and thematic analysis was performed.

RESULTS

Ultimately, 67 of 168 articles underwent data abstraction and assessment. Three themes surfaced during the analysis: effectiveness, efficiency, and equity. Regarding the survey, there were 62 participants (54.4% response rate). The three most important goals for the selection process were: (1) to fit the program culture, (2) to make good colleagues, and (3) to contribute to the program's diversity. The three biggest 'pain points' were as follows: (1) Large volume of applications, (2) Lack of reliable information about personal characteristics, and (3) Lack of reliable information about a genuine interest in the program.

CONCLUSIONS

Within this study, the depth and breadth of the literature on otolaryngology residency selection have been synthesized. Additionally, baseline data on selection practices within our specialty has been captured. With an informed understanding of our past and present, we can look to the future. Built upon the principles of person-environment fit theory, our proposed framework can guide research and policy discussions regarding the design of selection systems in otolaryngology, as we work to achieve more effective, efficient, and equitable outcomes.

LEVEL OF EVIDENCE

N/A Laryngoscope, 133:2929-2941, 2023.

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values

PURPOSE

The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda.

METHOD

Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion.

RESULTS

Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process.

CONCLUSIONS

This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

Baseline Assessments of Psychomotor Aptitude Are Associated With the Future Operative Performance of Surgical Trainees: A Prospective Observational Cohort Study

OBJECTIVE

This study sought to investigate the association between validated psychomotor ability tests and future in-theater and simulated operative performance.

BACKGROUND

Assessments of visuospatial ability, perceptual ability, and manual dexterity correlate with simulated operative performance. Data showing the predictive value of such assessments in relation to future performance in the workplace is lacking.

METHODS

Core surgical residents in Ireland recruited from 2016 to 2019 participated in assessments of baseline perceptual, visuospatial, and psychomotor ability; Pictorial Surface Orientation (PicSOr) testing, digital visuospatial ability testing, and manual dexterity testing. Operative performance was prospectively assessed using the in-theater Supervised Structured Assessment of Operative Performance (SSAOP) tool, and simulation-based Operative Surgical Skill (OSS) assessments performed over a 2-year core training period. SSAOP assessments were scored using a 15-point checklist and a global 5-point operative performance score. OSS assessments were scored using procedure-specific checklists. Univariate correlations and multiple linear regression analyses were used to explore the association between fundamental ability measures and operative performance.

RESULTS

A total of 242 residents completed baseline psychomotor ability assessments. Aggregated fundamental ability scores were associated with performance in submitted workplace-based SSAOP assessments using the Total Checklist score ( P =0.002) and Overall Performance scores ( P =0.002), independent of operative experience, and undergraduate centile scores. Aggregated ability scores were also positively associated with simulation-based OSS assessment scores on multivariable analysis ( P =0.03).

CONCLUSION

This study indicates that visuospatial, psychomotor, and perceptual ability testing scores are associated with the future operative performance of surgical residents.

The Undergraduate to Graduate Medical Education Transition as a Systems Problem: A Root Cause Analysis

The transition from undergraduate medical education (UME) to graduate medical education (GME) constitutes a complex system with important implications for learner progression and patient safety. The transition is currently dysfunctional, requiring students and residency programs to spend significant time, money, and energy on the process. Applications and interviews continue to increase despite stable match rates. Although many in the medical community acknowledge the problems with the UME-GME transition and learners have called for prompt action to address these concerns, the underlying causes are complex and have defied easy fixes. This article describes the work of the Coalition for Physician Accountability's Undergraduate Medical Education to Graduate Medical Education Review Committee (UGRC) to apply a quality improvement approach and systems thinking to explore the underlying causes of dysfunction in the UME-GME transition. The UGRC performed a root cause analysis using the 5 whys and an Ishikawa (or fishbone) diagram to deeply explore problems in the UME-GME transition. The root causes of problems identified include culture, costs and limited resources, bias, systems, lack of standards, and lack of alignment. Using the principles of systems thinking (components, connections, and purpose), the UGRC considered interactions among the root causes and developed recommendations to improve the UME-GME transition. Several of the UGRC's recommendations stemming from this work are explained. Sustained monitoring will be necessary to ensure interventions move the process forward to better serve applicants, programs, and the public good.

National trends in distribution of underrepresented minorities within United States general surgery residency programs: A longitudinal panel study

BACKGROUND

Cultural affinity with a provider improves satisfactoriness of healthcare. We examined 2005-2019 trends in racial/ethnic diversity/inclusion within general surgery residency programs.

METHODS

We triangulated 2005-2019 race/ethnicity data from Association of American Medical Colleges surveys of 4th-year medical students, the Electronic Residency Application Service, and Accreditation Council for Graduate Medical Education-affiliated general surgery residencies. Temporal trends in minority representation were tested for significance.

RESULTS

Underrepresented racial/ethnic minorities in medicine (URiMs) increased among graduating MDs from 7.6% in 2005 to 11.8% in 2019 (p < 0.0001), as did their proportion among surgery residency applicants during 2005-2019 (p < 0.0001). However, proportions of URiMs among general surgery residents (≈8.5%), and of programs without URiMs (≈18.8%), stagnated.

CONCLUSIONS

Growing URiM proportions among medical school graduates and surgery residency applicants did not improve URiM representation among surgery trainees nor shrink the percentage of programs without URiMs. Deeper research into motivators underlying URiMs' residency program preferences is warranted.

Black Physicians' Experiences with Anti-Black Racism in Healthcare Systems Explored Through An Attraction-Selection-Attrition Lens

Anti-Black racism is a specific form of racism directed at Black people. In healthcare, there are poignant examples of anti-Black racism in the recruitment, selection, and retention stages of the job cycle. Research shows that anti-Black racism is associated with inequitable work outcomes and the under-representation of Black physicians. However, empirical findings are scattered with no organizing framework to consolidate these findings. To add to the literature, in this paper we present the attraction-selection-attrition (ASA) model (Schneider, 1987) as an organizing framework to discuss Black physicians' experiences with anti-Black racism and discrimination throughout their careers. We draw from previous literature to highlight specific experiences of Black physicians at each stage of the job cycle (i.e., attraction, selection, retention), and we offer considerations on how practitioners can mitigate anti-Black racism throughout the job cycle. In the wake of COVID-19 and highly publicized social justice movements, healthcare systems are seeking ways to increase the recruitment, selection, and retention of Black physicians to ensure health equity. We believe this guide will be valuable to practitioners, leaders, researchers, and program directions seeking to advance diversity, equity, and inclusion of Black physicians in their healthcare systems. We conclude by providing practical implications and directions for future research.

Visuospatial ability is associated to 2D laparoscopic simulator performance amongst surgical residents

Background

The technical skills of a surgeon influence surgical outcome. Testing technical aptitude at point of recruitment of surgical residents is only conducted in a few countries. This study investigated the impact of visuospatial ability (VSA), background factors, and manual dexterity on performance in two different laparoscopic surgical simulators amongst applicants and 1st year surgical residents.

Method

Applicants from general surgery, pediatric surgery, and urology were included from seven hospitals in Sweden between 2017 and 2021. Some 73 applicants were invited and 50 completed. Participants filled out a background form, and were tested for manual dexterity, and visuospatial ability. Two laparoscopic simulators were used, one 2D video box trainer and one 3D Virtual Reality Simulator.

Results

A significant association was found between the visuospatial ability test and 2D video box laparoscopic performance (95 % CI: 1.028-1.2, p-value <0.01). For every point on the visuospatial test the odds of accomplishing the task increased by 11 %. No association was found between VSA and performance in a laparoscopic VR simulator using 3D vision. No other background factors were associated with performance in the two laparoscopic simulators.

Conclusion

Visuospatial ability in applicants to surgical residency positions is associated to performance in a 2D video box trainer. Knowledge of a resident's visuospatial ability can be used to tailor individualized laparoscopic training programs, and in the future might function as a selection tool concerning laparoscopic ability.

Key message

Visuospatial ability differs greatly amongst applicants for surgical residency and is associated to laparoscopic simulator performance. Testing applicants' visuospatial ability could possibly be used to tailor individualized laparoscopic training programs or in the future as a selection tool concerning laparoscopic ability.

Practical Strategies for Underrepresented Minority Recruitment in General Surgery Residency

OBJECTIVE

To analyze the effects of diversification efforts on underrepresented minority in medicine (URiM) resident recruitment in general surgery residency at a single large southeastern United States academic institution with five categorical positions.

METHODS

A retrospective review of applications from the 2016 to 2022 ACGME match cycles was conducted. In an effort to diversify resident recruitment, multiple new strategies were enacted in 2021. URiM candidates were identified via a more laborious review of individual applications to the program. In addition, a holistic review process was conducted, URiM faculty and residents were prominently featured, previous underperformance in diversity was openly addressed, and URiM applicants were contacted with follow up emails. Cohorts pre- and post-implementation of these strategies were analyzed. The proportion of URiM applicants invited, interviewed, ranked, and matched were compared.

RESULTS

Pre-intervention during the 2016to 20 match cycles, URiM candidates represented 4% of total applicants invited. Post-intervention during the 2021to 22 match cycles, URiM candidates represented 27% of total applicants invited. Over the past 5 years under the present program director, 1 URiM resident of 24 (4%) matched into the categorical program. Over the past 15 years under the direction of 3 program directors, a total of 6 out of 69 matched residents (9%) identified as URiM. Post intervention, the program matched on average 30% of its incoming categorical class from URiM candidates.

CONCLUSION

Recruitment and selection of diverse medical school applicants is an ongoing concern of general surgery residency program directors. Historically, URiM candidates are underrepresented in applicants selected for interview. Interventions aimed at increasing the matriculation of URiM include concentrated efforts to identify more URiM candidates for interview. However, importantly, transparency of ongoing diversity efforts and diversifying both faculty and trainees involved in the selection process may also improve general surgery URiM recruitment.

Identifying Potential Attrition during the Residency Applicant Screening Process Using a Situational Judgment Test

OBJECTIVE

As USMLE Step 1 moves to pass/fail, residency programs are seeking alternate interview selection processes. Attrition in general surgery is reported as high as 26%. Thus, it is important to ensure that programs are selecting and matching applicants with shared values. Situational judgment tests (SJTs) measure educational and cultural values by posing ambiguous situations and individuals rate the effectiveness of possible reactions. SJTs have previously been shown to identify job applicants with shared values while promoting diversity. Scoring categories are high, moderate, or low values congruence. We sought to explore predictive validity of the SJT relative to program attrition.

DESIGN

Residents who matched into our program between 2018 and 2021 completed the SJT. We tracked attrition.

SETTING

UT Health San Antonio, Texas PARTICIPANTS: Fifty-six categorical general surgery residents RESULTS: Per SJT ratings, the numbers of residents who had high, moderate, and low values congruence were 27, 16, and 13, respectively. Attrition numbers for residents who scored high and moderate congruence were similar, indicating that these ratings were indistinguishable. As such, we combined those 2 categories to create a 2 × 2 matrix and used signal detection theory as a framework for analysis. Overall attrition was 16.1% (9/56). Of the 43 residents who scored high or moderate congruence, 90.7% remained in the program. There was a 9.3% chance of attrition for these residents. Of the 13 residents who scored low congruence, 38.5% attrited. While scoring as low congruence on the SJT does not definitively indicate attrition, it does indicate that attrition is 4.14 times more likely for these residents (chi-square, p = 0.0121).

CONCLUSIONS

One of the most important aspects of residency applicant selection and interviewing is mitigating risk by identifying applicants who carry a high risk of attrition. The SJT significantly identifies at-risk applicants.

The Orthopaedic Resident Selection Process: Proposed Reforms and Lessons From Other Specialties

INTRODUCTION

Proposals for substantive reforms to the orthopaedic resident selection process are growing, given increasing applicant competitiveness, burgeoning inefficiencies and inequities of the current system, and impending transition of Step 1 to pass/fail. The COVID-19 pandemic has further catalyzed the need for reforms, offering unprecedented opportunities to pilot novel changes. However, a comprehensive collation of all proposed and implemented orthopaedic reforms is currently lacking. Thus, we aimed to characterize proposed orthopaedic-specific resident selection reforms in the context of reforms implemented by other specialties.

METHODS

EMBASE, MEDLINE, Scopus, and Web of Science databases were searched for references proposing reforms to the orthopaedic resident selection process published from 2005 to 2020. An inductive approach to qualitative content analysis was used to categorize reforms.

RESULTS

Twenty-six articles proposing 13 unique reforms to the orthopaedic resident selection process were identified. The most commonly proposed reforms included noncognitive assessments (n = 8), application caps (n = 7), standardized letters of recommendation (n = 5), program-specific supplemental applications (n = 5), creation of a centralized database of standardized program information (n = 4), use of a standardized applicant composite score (n = 4), and a moratorium on postinterview communication (n = 4). Importantly, nearly all of these reforms have also been proposed or implemented by other specialties.

DISCUSSION

Numerous reforms to the orthopaedic resident selection process have been suggested over the past 15 years, several of which have been implemented on a program-specific basis, including noncognitive assessments, supplemental applications, and standardized letters of recommendation. Careful examination of applicant and program experiences and Match outcomes after these reforms is imperative to inform future directions.

Innovation in Resident Selection: Life Without Step 1

INTRODUCTION

The announcement of Step 1 shifting to a Pass/Fail metric has prompted resident selection committees (RSCs) to pursue objective methods of evaluating prospective residents. Regardless of the program's specialty or affiliated hospital/school, RSCs universally aim to recognize and choose applicants who are an "optimal fit" to their programs.¹ An optimal fit can be defined as a candidate who thrives in the clinical and academic setting, both contributing to and benefiting from their respective training environments.

OBJECTIVE

The objective of this scoping review is to evaluate alternative, innovative methods by which RSCs can evaluate applicants and predict success during residency. Objective methods include: Step 2 scores, Traditionally Used Metrics (core clerkship scores), interview performance, musical talent, sports involvement, AOA membership, research publications, unprofessional behavior, Dean's letters, Rank list, judgement testing, and specialty-specific shelf exams.^13-15.

METHODS

A scoping review was performed in compliance with the guidelines indicated by the PRISMA Protocol for scoping review.¹⁸ 9308 results were identified in the original PubMed search for articles with the key words "Resident Success". Abstract screening and application of inclusion and exclusion criteria yielded 97 articles that were critically appraised via review of full manuscript.

RESULTS

Of the articles that focused on personality traits, situational judgement testing, and specialty specific pre-assessment, all of them demonstrated some level of predictability for resident success. Standardized Letter of Recommendations, Traditionally Used Metrics, and STEP 2 did not show a unanimous consensus in demonstrating predictability of a resident's success, this is because some articles suggested predictability and some articles disputed predictability.

CONCLUSION

The authors found personality traits, situational judgement testing, and specialty specific assessments to be predictive in selecting successful residents. Further research should aim to analyze exactly how RSCs utilize these assessment tools to aid in screening their large and competitive applicant pools to find residents that will be successful in their program.

Novel Strategies for Evaluating and Improving Plastic Surgery Applicant Selection

SUMMARY

Residency applicant evaluation and selection is a critical part of developing and maintaining a high-quality plastic surgery residency program. Currently, many programs rely on objective measures such as the United States Medical Licensing Exam scores, number of research publications, grade point average, Alpha Omega Alpha Honor Medical Society status, or a combination of these objective metrics. However, there is a growing body of literature suggesting that the current means of residency applicant evaluation and selection may not be the best predictive factors of future resident success. The aim of this study was to identify nontraditional means of evaluating plastic surgery residency candidates and discuss how these means have been implemented at the authors' institution. After reviewing industry hiring practices, the authors propose that standardized interviewing and personality testing can help evaluate some of the previously intangible parts of an applicant that may play a role in teamwork, commitment, and dedication to patient care.

The current status of the diversity pipeline in surgical training

OBJECTIVE

Recent initiatives have emphasized the importance of diversity, equity, and inclusion in academic surgery. Racial/ethnic disparities remain prevalent throughout surgical training, and the "diversity pipeline" in resident recruitment and retention remains poorly defined.

METHODS

Data was retrospectively collected using two separate datasets. The Association of American Medical Colleges database was used to obtain demographic data on US medical school graduates. The US Graduate Medical Education annual report was used to obtain demographic data on surgical residents. Wilcoxon signed-rank test was used to compare racial/ethnic distribution within surgical residency programs with graduating medical students. Linear regression analysis was performed to analyze population trends over time.

RESULTS

The study population included 184,690 surgical residents from 2011 to 2020. Nine resident cohorts were created according to surgical specialty - general surgery, neurosurgery, ophthalmology, orthopedic surgery, otolaryngology, plastic surgery, cardiothoracic surgery, urology, and vascular surgery. Among surgical programs, White residents were overrepresented in 8 of 9 specialties compared to the concurrent graduating medical student class for all years (p < 0.01 each, no difference in ophthalmology). Black residents were underrepresented in 8 of 9 specialties (p < 0.01 each, no difference in general surgery). Asian representation was mixed among specialties (4 overrepresented, 1 equal, 4 underrepresented), as was Hispanic representation (5 overrepresented, 4 equal) (p < 0.01 each).

CONCLUSIONS

These data suggest that racial/ethnic disparities are inherent to the process of recruitment and retention of surgical residents. Efforts to improve the "diversity pipeline" should focus on mentorship and development of minority medical students and creating an equitable learning environment.

Anatomy Scholars Program for Medical Students Entering a Surgical Residency

A near-peer teaching experience for upper-level medical students could help prepare them for surgical residency by providing specific education opportunities and exposure to the field of surgery. Five medical students were selected to be near-peer teachers (NPTs) in gross anatomy, and then they reflected on their experiences. The NPTs spent the majority of effort in a teaching role, and reported improved NTS, anatomy knowledge, and dissection skills. MS1s and faculty also reported on the value of the ASP. Further development and evaluation of the ASP may be an excellent opportunity for future surgeons.

Resident Selection: The Role of Assessment of Emotional Intelligence and Grit

SUMMARY

The ability of our resident selection process to identify individuals who will ultimately become competent plastic surgeons is crucial to the specialty's future. Current criteria in use are not productive of that outcome. The presence of emotional intelligence and the element of grit have been incorporated in business and the military as factors to be evaluated in potential candidates. Plastic surgery should initiate an investigation of inclusion of a similar assessment of resident applicants.

Reducing Over-Interviewing in the Anesthesiology Residency Match

Background The U.S. residency recruitment process is expensive and time-consuming because of application inflation and over-invitation. Objective Using interview and match data, we quantify the predicted effects if anesthesiology residency programs excluded interviews for applicants who are very unlikely to match. Methods We previously published the validity and accuracy of the logistic regression model based on data from interview scheduling software used by 32 U.S. anesthesiology residency programs and 1300 applicants from 2015-18. Data used were program region, applicant address, numbers of interviews of the interviewee, medical school US News and World Report (USNWR) rank, the difference between United States Medical Licensing Exam (USMLE) Step 1 and 2 Clinical Knowledge (CK) scores, and the historical average of USMLE scores of program residents. In the current study completed in 2020, the predicted probabilities and their variances were summed among interviewees for 30 deidentified programs. Results For anesthesiology, the median residency program could reduce their interviews by 16.9% (97.5% confidence interval 8.5%-24.1%) supposing they would not invite applicants if the 99% upper prediction limit for the probability of matching was less than 10.0%. The corresponding median savings would be 0.80 interviews per matched spot (0.34-1.33). In doing so, the median program would sustain a risk of 5.3% (97.5% confidence interval 2.3%-7.9%) of having at least one interviewee removed from their final rank-to-match list. Conclusion Using novel interview data and analyses, we demonstrate that residency programs can substantively reduce interviews with less effect on rank-to-match lists. The data-driven approach to manage marginal interviews allows program leadership to better weigh costs and benefits when composing their annual list of interviewees.

Early Detection and Remediation of Problem Learners

There are myriad types of problem learners in surgical residency and most have difficulty in more than 1 competency. Programs that use a standard curriculum of study and assessment are most successful in identifying struggling learners early. Many problem learners lack appropriate systems for study; a multidisciplinary educational team that is separate from the team that evaluates the success of remediation is critical. Struggling residents who require formal remediation benefit from performance improvement plans that clearly outline the issues of concern, describe the steps required for remediation, define success of remediation, and outline consequences for failure to remediate appropriately.

Different approaches to selection of surgical trainees in the European Union

BACKGROUND

There is an increasing global interest in selection processes for candidates to surgical training. The aim of the present study is to compare selection processes to specialist surgeon training in the European Union (EU). A secondary goal is to provide guidance for evidence-based methods by a proposed minimum standard that would align countries within the EU.

METHODS

Publications and grey literature describing selection strategies were sought. Correspondence with Union Européenne des Médecins Specialists (UEMS) Section of Surgery delegates was undertaken to solicit current information on national selection processes. Content analysis of 13 semi-structured interviews with experienced Swedish surgeons on the selection process. Two field trips to Ireland, a country with a centralized selection process were conducted. Based on collated information typical cases of selection in a centralized and decentralized setting, Ireland and Sweden, are described and compared.

RESULTS

A multitude of methods for selection to surgical training programs were documented in the 27 investigated countries, ranging from locally run processes with unstructured interviews to national systems for selection of trainees with elaborate structured interviews, and non-technical and technical skills assessments. Associated with the difference between centralized and decentralized selection systems is whether surgical training is primarily governed by an employment or educational logic. Ireland had the most centralized and elaborate system, conducting a double selection process using evidence-based methods along an educational logic. On the opposite end of the scale Sweden has a decentralized, local selection process with a paucity of evidence-based methods, no national guidelines and operates along an employment logic, and Spain that rely solely on examination tests to rank candidates.

CONCLUSION

The studied European countries all have different processes for selection of surgical trainees and the use of evidence-based methods for selection is variable despite similar educational systems. Selection in decentralized systems is currently often conducted non-transparent and subjectively. A suggested improvement towards an evidence-based framework for selection applicable in centralized and decentralized systems as well as educational and employer logics is suggested.

Systems-Level Reforms to the US Resident Selection Process: A Scoping Review

BACKGROUND

Calls to reform the US resident selection process are growing, given increasing competition and inefficiencies of the current system. Though numerous reforms have been proposed, they have not been comprehensively cataloged.

OBJECTIVE

This scoping review was conducted to characterize and categorize literature proposing systems-level reforms to the resident selection process.

METHODS

Following Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines, searches of Embase, MEDLINE, Scopus, and Web of Science databases were performed for references published from January 2005 to February 2020. Articles were included if they proposed reforms that were applicable or generalizable to all applicants, medical schools, or residency programs. An inductive approach to qualitative content analysis was used to generate codes and higher-order categories.

RESULTS

Of 10 407 unique references screened, 116 met our inclusion criteria. Qualitative analysis generated 34 codes that were grouped into 14 categories according to the broad stages of resident selection: application submission, application review, interviews, and the Match. The most commonly proposed reforms were implementation of an application cap (n = 28), creation of a standardized program database (n = 21), utilization of standardized letters of evaluation (n = 20), and pre-interview screening (n = 13).

CONCLUSIONS

This scoping review collated and categorized proposed reforms to the resident selection process, developing a common language and framework to facilitate national conversations and change.

Attrition From a National Surgical Training Program: A Qualitative Analysis

OBJECTIVE

The objective of this study was to examine the trainee experience to identify some of the factors which contribute to attrition from surgical training.

SUMMARY BACKGROUND

Not all trainees who commence a surgical training program continue and complete it. Surgical training can be personally and professionally demanding and trainees may, for a multitude of reasons, change career direction. Attrition from surgical training impacts upon multiple stakeholders: A decision to leave may be difficult and time consuming for the individual and can generate unanticipated inefficiency at a systems level. This project examined attrition from a national surgical training program to deepen understanding of some of the causes of the phenomenon.

METHODS

A qualitative study was performed. A purposeful sampling strategy was used to identify representative participants. Semistructured interviews were conducted with eleven trainees who withdrew or considered doing so. A thematic analysis was performed to examine the experiences of trainees and explore the factors which influenced a decision to withdraw.

FINDINGS

Five major themes emerged from the interview data: delivery of training, the training atmosphere, influence of seniors, concerns regarding progression, and the perception of the future role with respect to lifestyle.

CONCLUSIONS

The personal experience of surgical training is crucial in informing a decision to withdraw from a program. Voluntary attrition is appropriate where doctors, after experiencing some time in surgical training, recognize that a surgical career does not meet their expectation. However, improving the delivery of training by addressing the concerns identified in this study may serve to enhance the personal training experience and hence maximize retention.

Remediation for surgical trainees: recommendations from a narrative review

BACKGROUND

Remediation involves formalized support for surgical trainees with significant underperformance to return to expected standards. There is a need to understand current evidence of remediation for surgical trainees to inform practice and justify investment of resources.

METHODS

Following the principles of a systematic review, we conducted a narrative analysis to make recommendations for remediation of underperforming surgical trainees.

RESULTS

From a review of 55 articles on remediation of trainees in medical and surgical sub-specialities, we have identified system and process level recommendations. Remediation is reported as long-term, complex and resource-intensive. Establishing a defined and standardized remediation framework enables co-ordination of multi-modal interventions. System level recommendations aim to consolidate protocols via developing better assessment, intervention and re-evaluation modalities whilst also strengthening support to supervisors conducting the remediation. Process level recommendations should be tailored for the specific needs of each trainee, aiming to be proactive with interventions within a programmatic framework. Regular reassessment is required, and long-term follow-up shows that remediation efforts are often successful.

CONCLUSION

While remediation within a programmatic framework is complex, it is often a successful approach to return surgical trainees to their expected standard. Future directions involve applying learning theories, encouraging research methods and to develop integrated collaborate protocols and support to synergize efforts.

Interview Data Highlight Importance of "Same-State" on Anesthesiology Residency Match

BACKGROUND

The US residency application, interview, and match processes are costly and time-intensive. We sought to quantify the importance of an applicant being from the same-state as a residency program in terms of how this impacted the number of interviews needed to match.

METHODS

We examined data from interview scheduling software used by 32 programs located in 31 US states and 1300 applicants for the US anesthesiology recruitment cycles from 2015 to 2018. Interviewee data (distance from program, region, numbers of interviews, and program at which interview occurred) were analyzed to quantify the effect of the interviewee being from the same state as the residency program on the odds of matching to that program. Other variables of interest (medical school, current address, US Medical Licensing Exam [USMLE] Step 1 and 2 clinical knowledge [CK] scores, Alpha Omega Alpha [AOA] status, medical school ranking) were also examined as controls. Confidence intervals (CI) were calculated for the ratios of odds ratios.

RESULTS

An interviewee living in the same state as the interviewing program could have 5.42 fewer total interviews (97.5% CI, 3.02-7.81) while having the same odds of matching. The same state effect had an equivalent value as an approximately 4.14 USMLE points-difference from the program's mean (97.5% CI was 2.34-5.94 USMLE points). Addition of whether the interviewee belonged to an affiliated medical school did not significantly improve the model; same-state remained significant (P < .0001) while affiliated medical school was not (P = .40).

CONCLUSIONS

Our analysis of anesthesiology residency recruitment using previously unstudied interview data shows that same-state locality is a viable predictor of residency matching and should be strongly considered when evaluating whether to interview an applicant.

Personal Statements in General Surgery: An Unrecognized Role in the Ranking Process

OBJECTIVE

Personal statements are a requirement of general surgery residency applications. Yet, their role in an applicant's final rank within a program remains unclear. This study explores the language used in personal statements to differentiate applicants in the general surgery residency ranking process.

DESIGN

A textual analysis of general surgery residency applicant personal statements was performed. Using inductive coding and grounded theory, 3 main themes from personal statements were identified: my story, my future, my goals. These themes were utilized to build a dictionary consisting of over 400 descriptive terms in multiple categories. Data was extracted using the Linguistics Inquiry and Word Count program, which can linguistically determine basic characteristics from text. The data was stratified according to final rank and gender for analysis, using correlation and descriptive statistics.

SETTING

Large, urban, academic general surgery residency program.

PARTICIPANTS

One hundred nineteen personal statements during the 2018 to 2019 application cycle were analyzed. All applicants were interviewed and considered for inclusion on our final rank list.

RESULTS

There were 68 (57%) females on the final rank list with no difference in the distribution of gender between those in the top and bottom halves (p = 0.11). Overall, personal statements for the top applicants scored higher in grit than those in the bottom half (median 0.42% vs 0.35%, p = 0.03). Males ranked in the top half had less use of agentic (p = 0.04) and efficient/organized (p = 0.03) words when compared with males ranked in the bottom half. In contrast, females ranked in the top half used more grit words compared to those in the bottom half (median 0.45% vs 0.35%, p = 0.004).

CONCLUSIONS

Linguistic differences existed in the personal statements of top- and bottom-ranked applicants to a general surgery residency program. These findings provide an adjunctive tool for differentiating applicants based on this underutilized component of the ranking process.

The Art of Teaching, Training, and Putting the Scalpel in Residents' Hands

Surgical education is under tremendous pressure due to ever-increasing medical knowledge and demands on trainees' time. They must continually learn more in less time due to work hour limitations, regulations, and electronic medical record demands. Surgical training must become more efficient. There is an unprecedented array of education and training opportunities for resident preparation. The preparation for each case has to be maximal. Preoperative, intraoperative, and postoperative simulation and discussions improve the educational benefit of the trainee experience. For the teaching surgeon, putting a scalpel in residents' hands requires patience, knowledge, judgment, and a leap of faith in the resident.

Values and beliefs on trainee selection: What counts in the eye of the selector? A qualitative study exploring the program director's perspective

OBJECTIVES

Postgraduate trainee selection is a high-stakes process. While many studies focused on selection methods and psychometrics, little is known about the influence of selectors' personal values and beliefs in the judgment and decision-making process. A better understanding of these factors is vital since selectors determine the future workforce.

METHODS

We interviewed programme directors (PDs) from 11 specialties in one University Hospital. Thematic analysis was conducted with a combined approach of generic and in-vivo coding.

RESULTS

PDs value excellence, 'fit' and personal characteristics. The content of these values are subject to personal interpretation and differ between PDs. PDs use various 'proxies' as alternative indicators of performance. They consider intuition, teamwork and autonomy important in judgement and decision-making. PDs find selection challenging and feel great accountability towards candidates and society.

CONCLUSIONS

Selectors criteria of judgement- and decision-making often remain implicit and focus on prior achievements and 'fit' with the current trainee-pool, possibly compromising the workforce's diversity. Implicit 'proxies' and intuitive decision-making may be an unwitting source of judgemental bias. 'Making the implicit explicit', by increasing awareness of personal values and beliefs and structuring the selection interview, may improve the quality of trainee selection.

Initial Experience with a Virtual Platform for Advanced Gastrointestinal Minimally Invasive Surgery Fellowship Interviews

Background

The COVID-19 pandemic travel restrictions triggered a rapid alteration in the interview process for fellowships this spring. We describe our initial experience with virtual interviews for Advanced Gastrointestinal (GI) Minimally Invasive Surgery Fellowships and assess the value and limitations via a post-interview applicant survey.

Study Design

Twenty candidates were interviewed via Zoom teleconferencing during March and April 2020 using combined group and breakout rooms. An anonymous post-interview Likert and free text survey was sent to candidates with questions regarding feasibility, appropriateness, and acceptability of this method.

Results

Seventeen of 20 candidates (85%) responded to the survey. The candidates rated ease of interaction with the program director, faculty surgeons, and the current fellow highly: 94%, 83%, and 89%, respectively. The majority (53%) stated the virtual interviews exceeded or met expectations. Only a minority, 12%, reported the virtual platform was short of expectations. Approximately 70% noted little to no impact of not being able to conduct these interviews in-person and not being able to physically see the program institution. Overall, 94% were satisfied with their experience, and only 6% were neutral, with no respondents reporting dissatisfaction. Finally, 76% would recommend a virtual interview in the future. Most negative open response comments were secondary to issues with software rather than the lack of the in-person traditional interviews.

Conclusions

The use of a remote teleconferencing platform provides a favorable method for conducting fellowship interviews and results in a high degree of candidate satisfaction. Virtual interviews will likely be increasingly substituted for in-person interviews across the spectrum of medical training.

The neurosurgery residency interview: assessing applicant perspectives on question content, utility, and stress

OBJECTIVE

Residency interviews are integral to the recruitment process yet imperfect. Through surveys of neurosurgery residency applicants, the authors describe interview content and the perceived utility and stress of topics from the applicant's perspective.

METHODS

All 2018-2019 neurosurgery resident applicants applying to three particular programs were surveyed. Across 10 interview topics, survey questions assessed topic frequency and the applicant's opinion of the utility and stress of each topic (Likert scale 1-5). Analyses included descriptive statistics, Spearman's rank correlation, and logistic regression.

RESULTS

One hundred thirty-three of 265 surveyed US residency applicants (50%) responded. Extracurricular activities, research, future career, non-medicine interests, and small talk were discussed in all interviews. The least frequent topics included neurosurgical knowledge assessment (79%) and manual dexterity tests (45%). The most useful topics according to respondents were future career objectives (4.78 ± 0.49) and prior research (4.76 ± 0.50); the least useful were neurosurgical knowledge assessment (2.67 ± 1.09) and manual dexterity tests (2.95 ± 1.05). The most stressful topics were neurosurgical knowledge assessment (3.66 ± 1.23) and ethical/behavioral scenarios (2.94 ± 1.28). The utility and stress of manual dexterity tests and neurosurgical knowledge assessments were inversely correlated (r = -0.40, p < 0.01; r = -0.36, p < 0.01), whereas no such correlation existed for ethical/behavioral questions (r = -0.12, p = 0.18), indicating that ethical/behavioral questions may have been stressful but were potentially useful topics. Respondents who attended ≥ 15 interviews were more likely to be asked about the three most stressful topics (each p < 0.05). Respondents with children were less likely to be asked about ethical/behavioral scenarios (OR 0.13, 95% CI 0.03-0.52, p < 0.01).

CONCLUSIONS

Applicants found several of the most frequently discussed topics to be less useful, indicating a potential disconnect between applicant opinion and the faculty's preferred questions. Ethical/behavioral scenarios were rated as stressful but still useful, representing a potentially worthwhile type of question. These data provide several avenues for potential standardization and improvement of the interview process.

If You Build It, Will They Come? Candidate Completion of Preinterview Screening Assessments

INTRODUCTION

Residency applicant screening practices are inefficient and costly. However, programs may not consider using alternative assessments for fear that candidates will be "turned off" by additional hurdles in the application process. This study explores the relationship between candidate completion of preinterview screening assessments, applicant examination scores, and program factors.

METHODS

Applicants to any of 7 general surgery residency programs were invited to take a preinterview online assessment. Program characteristics and applicant United States Medical Licensing Exams scores were considered in relation to each program's assessment completion rate.

RESULTS

A total of 2960 applicants were invited to take the assessment and 97% (2870/2960) completed it. Program completion rates ranged from 95% to 98%. There was no correlation between program characteristics and applicant completion rates. Candidates who did not complete the assessment had significantly lower United States Medical Licensing Exams scores.

CONCLUSIONS

Incorporating preinterview assessments to objectively measure candidate competencies and fit will not detract applicants from a general surgery program.

Evaluation of Validity Evidence for Personality, Emotional Intelligence, and Situational Judgment Tests to Identify Successful Residents

Importance

The ability to identify candidates who will thrive and successfully complete their residency is especially critical for general surgery programs.

Objective

To assess the extent to which 3 screening tools used extensively in industrial selection settings-emotional intelligence (EQ), personality profiles, and situational judgment tests (SJTs)-could identify successful surgery residents.

Design, Setting, and Participants

In this analysis, personality profiles, EQ assessments, and SJTs were administered from July through August 2015 to 51 postgraduate year 1 through 5 general surgery residents in a large general surgery residency program. Associations between these variables and residency performance were investigated through correlation and hierarchical regression analyses.

Interventions

Completion of EQ, personality profiles, and SJT assessments.

Main Outcomes and Measures

Performance in residency as measured by a comprehensive performance metric. A score of zero represented a resident whose performance was consistent with that of their respective cohort's performance; below zero, worse performance; and greater than zero, better performance.

Results

Of the 61 eligible residents, 51 (84%) chose to participate and 22 (43%) were women. US Medical Licensing Examination Step 1 (USMLE1), but not USMLE2, emerged as a significant factor (t2,49 = 1.98; β = 0.30; P = .03) associated with overall performance. Neither EQ facets nor overall EQ offered significant incremental validity over USMLE1 scores. Inclusion of the personality factors did not significantly alter the test statistic and did not explain any additional portion of the variance. By contrast, inclusion of SJT scores accounted for 15% more of the variance than USMLE1 scores alone, resulting in a total of 25% of the variance explained by both USMLE1 and SJT scores (F2,57 = 7.47; P = .002). Both USMLE1 (t = 2.21; P = .03) and SJT scores (t = 2.97; P = .005) were significantly associated with overall resident performance.

Conclusions and Relevance

This study found little support for the use of EQ assessment and only weak support for some distinct personality factors (ie, agreeableness, extraversion, and independence) in surgery resident selection. Performance on the SJT was associated with overall resident performance more than traditional cognitive measures (ie, USMLE scores). These data support further exploration of these 2 screening assessments on a larger scale across specialties and institutions.

Interview Techniques Utilized in Radiology Resident Selection-A Survey of the APDR

RATIONALE AND OBJECTIVES

To evaluate interview techniques currently used in the selection of diagnostic radiology resident candidates and to identify factors influencing the use of alternative interview techniques.

MATERIALS AND METHODS

An anonymous 25 question e-mail survey was provided to 319 active members of the Association of Program Directors in Radiology. The survey included questions on residency demographics, organization of resident applicant interviews, types of interview techniques utilized, scoring and ranking of applicants, and facets of the interview/application felt most important to the selection process. Statistical analysis was performed to identify factors associated with the use of alternative interview techniques.

RESULTS

93.7% of responding programs use traditional interview techniques, with 92% using unblinded, unstructured interviews, 8% blinded, unstructured interviews. Structured interview questions were incorporated in 22%. Few programs used alternative techniques like the multiple mini-interview. None of the programs used written prompts during the interview, 3% used casual visual cognitive testing, 10% used panel interview techniques, and none used formal personality testing. For ranking candidates in the match, the most important facets considered were USMLE Step scores, performance on the interview, clinical course grades, and letters of reference. Factors associated with use of alternative techniques were domains associated with program size and number of faculty.

CONCLUSION

The majority of radiology training programs still rely upon the traditional unblinded interview technique. There is an opportunity for training programs to examine alternative techniques that reduce bias and may provide better insight into other aspects of the candidate that may not be as readily highlighted with the traditional, unblinded interview.

Identification of Warning Signs During Selection of Surgical Trainees

OBJECTIVE

The aim was to document empirical observations about antecedents to and practices of unsuitable behaviours amongst surgical trainees and develop an interview guide that could be used for the selection process.

DESIGN

A mixed methods design was adopted combining a survey distributed to senior surgeons and heads of departments, followed by semi-structured interviews with experienced surgeons.

SETTING

All surgical departments and hospitals in The South Swedish Health Care Region.

PARTICIPANTS

The survey was completed by 54 of 83 eligible surgeons above 50years of age, and 4 of 7 heads of surgical departments. Semi-structured interviews with 13 surgeons representing local, regional, and university hospitals from the same cohort.

RESULTS

Forty-six (85%) surgeons and four of seven heads of departments responded that they had come across surgical trainees deemed unsuitable to train and work as a surgeon. All heads of department and 31 of 54 of the surgeons believed tendencies towards unsuitability are evident early during training. From the survey, 107 statements described reasons for finding a trainee unsuitable. Qualitative analysis of the interviews and free-text answers of the survey led to identification of 11 problem domains with associated "warning signs". An interview guide to help detect unsuitability tendencies in candidates during selection procedures was constructed.

CONCLUSIONS

Experienced surgeons have quite consistent views on what makes a person unsuitable as a surgeon. Their views have been systematized into 11 problem domains, and a set of 'warning signs' for unsuitable behaviours and traits has been developed. Early detection of these signs and traits is important for the individual, the work environment, and patient safety. A recommendation for a minimum framework for selection including the constructed interview guide is presented.

Job Crafting to Persist in Surgical Training: A Qualitative Study From the Resident's Perspective

BACKGROUND

Attrition in surgical training, a result of poor well-being at work, continues to rise. Work engagement and persistence, the other side of the coin, depend on the proactivity of residents to optimize the demands and resources in the workplace to achieve a better fit with the environment. This type of proactivity refers to job-crafting. In this study, we sought to explore the specific mechanisms underlying residents' job-crafting to enhance work engagement and persistence, as well as the role of supervisors in this endeavor.

MATERIALS AND METHODS

We used a constructivist-grounded theory design. We conducted semistructured interviews of active residents. During the analysis of verbatim protocols, we developed codes and categories and identified the main themes (job-crafting mechanisms) related to work engagement and persistence. We anticipated procedures to engage in reflexivity.

RESULTS

The residents showed six job-crafting mechanisms (build trust with supervisors, manage proactively, seek help, see errors and frustrations as learning opportunities, find a suitable work-life balance, and seek challenges actively). The supervisors contributed to residents' job-crafting with four mechanisms (earning the trust of the residents, providing guidance and support, allowing residents to fulfill tasks befitting a surgeon in training, and offering increasingly challenging tasks to residents).

CONCLUSIONS

Work engagement and persistence in training are active processes that depend on the ability of residents to take advantage of resources and challenges and to diminish hindering demands at the workplace. Future research should focus on the perspectives of the supervisor, as well as on the effectiveness of job-crafting interventions, to enhance work engagement.

How Much Are We Spending on Resident Selection?

INTRODUCTION

Rigorous selection processes are required to identify applicants who will be the best fit for training programs. This study provides a national snapshot of selection practices used within surgical residency programs and their associated financial costs.

METHODS

A 17-item online survey was distributed to General Surgery Program Directors (PDs) via the Association of Program Directors in Surgery listserv. The survey examined program characteristics, applicant pool size, and interview day components of the prior match year. PD/coordinator teams also provided hard costs associated with interview day components, as well as time and effort estimations among program faculty, residents, and staff during the past interview season. Effort estimates were translated to dollar values via national salary data reports of hourly wages for faculty and annual wages for administrative staff and residents. Descriptive statistics and one-way analysis of variance via SPSS 24.0 were used to examine the data.

RESULTS

One-hundred and twenty-eight responses were received, reflecting 48% (128/267) of programs in the 2017 match. Average hard costs (±SD) were $8053 ± 6467, covering food ($3753 ± 4042), social sessions ($3175 ± 3749), supplies ($329 ± 866), hotel ($328 ± 1381), room reservations ($120 ± 658), shuttle fees ($84 ± 403), tour guide fees ($50 ± 379), and other ($146 + 824). Costs for personnel effort was $77,601 ± 62,413 for faculty, $12,393 ± 33,518 for residents, $6447 ± 11,107 for coordinators, and $1294 ± 1943 for staff. Total average cost associated with the interview process (hard + effort) was $100,438±87,919, with university-based programs ($128,686 ± 101,565) spending significantly more than independent-university affiliated ($61,162 ± 33,945), independent ($74,793 ± 73,261), and military ($62,495 ± 38,532) programs (p < 0.01). Average cost for each residency program per position being filled was $18,648 ± 13,383, and average cost per interviewee was $1221 ± 894.

CONCLUSIONS

In an era of declining resources for medical education, PDs must understand the time and effort associated with resident selection. These data reveal that residency programs are spending significant time and resources on the current selection process. Program leaders can use these data to assess their current selection strategies, review faculty and staff time allocation, and identify opportunities for making the process more efficient.