Still under the microscope: Can a surgical aptitude test predict otolaryngology resident performance?

Analysis of Current Neurological Surgery Residents and Prior Medical Education: Do Medical School Attributes Matter?

BACKGROUND

With the recent changes to the U.S. Medical Licensing Examination grading system, an understanding of the factors that influence the neurological surgery residency match process is crucial for residency directors. The aim of the present retrospective study was to explore the associations of medical school location, ranking, private school status, size, and presence of an American Association of Neurological Surgeons (AANS) chapter or neurological surgery interest group (NSIG) with the neurosurgery match outcomes.

METHODS

An enrollment list of all accredited U.S. neurosurgery residency programs was compiled on June 28, 2021. For the included residents, the residency program, degree, and previously attended medical school were retrieved. The geographic location, ranking, private school status, and size were collected for the residency programs and medical schools attended by the residents at each program.

RESULTS

A total of 1437 residents from 101 neurosurgery residency programs (89%) were included. Graduates from the top 25 medical schools were more likely to match into their home residency programs (P < 0.001) and highly ranked residency programs (P < 0.001). Students from larger medical schools were also more likely to match into larger (P < 0.001) and highly ranked (P < 0.001) programs than were applicants from smaller schools. Students from medical schools with an AANS chapter or NSIG were also more likely to match into top ranked programs (P < 0.001 for both).

CONCLUSIONS

Medical students from the top 25 medical schools, private medical schools, medical schools with an AANS chapter, and medical schools with an NSIG were more likely to match into a prestigious residency program. These findings suggest that underlying biases might be present for program directors to consider in the resident selection process.

Modeling Technical Skills Learning Curves of Incoming Surgical Trainees

OBJECTIVE

Not all trainees reach technical competency even after completing surgical training. While assessment of technical skill is not part of the residency interview process, identifying under-performers early on may help identify opportunities for individualized, targeted training. The objectives of this study were to (1) create predictive learning curve (LC) models for each of 3 basic laparoscopic tasks to identify performers versus underperformers and (2) evaluate the use of LCs to identify underperformers during selection into surgical training.

DESIGN

Predictive LC models were created for laparoscopic pattern cutting (PC), peg transfer (PT) and intra-corporeal knots (IC) over 40 repetitions by 65 novice trainees in 2014. Trainees were categorized as performers and underperformers. Receiver operator characteristic analysis determined the minimum number of repetitions required to predict individual LCs, which were then used to determine the proportion of underperformers.

SETTING

Technical performance was assessed onsite at the Canadian Residence Matching Service (CaRMS) interviews, after interview completion (January 2015).

PARTICIPANTS

Applicants to general surgery (GS) and gynecology (OBGYN) participated in a skills assessment during.

RESULTS

The PC, PT and IC tasks required a minimum of 8, 10, and 5 repetitions respectively, to predict overall performance. Predictive values for each task had excellent sensitivity and specificity: 1.00, 1.00 (PC); 1.00, 1.00 (PT); and 0.94, 1.00 (IC). Eighty applicants completed 8 PC repetitions; 16% were identified as underperformers.

CONCLUSIONS

Individual LCs for three different laparoscopic tasks can be predicted with excellent sensitivity and specificity based on 10 repetitions or less. This information can be used to identify trainees who may have difficulty with laparoscopic technical skills early on.

Simulation as Soothsayer: Simulated Surgical Skills MMIs During Residency Interviews are Associated With First Year Residency Performance

OBJECTIVE

The main consideration during residency recruitment is identifying applicants who will succeed during residency. However, few studies have identified applicant characteristics that are associated with competency development during residency, such as the Accreditation Council for Graduate Medical Education milestones. As mini multiple interviews (MMIs) can be used to assess various competencies, we aimed to determine if simulated surgical skills MMI scores during a general surgery residency interview were associated with Accreditation Council for Graduate Medical Education milestone ratings at the conclusion of intern year.

DESIGN

Retrospective cohort study. Interns' Step 1 and 2 clinical knowledge (CK) scores, interview day simulated surgical skills MMI overall score, traditional faculty interview scores, average overall milestone ratings in the spring of residency, and intern American Board of Surgery In-Training Examination (ABSITE) percentile scores were gathered. Two multiple linear regression were performed analyzing the association between Step 1, Step 2 CK, MMI, and traditional faculty interview scores with (1) average overall milestone rating and (2) ABSITE percentile scores, controlling for categorical/preliminary intern classification.

SETTING

One academic medical center PARTICIPANTS: General surgery interns matriculating in 2020-2021 RESULTS: Nineteen interns were included. Multiple linear regression revealed that higher overall simulated surgical skills MMI score was associated with higher average milestone ratings (β = .45, p = 0.03) and higher ABSITE score (β = .43, p = 0.02) while neither Step 1, Step 2 CK, nor faculty interview scores were significantly associated with average milestone ratings.

CONCLUSIONS

Surgical residency programs invest a tremendous amount of effort into training residents, thus metrics for predicting applicants that will succeed are needed. Higher scores on a simulated surgical skills MMIs are associated with higher milestone ratings 1 year into residency and higher intern ABSITE percentiles. These results indicate a noteworthy method, simulated surgical skills MMIs, as an additional metric that may select residents that will have early success in residency.

Real-Time Dual Video Conferencing of Simulated Tracheostoma Maturation During Residency Interviews

BACKGROUND

There are few reports of dexterity tests being done in a distance telecommunication setting for residency applicant evaluation.

OBJECTIVE

To report the feasibility and suitability of a virtual suturing skills assessment during residency interviews when added to the standard assessment process.

METHODS

A suturing simulation was developed and implemented during otolaryngology-head and neck surgery (OHNS) residency interviews for the 2020-2021 cycle at one program. On the day of the interview, the activity was completed in real time using 2-camera video conferencing with the 2 resident assessors providing a numerical assessment based on an adapted scoring rubric from prior suturing activities at the institution. The exercise involved suturing a 3/4-inch Penrose drain circumferentially with half-vertical mattress stitches to simulate the maturation of a tracheostoma. The residency selection committee then completed a 7-item Likert-type survey, developed by the authors, to evaluate the simulation exercise.

RESULTS

Fifty-one applicants representing all interviewees in the cycle successfully completed this assessment without technologic disruptions. The total cost associated with obtaining and providing the necessary supplies to applicants was $34.78 per interviewee. Time required to complete the suturing task was estimated to range from 10 to 20 minutes. The residency selection committee viewed this exercise as a success (14 of 16, 87.5%) and viewed the results as a valuable adjunct in the overall assessment of candidates (15 of 16, 93.8%).

CONCLUSIONS

A simple motor exercise completed over real-time telecommunication was feasible and perceived as helpful to the residency selection committee when assessing OHNS residency candidates.

Advanced undergraduate medical students' perceptions of basic medical competences and specific competences for different medical specialties - a qualitative study

BACKGROUND

Medical graduates should have acquired basic competences that enable them to practice medicine independently as physicians and to enter postgraduate training in any specialty they wish. Little is known about advanced undergraduate medical students' perceptions of basic medical competences needed to start postgraduate training and about specialty-specific competences. This qualitative study aims to identify medical students' perceptions of basic medical competences and specific competence requirements for different specialties.

METHODS

In December 2020, sixty-four advanced undergraduate medical students participated in the role of a resident in a competence-based telemedicine training simulating a first day in postgraduate training. After the training, eight focus group interviews were conducted about students' perceptions of basic medical competences and specialty-specific competences using a semi-structured interview guide. The interviews were transcribed and analysed thematically according to the six steps of Braun and Clarke. The analysis was carried out by an inductive search for themes, which were deductively assigned to the six competence areas of the requirement-tracking questionnaire (R-Track).

RESULTS

Regarding basic medical competences, four R-Track competence areas could be identified as main themes. The students considered 'Social-interactive competences' to be particularly relevant for basic clinical work, including 'Structuring information', 'Tactfulness', and 'Stress resistance'. Students especially emphasized 'Concentration' as an important aspect of the competence area 'Mental abilities'. Among 'Personality traits', 'Honesty' was mentioned most frequently, and students were also aware that 'Expertise' is particularly important for 'Motivation'. For different specialties, some competence areas were newly added to the competences needed for the respective specialty. For surgery, the competence areas 'Sensory abilities' and 'Psychomotor & multitasking abilities' were mentioned anew. 'Sensory abilities' were also newly attributed to radiology. 'Mental abilities' were mentioned as new competence area for psychiatry and internal medicine, while for anaesthesiology, 'Psychomotor & multitasking abilities' were newly added.

CONCLUSIONS

Advanced students seem to be well aware of basic competences needed for clinical practice. Good consensus between students and physicians was only found for psychiatry-specific competences. Medical schools should support their students in matching their perceptions of competences needed for specific specialties with specialty-specific requirements for a realistic choice of a specialty for postgraduate training.

Reshaping Residency Recruitment: Achieving Alignment Between Applicants and Programs in Surgery

OBJECTIVE

The residency recruitment process has become increasingly challenging for both applicants and program directors, in part, due to the inflation in the number of applications per student. As a result, it has become more daunting for programs to design processes that evaluate applicants holistically. Furthermore, the existing methods used to evaluate and select applicants do not necessarily predict success in residency and may inadvertently lend to gender, racial, and ethnic bias. This narrative review aims to identify innovative tools used in residency recruitment that will allow programs and applicants to better determine concordance of interests and achieve value alignment while supporting improved, objective evaluation of an applicant's unique attributes and experiences.

DESIGN

PubMed was used to conduct a narrative review of recruitment strategies in admission processes of undergraduate and graduate medical education between 1975 and June 2021, using the designated Medical Subject Heading (MeSH0 terms. Inclusion criteria were established surrounding innovative tools to better objectively screen, evaluate, or select applicants. Strategies relying primarily on traditional metrics (United States Medical Licensing Examination (USMLE) scores, Alpha Omega Alpha status, and clerkship grades) were excluded.

RESULTS

Forty-two articles met specific inclusion criteria. Using these articles, a framework was created with two specific aims: (1) to allow applicants and programs to express or assess interest and (2) to foster objective review of unique applicant attributes, skills, experiences, and competencies that align with program mission and values. The following five innovative tools for recruitment were identified: preference signaling, secondary applications, standardized letters of recommendation, situational judgment testing, and surgical simulation.

CONCLUSIONS

As the number of applications continues to rise, strategies must be implemented to allow applicants and institutions to achieve better alignment or "fit," while also giving balanced consideration to all of an applicant's unique characteristics. A more holistic approach to applicant selection is a necessary tool in order to increase diversity and inclusion within the field of surgery.

The innate aptitude's effect on the surgical task performance: a systematic review

Surgery is known to be a craft profession requiring individuals with specific innate aptitude for manipulative skills, and visuospatial and psychomotor abilities. The present-day selection process of surgical trainees does not include aptitude testing for the psychomotor and manual manipulative skills of candidates for required abilities. We aimed to scrutinize the significance of innate aptitudes in surgical practice and impact of training on skills by systematically reviewing their significance on the surgical task performance. A systematic review was performed in compliance with PRISMA guidelines. An initial search was carried out on PubMed/Medline for English language articles published over 20 years from January 2001 to January 2021. Search strategy and terms to be used included ‘aptitude for surgery’, ‘innate aptitude and surgical skills, ‘manipulative abilities and surgery’, and ‘psychomotor skills and surgery’. MERSQI score was applied to assess the quality of quantitatively researched citations. The results of the present searches provided a total of 1142 studies. Twenty-one studies met the inclusion criteria out of which six citations reached high quality and rejected our three null hypothesis. Consequently, the result specified that all medical students cannot reach proficiency in skills necessary for pursuing a career in surgery; moreover, playing video games and/or musical instruments does not promote skills for surgery, and finally, there may be a valid test with predictive value for novices aspiring for a surgical career. MERSQI mean score was 11.07 (SD = 0.98; range 9.25–12.75). The significant findings indicated that medical students with low innate aptitude cannot reach skills necessary for a competent career in surgery. Training does not compensate for pictorial-skill deficiency, and a skill is needed in laparoscopy. Video-gaming and musical instrument playing did not significantly promote aptitude for microsurgery. The space-relation test has predictive value for a good laparoscopic surgical virtual-reality performance. The selection process for candidates suitable for a career in surgery requests performance in a simulated surgical environment.

Skull Base Training and Mentorship

OBJECTIVE

To review teaching and mentoring techniques of experienced skull base surgeons and educators STUDY DESIGN:: Expert commentary.

SETTING

8th Quadrennial International Conference on vestibular schwannoma and other CPA tumors, panel on teaching, and mentoring.

MAIN OUTCOME MEASURES

Experiences and opinions of experienced skull base surgeons, both neurosurgeons and neurotologists, presented and discussed at the conference.

CONCLUSIONS

Obtaining surgical mastery is essential for the teachers of skull base surgery. Hard work and practice with immediate and constant feedback on performance is an essential component to success. Creating a patient-centered culture that encourages academic achievement is an accelerator for success of a training program. Both the mentor and the mentee must play an intentional and active role to maximize learning.

Increasing the Number of Black Otolaryngologists

Otolaryngology continues to have one of the lowest percentages of black physicians of any surgical specialty, a number than has not improved in recent years. The history of exclusion of black students in medical education as well as ongoing bias affecting examination scores, clerkship grades and evaluations, and honors society acceptance of black students may factor into the disproportionately low number of black otolaryngology residents. In order to increase the number of black physicians in otolaryngology, intentional steps must be taken to actively recruit, mentor, and train black physicians specializing in otolaryngology.

Postgraduate Assessment Approaches Across Surgical Specialties: A Systematic Review of the Published Evidence

PURPOSE

Ensuring competence for surgical trainees requires holistic assessment of the qualities and competencies necessary to practice safely and effectively. To determine the next steps toward achieving this aim, the authors conducted a systematic review to summarize and appraise the available evidence related to any assessment approach to postgraduate surgical training and to identify the dominant themes for assessment approaches across different specialties or countries.

METHOD

Medline and Embase were searched on January 10, 2019, without language or time restrictions. Any peer-reviewed study that described an assessment framework (in practice or novel) throughout postgraduate surgical training globally was included. An iterative review and thematic analysis were performed on full-text articles to determine assessment themes. Studies were then grouped by assessment themes. A tailored quality assessment of the studies included in the final analysis was conducted. Assessment themes and validity were compared across surgical specialties and countries.

RESULTS

From an initial 7,059 articles, 91 studies (evaluating 6,563 surgical trainees) were included in the final analysis. Ten defined assessment themes were extracted. Ten studies (11.0%) were deemed low risk of bias based on the quality assessment tool used and thus were determined to be high quality. Minor differences in assessment themes were observed between specialties and countries. Assessment themes neglected by individual surgical specialties and assessment themes that need validated assessment tools were identified.

CONCLUSIONS

This review highlights the low quality of evidence and fragmented efforts to develop and optimize surgical assessments. The minor differences observed demonstrate a common approach, globally and across specialties, related to surgical assessments. A paradigm shift in assessment approaches, which will require national and international collaboration, is required to optimize design and validation so that a comprehensive assessment of surgical competence can be implemented.

Ideas and Innovations to Improve the Otolaryngology–Head and Neck Surgery Residency Application and Selection Process

Objective

To (1) summarize strategies proposed in the recent otolaryngology–head and neck surgery (OTO-HNS) literature for improving the residency application and selection process, (2) evaluate the effects of recently implemented changes to the OTO-HNS match, and (3) discuss recommendations for future changes to the OTO-HNS residency application and selection process.

Data Sources

PubMed, Medline Ovid database, and article reference lists.

Review Methods

A structured literature search was performed to identify current English language articles relating to the objectives of this study using the aforementioned data sources, in accordance with the Preferred Reporting Items for Systematic Reviews and Meta Analyses (PRISMA) guidelines. The search was limited to submissions published between January 1, 2015, and January 1, 2020.

Conclusions

Numerous proposals have been made for improving the otolaryngology residency application and selection process and addressing the competitive nature of the Match. These proposals include but are not limited to mandating a secondary essay, implementing consortia and early match processes, using a signaling system, conducting regional and web-based interviews, offering early engagement with interest groups, instituting a hard cap on applications, increasing costs of applying, counseling self-restraint to prospective applicants, and creating application filters.

Implications for Practice

As the volume of literature surrounding the OTO-HNS Match continues to increase, this review aims to provide a summary of past proposals and serve as a guide for possible future innovations. We propose 3 initiatives that may improve the residency application and selection process for both program and resident, with minimal impact to the current National Resident Matching Program (NRMP) Match structure.

Otolaryngology Residency Interviews in a Socially Distanced World: Strategies to Recruit and Assess Applicants

Due to concerns surrounding travel during the COVID-19 pandemic, the 2020-2021 otolaryngology residency application cycle will be conducted virtually for the first time. Residency programs should consider the logistics of video interviews, drawing on experiences of other programs that have successfully performed virtual interviews in the past. The lack of in-person interviews will create challenges in assessing applicants, and we recommend that programs develop structured and targeted questions and even consider having candidates answer standardized questions prior to the virtual interview day. From an applicant perspective, gauging the intangibles of individual residency programs, such as resident camaraderie, program culture, and program location, will be difficult. To address this, programs should consider hosting informal virtual gatherings, create videos that highlight the resident experience, and ensure that program websites are up-to-date. Ultimately, adaptability, resilience, and innovation will allow residency programs to achieve a successful 2021 otolaryngology match.

The neurosurgery residency interview: assessing applicant perspectives on question content, utility, and stress

OBJECTIVE

Residency interviews are integral to the recruitment process yet imperfect. Through surveys of neurosurgery residency applicants, the authors describe interview content and the perceived utility and stress of topics from the applicant's perspective.

METHODS

All 2018-2019 neurosurgery resident applicants applying to three particular programs were surveyed. Across 10 interview topics, survey questions assessed topic frequency and the applicant's opinion of the utility and stress of each topic (Likert scale 1-5). Analyses included descriptive statistics, Spearman's rank correlation, and logistic regression.

RESULTS

One hundred thirty-three of 265 surveyed US residency applicants (50%) responded. Extracurricular activities, research, future career, non-medicine interests, and small talk were discussed in all interviews. The least frequent topics included neurosurgical knowledge assessment (79%) and manual dexterity tests (45%). The most useful topics according to respondents were future career objectives (4.78 ± 0.49) and prior research (4.76 ± 0.50); the least useful were neurosurgical knowledge assessment (2.67 ± 1.09) and manual dexterity tests (2.95 ± 1.05). The most stressful topics were neurosurgical knowledge assessment (3.66 ± 1.23) and ethical/behavioral scenarios (2.94 ± 1.28). The utility and stress of manual dexterity tests and neurosurgical knowledge assessments were inversely correlated (r = -0.40, p < 0.01; r = -0.36, p < 0.01), whereas no such correlation existed for ethical/behavioral questions (r = -0.12, p = 0.18), indicating that ethical/behavioral questions may have been stressful but were potentially useful topics. Respondents who attended ≥ 15 interviews were more likely to be asked about the three most stressful topics (each p < 0.05). Respondents with children were less likely to be asked about ethical/behavioral scenarios (OR 0.13, 95% CI 0.03-0.52, p < 0.01).

CONCLUSIONS

Applicants found several of the most frequently discussed topics to be less useful, indicating a potential disconnect between applicant opinion and the faculty's preferred questions. Ethical/behavioral scenarios were rated as stressful but still useful, representing a potentially worthwhile type of question. These data provide several avenues for potential standardization and improvement of the interview process.

Integrated Plastic Surgery Applicant Review: Important Factors and Selection Criteria

Matching into integrated plastic surgery residency is highly competitive. Applicants to these programs are among the most accomplished graduating medical students, consistently demonstrating some of the highest United States Medical Licensing Examination scores, mean numbers of research publications, and rates of Alpha Omega Alpha Honor Medical Society membership. The applicant review process requires programs to rely on a number of objective and subjective factors to determine which of these qualified applicants have the most potential for success. We outline these factors, discuss their correlation with resident performance, and provide our institution’s applicant review process both for applicants hoping to optimize their applications for success in the National Resident Matching Program and for program faculty hoping to optimize their resident selection process.

Identification and development of talent in surgery

Purpose

Education of the surgeon and development of surgical expertise have been debated for centuries. Today, research in surgical education applies terms and methods from other performance domains such as sport and music. However, there still seems to be a lack of consensus as to how talent may be brought into the discourse about surgical education. Especially, when it comes to identifying and developing trainees who in the future will perform better than the average surgeon.

Design/methodology/approach

This five-step scoping study aims to map existing literature about talent identification, talent development and development of expertise in the domains of surgery, sport and music in the period of 1985-2014.

Findings

A total of 242 studies, divided in the four domains of surgery (69 studies), sport (115 studies), music (34 studies) and cross-disciplinary studies (24 studies) published in the period 1985-2014 were included.

Originality/value

Informed by the performance domains of sports and music and their inclusion of a holistic, ecological approach to research, this study suggests that research in surgical education may benefit from broadening its view on talent by including psychosocial variables and environmental, demographic and structural influencers when considering how surgical talent may be identified and developed.

Development and evaluation of rhinoplasty spreader graft suture simulator for novice surgeons

OBJECTIVE

Surgical simulators aimed at mimicking elements of rhinoplasty surgery, specifically those aimed at improving cartilage suturing, are not available. Here, we present a surgical simulator for spreader graft placement that uses cartilage rather than synthetic materials and gauge improvement using objective measures for suture placement accuracy, speed, and efficiency of hand motion.

METHODS

Twenty-two otolaryngologists in two groups (residents [10] and experts [12]) were instructed to secure the two spreader graft specimen into position with three mattress sutures on a nose model that used porcine septal cartilage as a proxy for the human counterpart. Hand motion was tracked using an electromagnetic position sensing device. The time required to complete the suture task, total hand displacement, cumulative number of hand motion direction changes, and accuracy of suture insertion were measured. These measurements were compared between the two cohort groups for construct validity. The subjects completed a survey to evaluate realism and value of the model.

RESULTS

The expert group had a lower mean time required to complete the task (P < 0.05), total hand displacement (P < 0.01), and number of hand motion direction changes (P < 0.001). No significant difference was observed between the two groups in suture precision measurement. The subjects agreed on the face validity and usefulness of the trainer.

CONCLUSIONS

Our study suggests that the simulator may be a useful tool to objectively gauge suturing efficiency. Devices such as this may be useful for developing skill with suturing cartilage tissue and potentially be used to assess resident acquisition of surgical skill.

LEVEL OF EVIDENCE

NA Laryngoscope, 129:344-350, 2019.

United States Medical Licensing Exam Step I Score as a Predictor of Neurosurgical Career Beyond Residency

BACKGROUND

United States Medical Licensing Exam (USMLE) Step I score is cited as one of the most important factors when for applying to neurosurgery residencies. No studies have documented a correlation between USMLE Step I score and metrics of neurosurgical career trajectory beyond residency.

OBJECTIVE

To determine whether USMLE Step I exam scores are predictive of neurosurgical career beyond residency, as defined by American Board of Neurological Surgery (ABNS) certification status, practice type, academic rank, and research productivity.

METHODS

A database of neurosurgery residency applicants who matched into neurosurgery from 1997 to 2007 was utilized that included USMLE Step I score. Online databases were used to determine h-index, National Institutes of Health (NIH) grant funding, academic rank, practice type, and ABNS certification status of each applicant. Linear regression and nonparametric testing determined associations between USMLE Step I scores and these variables.

RESULTS

USMLE Step I scores were higher for neurosurgeons in academic positions (237) when compared to community practice (234) and non-neurosurgeons (233, P < .01). USMLE Step I score was not different between neurosurgeons of different academic rank (P = .21) or ABNS certification status (P = .78). USMLE Step I score was not correlated with h-index for academic neurosurgeons (R2 = 0.002, P = .36).

CONCLUSION

USMLE Step I score has little utility in predicting the future careers of neurosurgery resident applicants. A career in academic neurosurgery is associated with a slightly higher USMLE Step I score. However, USMLE Step I score does not predict academic rank or productivity (h-index or NIH funding) nor does USMLE Step I score predict ABNS certification status.

Simulation-Based Laparoscopic Surgery Crisis Resource Management Training-Predicting Technical and Nontechnical Skills

OBJECTIVES

To develop a unique simulation-based assessment using a laparoscopic inferior vena cava (IVC) injury scenario that allows for the safe assessment of urology resident's technical and nontechnical skills, and investigate the effect of personality traits performance in a surgical crisis.

METHODS

Urology residents from our institution were recruited to participate in a simulation-based training laparoscopic nephrectomy exercise. Residents completed demographic and multidimensional personality questionnaires and were instructed to play the role of staff urologist. A vasovagal response to pneumoperitoneum and an IVC injury event were scripted into the scenario. Technical and nontechnical skills were assessed by expert laparoscopic surgeons using validated tools (task checklist, GOALS, and NOTSS).

RESULTS

Ten junior and five senior urology residents participated. Five residents were unable to complete the exercise safely. Senior residents outperformed juniors on technical (checklist score 15.1 vs 9.9, p < 0.01, GOALS score 18.0 vs 13.3, p < 0.01) and nontechnical performance (NOTSS score 13.8 vs 10.1, p = 0.03). Technical performance scores correlated with NOTSS scores (p < 0.01) and pass/fail rating correlated with technical performance (p < 0.01 for both checklist and GOALS), NOTSS score (p = 0.02), and blood loss (p < 0.01). Only the conscientiousness dimension of the big five inventory correlated with technical score (p = 0.03) and pass/fail rating (p = 0.04).

CONCLUSIONS

Resident level of training and laparoscopic experience correlated with technical performance during a simulation-based laparoscopic IVC injury crisis management scenario, as well as multiple domains of nontechnical performance. Personality traits of our surgical residents are similar and did not predict technical skill.

Orthopaedic Surgery Residency Training: Consideration for a Surgical and Procedural Skills Competency

This perspectives report discusses the need to create a surgical and procedural skills competency for orthopedic surgery residency training programs.

Utility of selection methods for specialist medical training: A BEME (best evidence medical education) systematic review: BEME guide no. 45

BACKGROUND

Selection into specialty training is a high-stakes and resource-intensive process. While substantial literature exists on selection into medical schools, and there are individual studies in postgraduate settings, there seems to be paucity of evidence concerning selection systems and the utility of selection tools in postgraduate training environments.

AIM

To explore, analyze and synthesize the evidence related to selection into postgraduate medical specialty training.

METHOD

Core bibliographic databases including PubMed; Ovid Medline; Embase, CINAHL; ERIC and PsycINFO were searched, and a total of 2640 abstracts were retrieved. After removing duplicates and screening against the inclusion criteria, 202 full papers were coded, of which 116 were included.

RESULTS

Gaps in underlying selection frameworks were illuminated. Frameworks defined by locally derived selection criteria, and heavily weighed on academic parameters seem to be giving way to the evidencing of competency-based selection approaches in some settings. Regarding selection tools, we found favorable psychometric evidence for multiple mini-interviews, situational judgment tests and clinical problem-solving tests, although the bulk of evidence was mostly limited to the United Kingdom. The evidence around the robustness of curriculum vitae, letters of recommendation and personal statements was equivocal. The findings on the predictors of past performance were limited to academic criteria with paucity of long-term evaluations. The evidence around nonacademic criteria was inadequate to make an informed judgment.

CONCLUSIONS

While much has been gained in understanding the utility of individual selection methods, though the evidence around many of them is equivocal, the underlying theoretical and conceptual frameworks for designing holistic and equitable selection systems are yet to be developed.

Chicken thigh microvascular training model improves resident surgical skills

Objectives

Microsurgical techniques are essential for vessel anastomosis in free flap reconstructive surgery. However, teaching these skills intraoperatively is difficult. The chicken thigh microvascular model is a high‐fidelity model that has been previously validated to differentiate between skill levels of surgeons. This study aims to determine if this model objectively improves microsurgical skills.

Study Design

Validation study

Methods

Thirteen residents were given a tutorial on microvascular anastomosis and asked to perform anastomoses on the microvascular model. Anastomoses were video‐recorded and the time required for trainees to complete the first stitch of their first anastomosis was compared to the time required for the first stitch of their last anastomosis. Comparison of first and last stitch times was completed using a paired student t‐test. All participants completed a survey regarding their experience with the simulator.

Results

There was a statistically significant decrease between the time required for the first stitch (235 s, 95%CI 198–272 s) compared to last stitch (120 s, 95%CI 92–149 s), and an average 48.7% (115 s) decrease in time (p < 0.001). Junior (PGY 2/3) and senior (PGY 4/5) residents had similar decreases in time, 49.1% and 48.21%, respectively. One hundred percent of residents felt they improved during the session and 92% of residents agreed or strongly agreed that their final stitch was better than their last stitch. All residents agreed or strongly agreed that the simulation is realistic, effective in teaching the procedure, and would translate to improved intraoperative performance.

Conclusions

The chicken thigh model demonstrates objective improvements in resident microvascular surgical skills.

Level of Evidence

NA

Passing a Technical Skills Examination in the First Year of Surgical Residency Can Predict Future Performance

BACKGROUND

The ability of an assessment to predict performance would be of major benefit to residency programs, allowing for early identification of residents at risk.

OBJECTIVE

We sought to establish whether passing the Objective Structured Assessment of Technical Skills (OSATS) examination in postgraduate year 1 (PGY-1) predicts future performance.

METHODS

Between 2002 and 2012, 133 PGY-1 surgery residents at the University of Toronto (Toronto, Ontario, Canada) completed an 8-station, simulated OSATS examination as a component of training. With recently set passing scores, residents were assigned a pass/fail status using 3 standards setting methods (contrasting groups, borderline group, and borderline regression). Future in-training performance was compared between residents who had passed and those who failed the OSATS, using in-training evaluation reports from resident files. A Mann-Whitney U test compared performance among groups at PGY-2 and PGY-4 levels.

RESULTS

Residents who passed the OSATS examination outperformed those who failed, when compared during PGY-2 across all 3 standard setting methodologies (P < .05). During PGY-4, only the contrasting groups method showed a significant difference (P < .05).

CONCLUSIONS

We found that PGY-1 surgical resident pass/fail status on a technical skills examination was associated with future performance on in-training evaluation reports in later years. This provides validity evidence for the current PGY-1 pass/fail score, and suggests that this technical skills examination may be used to predict performance and to identify residents who require remediation.

Associations between Otolaryngology Applicant Characteristics and Future Performance in Residency or Practice: A Systematic Review

Objective

This systematic review aims to evaluate which applicant characteristics available to an otolaryngology selection committee are associated with future performance in residency or practice.

Data Sources

PubMed, Scopus, ERIC, Health Business, Psychology and Behavioral Sciences Collection, and SocINDEX.

Review Methods

Study eligibility was performed by 2 independent investigators in accordance with the PRISMA protocol (Preferred Reporting Items for Systematic Reviews and Meta-analyses). Data obtained from each article included research questions, study design, predictors, outcomes, statistical analysis, and results/findings. Study bias was assessed with the Quality in Prognosis Studies tool.

Results

The initial search identified 439 abstracts. Six articles fulfilled all inclusion and exclusion criteria. All studies were retrospective cohort studies (level 4). Overall, the studies yielded relatively few criteria that correlated with residency success, with generally conflicting results. Most studies were found to have a high risk of bias.

Conclusion

Previous resident selection research has lacked a theoretical background, thus predisposing this work to inconsistent results and high risk of bias. The included studies provide historical insight into the predictors and criteria (eg, outcomes) previously deemed pertinent by the otolaryngology field. Additional research is needed, possibly integrating aspects of personnel selection, to engage in an evidence-based approach to identify highly qualified candidates who will succeed as future otolaryngologists.

Optimizing the Selection of General Surgery Residents: A National Consensus

BACKGROUND

Surgical programs strive to recruit trainees who will graduate as competent surgeons; however, selection processes vary between institutions. The purpose of the present study was to (1) solicit program directors' (PDs) opinions on the proportion of trainees who have difficulty achieving competence and (2) establish consensus on the desired attributes of general surgery (GS) candidates and the technical skills that would be most indicative of future performance.

METHODS

Delphi consensus methodology was used. An open-ended questionnaire, followed by a closed-ended questionnaire, formulated as a 5-point Likert scale, was administered. A Cronbach α ≥ 0.8 with 80% of responses in agreement (4-agree and 5-strongly agree) determined the threshold for consensus.

RESULTS

The first and second rounds were completed by 14 and 11, of a potential 17, GS PDs, respectively. PDs felt that 5% or less of trainees have difficulty reaching competence in clinical knowledge, 5% to 10% in decision-making, and 5% to 15% in technical skill by the time of completion of training. Consensus was excellent (α = 0.92). The top attributes for success in GS included work ethic and passion for surgery. Technical skills that felt to be most appropriate were open tasks (one-handed tie and subcuticular suture) and laparoscopic tasks (coordination, grasping, and cutting).

CONCLUSION

PDs indicate that of the 3 domains, the largest proportion of trainees had difficulty reaching competence in technical skill. Consensus among PDs suggests that top personal attributes include work ethic and passion for surgery. Consensus of technical tasks for inclusion into selection was basic open and laparoscopic skills.

Selection for Surgical Training: An Evidence-Based Review

PURPOSE

The predictive relationship between candidate selection criteria for surgical training programs and future performance during and at the completion of training has been investigated for several surgical specialties, however there is no interspecialty agreement regarding which selection criteria should be used. Better understanding the predictive reliability between factors at selection and future performance may help to optimize the process and lead to greater standardization of the surgical selection process.

METHODS

PubMed and Ovid MEDLINE databases were searched. Over 560 potentially relevant publications were identified using the search strategy and screened using the Cochrane Collaboration Data Extraction and Assessment Template.

RESULTS

57 studies met the inclusion criteria. Several selection criteria used in the traditional selection demonstrated inconsistent correlation with subsequent performance during and at the end of surgical training. The following selection criteria, however, demonstrated good predictive relationships with subsequent resident performance: USMLE examination scores, Letters of Recommendation (LOR) including the Medical Student Performance Evaluation (MSPE), academic performance during clinical clerkships, the interview process, displaying excellence in extracurricular activities, and the use of unadjusted rank lists.

CONCLUSIONS

This systematic review supports that the current selection process needs to be further evaluated and improved. Multicenter studies using standardized outcome measures of success are now required to improve the reliability of the selection process to select the best trainees.

International assessment practices along the continuum of surgical training

BACKGROUND

The objectives of this study were to assemble an international perspective on (1) current, and (2) ideal technical performance assessment methods, and (3) barriers to their adoption during: selection, in-training, and certification.

METHODS

A questionnaire was distributed to international educational directorates.

RESULTS

Eight of 10 jurisdictions responded. Currently, aptitude tests or simulated tasks are used during selection, observational rating scales during training and nothing is used at certification. Ideally, innate ability should be determined during selection, in-training evaluation reports, and global rating scales used during training, whereas global and procedure-specific rating scales used at the time of certification. Barriers include lack of predictive evidence for use in selection, financial limitations during training, and a combination with respect to certification.

CONCLUSIONS

Identifying current and ideal evaluation methods will prove beneficial to ensure the best assessments of technical performance are chosen for each training time point.

Standardized letters of recommendation and successful match into otolaryngology

OBJECTIVES/HYPOTHESIS

Historically, narrative letters of recommendation have been utilized in the selection of applicants for otolaryngology residency programs. In the last two application cycles, our specialty adopted a standardized letter of recommendation (SLOR). The intent was to decrease time burden for letter writers and to provide readers with an objective evaluation of applicants. The objective of this study was to determine attributes in the SLOR that correlate with matching into a residency program.

STUDY DESIGN

We performed a retrospective study using SLOR, United States Medical Licensing Examination (USMLE) step 1 scores, and matched outcomes of applicants who applied to our institution for the 2013 and 2014 match cycle.

METHODS

We included the following variables from the SLOR in the statistical analysis to determine which ones were associated with matching: patient care, medical knowledge, communication skills, procedural skills, research, initiative and drive, commitment to otolaryngology, commitment to academic medicine, match potential, and USMLE1 scores.

RESULTS

We identified 532 applicants and 963 SLOR. In successful applicants, scores for patient care, medical knowledge, communication skills, initiative and drive, and match potential were statistically higher (P < 0.05). Scores for professionalism, procedural skills, research, commitment to otolaryngology, commitment to academic medicine, and USMLE step 1 scores were not higher among successfully matched applicants.

CONCLUSION

Although SLOR can save time for letter writers and provide an objective description of applicants, the utility of individual domains within the SLOR is questionable. Additionally, it is concerning that applicants' professionalism and procedural skills are not correlated with matching in our specialty.

LEVEL OF EVIDENCE

NA. Laryngoscope, 126:1071-1076, 2016.

Use of the Interview in Resident Candidate Selection: A Review of the Literature

BACKGROUND

Although the resident candidate interview is costly and time-consuming for both applicants and programs, it is considered critically important for resident selection. Noncognitive attributes, including communication skills and professionalism, can be assessed by the personal interview.

OBJECTIVE

We conducted a review of the literature on the residency interview to identify the interview characteristics used for resident selection and to ascertain to what extent the interview yields information that predicts future performance.

METHODS

We searched PubMed and Scopus using the following search terms: residency, internship, interview, selection, and performance. We extracted information on characteristics of the interview process, including type of interview format, measures taken to minimize bias by interviewers, and testing of other clinical/surgical skills.

RESULTS

We identified 104 studies that pertained to the resident selection interview, with highly varied interview formats and assessment tools. A positive correlation was demonstrated between a medical school academic record and the interview, especially for unblinded interview formats. A total of 34 studies attempted to correlate interview score with performance in residency, with mixed results. We also identified a number of studies that included personality testing, clinical skills testing, or surgical skills testing.

CONCLUSIONS

Our review identified a wide variety of approaches to the selection interview and a range of factors that have been studied to assess its effectiveness. More research needs to be done not only to address and ascertain appropriate interview formats that predict positive performance in residency, but also to determine interview factors that can predict both residents' "success" and program attrition.