Otolaryngology resident selection: do rank lists matter?

Relationship Between National Residency Matching Program (NRMP) Rank Order and Otolaryngology Residency Performance

Objective

The process of resident recruitment is costly, and our surgical residency program expends significant time on the resident selection process while balancing general duties and responsibilities. The aim of our study was to explore the relationship between otolaryngology-head and surgery (OHNS) residents' National Residency Matching Program (NRMP) rank-list position at our institution and their subsequent residency performance.

Study Design

Retrospective cohort study.

Setting

Single site institution.

Methods

We retrospectively reviewed 7 consecutive resident classes (2011-2017) at a single tertiary OHNS residency program. We reviewed each resident's absolute rank order in the NRMP matches. Measures of residency performance included overall faculty evaluation during postgraduate year 5 (PGY5), annual in-service examination scores (scaled score), and the number of manuscripts published in peer-reviewed journals. Correlations between NRMP rank order and subsequent residency performance were assessed using Spearman's rho correlation coefficients (ρ).

Results

Twenty-eight residents entered residency training between 2011 and 2017. The average rank position of the trainees during this study was 9.7 (range: 1-22). We found no significant correlation between rank order and faculty evaluation during PGY5 (ρ = 0.097, P = .625) or number of publications (ρ = -0.256, P = .189). Additionally, when assessing the association between rank order and annual Otolaryngology Training Examination-scaled scores, no statistically significant relationship was found between the 2 (P > .05).

Conclusion

Our results showed that there were no significant correlations between OHNS rank order and various measures of success in residency training, which aligns with existing literature. Further investigation of this relationship should be conducted to ensure the applicability of our findings.

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values

PURPOSE

The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda.

METHOD

Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion.

RESULTS

Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process.

CONCLUSIONS

This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

Real-Time Dual Video Conferencing of Simulated Tracheostoma Maturation During Residency Interviews

BACKGROUND

There are few reports of dexterity tests being done in a distance telecommunication setting for residency applicant evaluation.

OBJECTIVE

To report the feasibility and suitability of a virtual suturing skills assessment during residency interviews when added to the standard assessment process.

METHODS

A suturing simulation was developed and implemented during otolaryngology-head and neck surgery (OHNS) residency interviews for the 2020-2021 cycle at one program. On the day of the interview, the activity was completed in real time using 2-camera video conferencing with the 2 resident assessors providing a numerical assessment based on an adapted scoring rubric from prior suturing activities at the institution. The exercise involved suturing a 3/4-inch Penrose drain circumferentially with half-vertical mattress stitches to simulate the maturation of a tracheostoma. The residency selection committee then completed a 7-item Likert-type survey, developed by the authors, to evaluate the simulation exercise.

RESULTS

Fifty-one applicants representing all interviewees in the cycle successfully completed this assessment without technologic disruptions. The total cost associated with obtaining and providing the necessary supplies to applicants was $34.78 per interviewee. Time required to complete the suturing task was estimated to range from 10 to 20 minutes. The residency selection committee viewed this exercise as a success (14 of 16, 87.5%) and viewed the results as a valuable adjunct in the overall assessment of candidates (15 of 16, 93.8%).

CONCLUSIONS

A simple motor exercise completed over real-time telecommunication was feasible and perceived as helpful to the residency selection committee when assessing OHNS residency candidates.

National Resident Matching Program Rank Order and Performance in an Internal Medicine Residency

OBJECTIVES

Prospective first-year house staff and residency program leaders spend substantial time, effort, and expense preparing a rank order list for the National Resident Matching Program (NRMP). Previous studies have mostly shown minimal or no relation between rank order and subsequent resident performance, raising questions about the value of this process. Furthermore, no previous studies have been done with Internal Medicine residencies. As such, the purpose of this study was to compare NRMP rank order to multiple objective outcomes of an Internal Medicine residency.

METHODS

A retrospective cohort of Internal Medicine residents from five consecutive graduating classes, trained between July 1, 2013 and July 31, 2020, were evaluated for five objective outcomes: Accreditation Council for Graduate Medical Education (ACGME) milestones, faculty rankings of quality, National In-Training Examination scores, chief resident attainment, and fellowship attainment. Outcomes were analyzed in relation to eight potential predictors: NRMP rank, medical school type and grades, immigration status, added qualifications, sex, age and US Medical Licensing Examination (USMLE) scores, using univariate and multivariate analyses.

RESULTS

From a cohort of 61 residents, 56 were eligible. All eligible residents' data were included, for a participation rate of 100% (56 of 56). There were no statistically significant univariate or multivariate predictors for the endpoint of fellowship attainment. Higher USMLE scores were predictive of chief resident status in univariate analysis only. NRMP rank was significantly correlated with ACGME milestones in the univariate analysis. The multivariate analysis revealed that higher USMLE score was statistically significantly predictive of more favorable milestones, faculty ranking, and National In-Training Examination score.

CONCLUSIONS

Higher USMLE score was statistically significantly associated with multiple favorable objective residency outcomes in an Internal Medicine residency. A better NRMP rank was correlated with favorable ACGME milestones in univariate analysis, but USMLE score emerged as the strongest predictor in multivariate analysis.

Does medical school research productivity predict a resident's research productivity during residency?

BACKGROUND

Research productivity is an important component of the CanMEDS Scholar role and is an accreditation requirement of Canadian Otolaryngology training programs. Our objective was to determine if an association exists between publication rates before and during Otolaryngology residency.

METHODS

We obtained the names for all certified Canadian Otolaryngologists who graduated between 1998 and 2013 inclusive, and conducted a Medline search for all of their publications. Otolaryngologists were subgrouped based on year of residency graduation and the number of articles published pre-residency and during residency (0 or ≥1). Chi-squared analyses were used to evaluate whether publications pre-residency and year of graduation were associated with publications during residency.

RESULTS

We obtained data for 312 Canadian Otolaryngologists. Of those 312 graduates, 46 (14.7%) had no identifiable publications on PubMed and were excluded from the final data analysis. Otolaryngology residents had a mean 0.65 (95% CI 0.50-0.80) publications before residency and 3.35 (95% CI 2.90-3.80) publications during residency. Between 1998 and 2013, mean publication rates before and during residency both increased significantly (R ² = 0.594 and R ² = 0.759, respectively), whereas publication rates after residency graduation has stagnated (R ² = 0.023). The odds of publishing during residency was 5.85 times higher (95% CI 2.69-12.71) if a resident published prior to residency (p < 0.0001). The Spearman correlation coefficient between publications before and during residency is 0.472 (p < 0.0001).

CONCLUSION

Residents who publish at least one paper before residency are nearly six times as likely to publish during residency than those who did not publish before residency. These findings may help guide Otolaryngology program selection committees in ranking the best CaRMS candidates.

Associations between Otolaryngology Applicant Characteristics and Future Performance in Residency or Practice: A Systematic Review

Objective

This systematic review aims to evaluate which applicant characteristics available to an otolaryngology selection committee are associated with future performance in residency or practice.

Data Sources

PubMed, Scopus, ERIC, Health Business, Psychology and Behavioral Sciences Collection, and SocINDEX.

Review Methods

Study eligibility was performed by 2 independent investigators in accordance with the PRISMA protocol (Preferred Reporting Items for Systematic Reviews and Meta-analyses). Data obtained from each article included research questions, study design, predictors, outcomes, statistical analysis, and results/findings. Study bias was assessed with the Quality in Prognosis Studies tool.

Results

The initial search identified 439 abstracts. Six articles fulfilled all inclusion and exclusion criteria. All studies were retrospective cohort studies (level 4). Overall, the studies yielded relatively few criteria that correlated with residency success, with generally conflicting results. Most studies were found to have a high risk of bias.

Conclusion

Previous resident selection research has lacked a theoretical background, thus predisposing this work to inconsistent results and high risk of bias. The included studies provide historical insight into the predictors and criteria (eg, outcomes) previously deemed pertinent by the otolaryngology field. Additional research is needed, possibly integrating aspects of personnel selection, to engage in an evidence-based approach to identify highly qualified candidates who will succeed as future otolaryngologists.

Technical Innovation: The Automated Residency Match Rank List

The creation of the final rank list for the National Residency Matching Program every year is a laborious task requiring the time and input of numerous faculty members and residents. This article describes the creation of an automated visual rank list to efficiently organize and guide discussion at the yearly rank meeting so that the task may be efficiently and fairly completed. The rank list was created using a PowerPoint (Microsoft) macro that can pull information directly from a spreadsheet to generate a visual rank list that can be modified on-the-fly during the final rank list meeting. An automatically created visual rank list helps facilitate an efficient meeting and creates an open and transparent process leading to the final ranking.

Understanding the Interview and Ranking Behaviors of Unmatched International Medical Students and Graduates in the 2013 Main Residency Match

BACKGROUND

Over the past decade, the number of unfilled positions in the National Resident Matching Program (NRMP) Main Residency Match has declined by one-third, while the number of unmatched applicants has grown by more than 50%, largely due to a rise in the number of international medical school students and graduates (IMGs). Although only half of IMG participants historically have matched to a first-year position, the Match experiences of unmatched IMGs have not been studied.

OBJECTIVE

We examined differences in interview and ranking behaviors between matched and unmatched IMGs participating in the 2013 Match and explored strategic errors made by unmatched IMGs when creating rank order lists.

METHODS

Rank order lists of IMGs who failed to match were analyzed in conjunction with their United States Medical Licensing Examination (USMLE) Step 1 scores and responses on the 2013 NRMP Applicant Survey. IMGs were categorized as "strong," "solid," "marginal," or "weak" based on the perceived competitiveness of their USMLE Step 1 scores compared to other IMG applicants who matched in the same specialty. We examined ranking preferences and strategies by Match outcome.

RESULTS

Most unmatched IMGs were categorized as "marginal" or "weak". However, unmatched IMGs who were non-US citizens presented more competitive USMLE Step 1 scores compared to unmatched IMGs who were US citizens. Unmatched IMGs were more likely than matched IMGs to rank programs at which they did not interview and to rank programs based on their perceived likelihood of matching.

CONCLUSIONS

The interview and ranking behaviors of IMGs can have far-reaching consequences on their Match experience and outcomes.

Use of the Interview in Resident Candidate Selection: A Review of the Literature

BACKGROUND

Although the resident candidate interview is costly and time-consuming for both applicants and programs, it is considered critically important for resident selection. Noncognitive attributes, including communication skills and professionalism, can be assessed by the personal interview.

OBJECTIVE

We conducted a review of the literature on the residency interview to identify the interview characteristics used for resident selection and to ascertain to what extent the interview yields information that predicts future performance.

METHODS

We searched PubMed and Scopus using the following search terms: residency, internship, interview, selection, and performance. We extracted information on characteristics of the interview process, including type of interview format, measures taken to minimize bias by interviewers, and testing of other clinical/surgical skills.

RESULTS

We identified 104 studies that pertained to the resident selection interview, with highly varied interview formats and assessment tools. A positive correlation was demonstrated between a medical school academic record and the interview, especially for unblinded interview formats. A total of 34 studies attempted to correlate interview score with performance in residency, with mixed results. We also identified a number of studies that included personality testing, clinical skills testing, or surgical skills testing.

CONCLUSIONS

Our review identified a wide variety of approaches to the selection interview and a range of factors that have been studied to assess its effectiveness. More research needs to be done not only to address and ascertain appropriate interview formats that predict positive performance in residency, but also to determine interview factors that can predict both residents' "success" and program attrition.

Associations between residency selection strategies and doctor performance: a meta-analysis

OBJECTIVES

The purpose of this study was to use meta-analysis to establish which of the information available to the resident selection committee is associated with resident or doctor performance.

METHODS

Multiple electronic databases were searched to 4 September 2012. Two reviewers independently selected studies that met the present inclusion criteria and extracted data in duplicate; disagreement was resolved by consensus. Risk for bias was assessed using a customised bias assessment tool. Measures of association were converted to a common effect size (Hedges' g). Meta-analysis was performed using the random-effects model for each selection strategy and all outcomes without pooling. Sensitivity analysis for each selection strategy-outcome pair was performed with pooling of effect size.

RESULTS

Eighty studies involving a total of 41 704 participants were included in the meta-analysis. Seventeen different selection strategies and 17 outcomes were assessed across these studies. The strongest positive associations referred to examination-based selection strategies, such as the US Medical Licensing Examination (USMLE) Step 1, and examination-based outcomes, such as scores on in-training examinations. Moderate positive associations were present for medical school marks and both examination-based and subjective outcomes. Minimal or no associations were seen for the selection tools represented by interviews, reference letters and deans' letters.

CONCLUSIONS

Standardised examination performance and medical school grades show the strongest associations with current measures of doctor performance. Deans' letters, reference letters and interviews all show a lower than expected strength of association given the relative value often assigned to them during resident doctor selection. Objective selection strategies are potentially the most useful to residency selection committees based on current evaluative methods. However, reports in the literature of validated long-term doctor performance outcomes are scant.

Standardized letter of recommendation for otolaryngology residency selection

OBJECTIVES/HYPOTHESIS

Develop a standardized letter of recommendation (SLOR) for otolaryngology residency application that investigates the qualities desired in residents and the letter writer's experience. Compare this SLOR to narrative letters of recommendation (NLORs).

STUDY DESIGN

Prospective SLOR/NLOR comparison.

METHODS

The SLOR was sent to an NLOR writer for each applicant. The applicant's NLOR/SLOR pair was blinded and ranked in seven categories by three reviewers. Inter-rater reliability and NLOR/SLOR rankings were compared. Means of cumulative NLOR and SLOR scores were compared to our departmental rank list.

RESULTS

Thirty-one SLORs (66%) were collected. The SLORs had higher inter-rater reliability for applicant's qualifications for otolaryngology, global assessment, summary statement, and overall letter ranking. Writer's background, comparison to contemporaries/predecessors, and letter review ease had higher inter-rater reliability on the NLORs. Mean SLOR rankings were higher for writer's background (P = .0007), comparison of applicant to contemporaries/predecessors (P = .0031), and letter review ease (P < .0001). Mean SLOR writing time was 4.17 ± 2.18 minutes. Mean ranking time was significantly lower (P < .0001) for the SLORs (39.24 ± 23.45 seconds) compared to the NLORs (70.95 ± 40.14 seconds). Means of cumulative SLOR scores correlated with our rank list (P = .004), whereas means of cumulative NLOR scores did not (P = .18). Means of cumulative NLOR and SLOR scores did not correlate (P = .26).

CONCLUSIONS

SLORs require little writing time, save reviewing time, and are easier to review compared to NLORs. Our SLOR had higher inter-rater reliability in four of seven categories and was correlated with our rank list. This tool conveys standardized information in an efficient manner.